Xpk File System Handler v1.39 By Kristian Nielsen This package may be distributed freely for non-commercial purposes along with Xpk or on its own, provided that all files and subdirectories are left intact. However, all rights remain with the author. And as usual, the author will in no way be held responsible for any damage, directly, implicit or otherwise, resulting from the use of this program. After all, you get this program for free; use it or not as you like, and at your own risk. The file 'XFH.doc' (which should be distributed alone with this file) gives information on how to get started quickly while this text gives a more technical discussion of the XFH file system handler and describe some of the more obscure options that are not accessible through the gui. Some knowledge of the more technical aspects of the Amiga is assumed. Also, this document is pretty unstructured, being just a random collection of notes gathered together over the last year of XFH development. Do read XFH.doc first to get an overview of what XFH is. A lot of users probably don't wan't to read this file at all; XFH.doc will tell them all they need to know. --- O --- What's possible. XFH, when used with Xpk, makes it possible to store data in compressed format without this being visible to the user or to application programs - XFH will make the compressed data appear like ordinary files. This is by no means a new idea. In MS-DOS world, a (commercial) program called 'Stacker' has been available for some time, which makes it possible to compress a whole partition on a harddisk. On the amiga, people have long used powerpacker along with programs like PPMore to store data files in packed formats (but relying on the application to recognise that the file is in compressed format), and programs like PPPatch have been used to change dos.library to recognise the powerpacker format automatically. However, none of these approaches are perfect in all situations, and so there is room for another alternative - the XFH. XFH works in conjuction with the Xpk approach to data compression. This in itself gives a number of advantages - a flexible interface to the compressing algorithms, lots of different packers available with the possibility to add new ones etc. But compared to approaches like 'Stacker', there is an additional advantage: access to the compressed data is not limited to XFH - most of the time, the XFH way will be the most convenient way to access data, but if needed, the complete range of Xpk applications is available to the user. The file orientated nature of Xpk also means that XFH - like PPPatch etc., but unlike Stacker - will co-exists nicely with any other Amiga filing system without the need to set up any new partitions or prepare the disk with a special program. In fact, if you happened to have a CD-Rom (read only) stuffed with powerpacked files (or any other format that the current version of Xpk supports), you could dump an XFH unit on top of it and largely forget about the disk being compressed from then on. So, what is possible is to mount a XFH unit on top of each of one or more standard AmigaDOS directories. It is now possible for programs to access the directories as usual, data being decompressed and optionally compressed as needed in a completely transparent way - programs will never know the difference between compressed and uncompressed files - while still having access to the compressed data using conventional compress/uncompress programs. So the dream is that of doubling the capacity of your hard- or floppydisks for zero cost. As an example of the possibilities of the XFH, I have been using it for holding various doc files, metafont sources and little used emacs scripts and shell commands on HD, saving in the order of perhaps 60% file space with no noticeable degrading of overall system performance. --- O --- What is the catch? Of course, as everyone knows, nothing comes for free. Obviously, there is a speed penalty to compressing or uncompressing a file. This speed penalty will be highly dependent on the actual compression algorithm and data media used; for example an algebraic compression scheme used with a superfast HD will probably be rather slow, while a fast algorithm may actually result in a speed-up of floppy access since the disk access time saved by the smaller file size more that accounts for the time taken decompressing. And of course with the coming of ever more powerful CPU's the speed will be less of a problem. Some things will always be slow, though. Especially directory listning is a problem, since every single file has to be opened in order to check whether the file is compressed or not. In fact, I'll admit that running XFH from floppy on an unaccelerated amiga will sometimes seem a bit slow. However, XFH was not written to be as fast and small as possible, but rather to be flexible and expandable. And I'm sure a lot of users will find it very useful even on 'small' amigas. On an A3000, XFH runs like a dream, of course. And there is still the possibility for speed improvements in later versions. Aside from the problem of speed, V1.39 of the XFH comes with a few other limitations that I'm hoping to remove in later versions. Most important is the lack of some of the new 2.0 packets, a lack that will become more severe as more 2.0-only programs start to depend on these packets. Another problem is that XFH in the present version is somewhat memory-hungry, in that it will keep any compressed file completely unpacked in memory as long as that file is open. This is mostly due to limitations in the current Xpk interface, and I'm hoping to remove this in a later release. However, currently this means that it is impossible to open a compressed file if it is larger than available memory. [One way to solve this would be to have XFH use one of the 'virtual memory' programs that are starting to appear. In fact, I have had one or two reports about this actually working. If someone wants to discuss this I'd very much wellcome it.] A number of other problems to look out for are detailed in a later section in this doc file. Most of these are things that could be fixed in later versions. --- O --- Instructions for use. XFH is implemented as an AmigaDOS device handler. The 'L:' directory contains examples of other such device handlers, and this is usually also the best place to put XFH (the file 'XFH-Handler', to be precise). Like other handlers, XFH must be mounted before it can be used. This can be done by creating an entry for it in the 'DEVS:Mountlist' file (just the mountlist for short) and issuing the command 'Mount ', where is the name given to the device in the mountlist. The supplied file 'Devs/Mountlist.custom' contains example entries for the XFH; it might be convenient to append this file to the end of 'DEVS:Mountlist' (the entries assume an assign 'xfhdir:' to the XFH distribution directory). The most important part of each mountlist entry is the Device name (XH1: etc) and the 'Startup' entry, since these are used to configure the various options of XFH.. Further information on the mechanism of mountlists and handlers in general can be found in various places; I will not attempt any lenghty explanation here. Unlike most other 'normal' device handlers, XFH needs additional information to function correctly. Most importantly it needs to be told what directory it should use as its root directory (that is, where it should look for compressed files). In the simplest case, this can be done by having an assigning (or device volume name) 'nn:' to this directory and mounting an XFH partition with the name 'Xnn:' (any letter will do, not just 'X'). However, XFH also provides 'options', some of which cannot be controlled by the gui but only through the Mountlist or by using Arexx. Options can be specified in the mountlist or in an option file, and can be changed dynamically using AREXX. An option is specified as a string following the same conventions as the TOOLTYPES of the Workbench. If the number of options that should be set are limited, it can be convenient to put them directly in the mountlist entry. This is done by using the 'Startup' keyword. Use a line of the form Startup = "!=!=...!=" (that is, a list of option assignments preceded by '!' (no spaces). The '"'s are optional and the '='s can be replaced with '&'s (this is nessesary to be compatible with some mount commands). See the entry for 'XH0:' in 'Devs/Mountlist.custom' for an example. For a larger number of options, option files should be used. XFH has the concept of primary and secondary option files. The primary option file is used by putting a line of the form Startup = xfhdir:optionfiles/.xfhrc_1 into the mountlist entry (distinguised from the other use of this keyword by the fact that no '!' appears). The filename can be anything you like, of course. The secondary option file is named '.xfhrc' and is placed in the root directory of the XFH unit (uncompressed!). The settings in the secondary option file overrides any settings in the primary option file, but there are some restrictions on the options that can be used in the secondary option file, see below. Both files are optional, the handler will pick default values for any options that are not set by the user. In the option files, options are given each on a line of its own. Some examples are given in the directory 'Optionfiles'. Be careful when using the secondary option file; this is also used by XFH itself to store information that must be preserved across reboots (notably the volume name for Relabel). When the handler is mounted, most options can be changed by sending an approproate AREXX command to the handler. The name of the port is the same as the name of the device, though this can be changed (see below). Note that XFH will refuce to mount if it cannot open the AREXX port because of a naming conflict. To set an option, send a command SETOPTION to the port, where should be in the same format as that used in option files. The directory 'Arexx/' contains example AREXX scripts that may be usefull. For example the command rx SetAutocompress XDH1 ON will enable automatic compression on XDH1:. Boolean options can be specified with "NO" / "OFF" or. "YES" / "ON". String options are specified by simply putting the string after the 'OPTION=' bit, no quotes are needed. The available options are detailed below: ROOTDIR VOLUMENAME AUTOCOMPRESS XSCAN PACKMODE STEPDOWN PASSWORD XPKPRIORITY TRUNCATEONPACK FAILONEXNEXT KILLSTARTUP COMPRESSREADWRITE ALLOWAPPEND PORTNAME ENVOYKLUDGE Option ROOTDIR: This option is used to set the name of the directory that XFH: is to reside in. For example 'ROOTDIR=Work:xfh' would cause XFH: to use that directory as root. Note that this option can only be used in the primary option file or directly in the mountlist. When used elsewhere it will simply be ignored. The default is to use the name of the XFH device itself with the first character removed (so XDH1: becomes DH1:). [Note that this is different from XFH v1.12 and earlier.] Option VOLUMENAME: This option is used to set the name of the XFH: volume. This is the name that will be used in absolute path specifications, as well as the one returned by Info(). Ie. if you go 'VOLUMENAME=Manuals', access can be by 'Manuals:' as well as by 'XHn:' (or whatever). If this option is not specified, the default is to use the name of the directory in the underlying file system, or (if this is itself the root of a volume) to use the name with 'XFH_' prepended. Again, this is mainly for compatibility with early versions of XFH:. An alternative to using this option is to use the normal Relabel (from shell or WB); this will automatically create an entry for this option in the secondary option file (replacing any existing entry). Option AUTOCOMPRESS: This options tells whether XFH: should attempt to compress the files written through it. When this option is set, everytime a file written to the XFH: is closed, an attempt will be made to compress the file to a temporary file using Xpk. If this is succesfull, the temporary file will be renamed to the original name and the uncompressed file will be deleted. If the compression fails for any reason, the uncompressed file will simply remain intact. This also means that if anything should go wrong during compression (like a disk full error), it is unlikely that any data will be lost since at least one of the two files should be intact (though possibly with a wierd name). This option is OFF by default. Option XSCAN: This option tells whether XFH: should create xScan-like comment automatically. These comments will allow XFH to read directories much faster, see "xScan.doc" for more information. This option is OFF by default. Option PACKMODE: This option selects the mode that Xpk should use when compressing files. It is specified in the usual way when using Xpk. For example, to use 12-bit BLZW compression, 'PACKMODE=BLZW.12' would be used. The default is to use the NUKE compression ('PACKMODE=NUKE'). Of course, to use a specific compression method, the nessesary sublibrary must be available in LIBS:. Option STEPDOWN: This option controls the Xpk flag 'XPK_StepDown' during packing. If set, it means that Xpk is allowed to reduce packing efficiency if nessesary to save memory. Refer to the Xpk documentation for details. Default is OFF. Option PASSWORD: This option is used to set the password that XFH: should pass on to Xpk when compressing or uncompressing. It should be noted that there is no attempt to keep this password safe from 'memory peekers'. Of course, storing the password in the option file isn't a good idea. A better idea is to use AREXX; the AREXX script 'SetPassword.rexx' in the directory 'Arexx/' may be helpful here. Note that if AUTOCOMPRESS is requested, files may still be saved unencrypted in low-memory or low-diskspace situations, and (depending on the underlying file system / disk device) part of the unencrypted data may still be physically stored on the disk after the deletion of the unencrypted file. If an attempt to open a file is made when XFH has been given the wrong password, the open will fail with error code 212 (Object wrong type). Option XPKPRIORITY: This option is used to set the task priority that should be used when doing Xpk operations (compress/uncompress etc). Setting this to zero or less will prevent XFH from stealing all CPU-time from tasks running at a 'normal' priority. Note that it is possible to set the priority of the handler itself in the mountlist. If this option is not used, the handler will keep whatever priority it is running at when calling the Xpk library. Option TRUNCATEONPACK: This option is somewhat technical in nature and can be safely ignored. It is only used in case of an error occuring during the compression of a file. In this case, the compressed file has to be deleted, and if this option is set, the handler will try to call SetFileSize() first to truncate the file to 0 bytes (perhaps saving the flushing of a few buffers). However, due to sparse documentation I'm uncertain whether this feature is stable, and hence it is OFF by default. Again, unless you are curious and don't mind risking crashes/data losses, forget about this option. Option FAILONEXNEXT: This option was included to solve a problem with some (or more likely most) programs that perform directory scanning. The problem appears when listing a directory containing a file that is opened with an exclusive lock (for example, MODE_NEWFILE). In this case, XFH cannot determine the correct size of the file and thus fails with an appropriate error code. However, many programs just assume that the end of the directory has been reached. Setting this option to OFF will prevent XFH from reporting failure, returning a fileinfoblock with the wrong file size instead. Since this could result in data loss because programs will se the wrong file sizes, this option is ON by default. Option KILLSTARTUP: This option has been included to fix some problems with programs like 'Info' that examine data about file system handlers. It seems that these programs assume that the 'Startup' entry in the device node of any handler that supports volumes will be a FileSystemStartupMsg. However, in the case of XFH, it is a string. This behaviour seems to me to be completely unresonable, and thus a bug in these programs. The only way I could find to fix this was to have XFH manually erase the Startup field in the device node. This is not likely to be an officially supported way of poking the device node. Nevertheless, I had to agree that the problem is intolerable, and thus this option is on by default. However, it can be turned off if you have a mount command that does not like handlers that modify their device node. Option COMPRESSREADWRITE: This option controls whether new files opened with MODE_READWRITE will be compressed upon Close(). It is ON by default. Option ALLOWAPPEND: This option must be set to allow the writing to existing files (using MODE_READWRITE or MODE_OLDFILE). It is my personal opinion that compressed files shouldn't really be updated in place, since it requires first uncompressing the file on disk, then doing the Write(), then compressing the file again. However, some programs need this ability. This option is ON by default. Option PORTNAME: This option can be used to set the name of the AREXX port. This option can only be set in the option file or directly in the mountlist (no, you cannot set it using AREXX...). The default name is the same as the name of the XFH device. Option ENVOYKLUDGE: This option tells XFH: not to use ACTION_FH_FROM_LOCk for internal purpose to avoid trouble with the buggy implementation of this packet in Envoy FS. Without this kludge using a XFH: on a Envoy FS volume will crash the FileSystem on the server. The kludge may be not necessary for future version of Envoy. This option is OFF by default. In case of an error during the scanning of the option files the handler will fail its initialisation and hence refuse to load with error code 114 (Bad Template). Sorry, but there are currently no real error messages implented (this will hopefully be fixed in a later version). If the handler refuses to work for no apparent reason, be sure to tripple-check your option files for errors. --- O --- Limitations and known bugs. The error detection code in the initialisation part of the handler is somewhat flaky - I've tried to make it resonably safe, but documentation on the right way to start a handler is hard to find. What it means is that it is a good idea to make sure that the handler is placed in L: and that the nessesary Xpk libraries are placed in libs: before starting the handler (remember, XFH won't be able to tell you the reason if it was unable to initialise for some reason). Another subtle problem is that due to a quirk of the device list locking, it is vital (using XFH v1.39) that the handler for the directory that XFH is to sit in is already loaded. Usually this will not be a problem; however, if you are using 3rd party filesystem handlers that are mounted after boot-up, you can avoid problems by accessing them before mounting XFH (for example by creating an assign (not late-binding) to them). It should be noted that the way the XFH makes the same files available by two different routes is not without its problems. One problem is connected to the volume name - XFH tries to be smart about it, but it will sometimes create a duplicate volume name which is a bad idea. To solve this problem the VOLUMENAME option should be used in an option file, or the XFH volume should be changed with Relabel. Another problem when mounting XFH in the root dir appears when using the 'Leave Out' feature of the 2.0 Workbench. Here, the '.backdrop' is duplicated in both volumes, making the left-out icons appear twice. I'm working on a decent solution to this problem. Meanwhile, I would recommeys at Commodore for bringing to us the wonderful Amiga.