Compressdisk.device 37.25 (release 1.6) by Antonio Schifano 1 - Introduction. Data compression is not a new thing in the Amiga scene: since many years there exist many commercial and public domain utilities that give to all Amiga users the possibility to save disk space. We can roughly divide data compressors in two main classes: the class of archivers (like ZOO, LHARC, etc.), and the class of "file oriented" compressors (like COMPRESS, POWERPACKER, IMPLODER, etc.). Archivers' main feature is their ability to preserve the files hierarchy; the price to pay is the impossibility to use the archived files in the compressed form. On the contrary, file oriented compressors sacrifice this feature to achieve a better usability of compressed files: programs compressed by PowerPacker, for instance, remain executable; moreover, the package includes some useful tools (like PPMORE) that compensate for the versatility lost because of the compression (for instance, you can still read compressed text files, but you cannot edit them). Compressdisk.device tries to resolve the defects of both types of packers with a different approach to the problem. 2 - What is compressdisk.device? As its name probably suggests, it is an Exec device, similar in the way of operating to Matt Dillon's fmsdisk.device. Like that, in fact, it allows you to create virtual disks, whose tracks are saved on a file. Unlike that, however, it uses a different file for each track, and compresses the tracks. This allows you to create virtual partitions on your hard-disk, usable like normal partitions, but having the great advantage that every file that you write in them is automatically compressed (and obviously, it is decompressed whenever you read it back). Files stored on virtual partitions can be rewritten (even partially), and applications can even Seek() into them, as with usual files. 3 -Installation. The device can be installed only on Amigas with kickstart V37+. The installation is (almost) totally performed by the programs "CDInstall" and "CDPartition", who take care of copying all necessary files in the right directories: - The file "compressdisk.device" is copied in the DEVS: directory. - For each virtual partition desired, an opportune entry is created in the DEVS:mountlist file. If you are using Workbench V38+, all entries will be created in the DEVS:DOSDrivers directory. - The preference editor "CDPrefs" is copied in the SYS:Prefs directory. Notice that CDInstall and CDPartition need a standard environment to work properly; I have received many complains from people who claimed that these programs refused to work on their machines, or had strange behaviours: anyway, CDPartition and CDPrefs have been tested on many Amigas (including the A4000) and under OS2.04 (even with Workbench 2.1) and 3.0. Make sure, in particular, that the environment variables "Kickstart" and "Workbench" exist. For the device work properly, you should add the following line to your s:startup-sequence file, just after the command SetPatch: assign CDISK: where is the complete path of the directory in which you want the device to store the compressed tracks. Else, the device will put the tracks in the root directory of the boot partition. Once mounted, each virtual partition must be initialized; this can be easily accomplished with the system command "format". Since the device stores only the tracks written at least once, it is advisable to use the QUICK option: this will let you save disk space and time (non-quick formatting of a 40M partition can take more than thirty minutes on a standard Amiga 500). To speed up read/write operations, the device features a built-in cache in which it stores the most recently used tracks; the cache size can range from one to thirtytwo tracks. The default configuration is eight tracks per unit; from tests the device proved to be usable with four or, if you want to save your memory, even with two tracks, whereas with only one track the performance reduced significantly, expecially in writing. 4 - How to use compressdisk.device. You can use virtual partitions created with compressdisk.device like usual ones; nevertheless, it is better to follow some guidelines. First of all, it is recommendable not to write any files into any virtual partition if the device work disk (CDISK:) is full: if the device cannot store some tracks, you may lose some data. Moreover, you should not use virtual partitions for storing temporary files; remember that deleting a file (on the standard file system), does not clear the blocks that it occupied (in fact, you can recover it using proper utilities), until the file system does not overwrite them. This means that when you delete a file you will NOT free any space on the device work disk. Virtual paritions created with the partitioning program use the Fast File System; however you can use any file system you want. Since Standard File System and the correspondent International version reserve part of the data blocks for internal informations, they can reduce the tracks compression; for this reason their use is not recommended. Users of kickstart V39 can take full advantage of the new directory-caching file systems. 5 - The compression algorithm. The more decisive choice in the device realization was the selection of the compression algorithm: it is clear that the device performance depends on the data compressor and decompressor ones. The algorithm had to be as fast as possible, to use little memory, and reasonably good at compressing. Originally (release 1.0 and 1.5) the device compressed the tracks with Ross William's LZRW3A, which features 16K/sec average compression speed and 64K/sec average decompression speed, using only 16k of working memory (assembler optimized version on a plain A500). Release 1.6 of compressdisk.device uses a new compression algorithm, derived from LZRW3-A, which is a little bit faster and crunches data a little better (but uses an extra 16K of working memory, for a total of 32K). The following table gives a rough idea of the performances obtained by the new algorithm (that I named LZRWS3) on the standard Calgary corpus files, compared to the old LZRW-3A: -----------------+---------------------------+-------------------------- | lzrw3a | lzrws3 -----------------+---------------------------+-------------------------- file size | size K/s K/s % rem | size K/s K/s % rem | com dec | com dec | | bib 111261 | 49047 17.8 70.9 44.1 | 45268 20.1 77.3 40.7 book1 768771 | 416139 15.3 65.3 54.1 | 390213 16.9 69.5 50.8 book2 610856 | 274378 18.3 74.8 44.9 | 260781 20.0 79.4 42.7 geo 102400 | 84221 8.7 34.0 82.3 | 81333 9.3 35.6 79.4 news 377109 | 191298 15.2 60.3 50.7 | 174211 17.3 66.7 46.2 obj1 21504 | 12655 11.9 44.8 58.8 | 12388 11.9 44.8 57.6 obj2 246814 | 108048 17.1 68.5 43.8 | 101917 18.5 68.4 41.3 paper1 53161 | 24530 17.0 63.3 46.1 | 24170 17.7 72.8 45.5 paper2 82199 | 39486 17.1 68.5 48.0 | 38469 18.5 76.1 46.8 pic 513216 | 111008 40.7 133.3 21.6 | 72802 38.8 137.6 14.2 progc 39611 | 17926 17.3 66.0 45.3 | 17482 17.4 66.0 44.1 progl 71646 | 24366 23.8 85.3 34.0 | 22232 25.9 85.3 31.0 progp 49379 | 16808 22.9 82.3 34.0 | 15136 27.4 82.3 30.6 trans 93695 | 30292 23.7 86.7 32.3 | 25122 29.9 97.6 29.9 -----------------+---------------------------+-------------------------- average 224402 | 19.0 71.7 45.7 | 20.7 75.7 42.9 total 3141622 | 1400202 | 1281524 TABLE 1 - LZRW3A and LZRWS3 performances on the standard Calgary corpus (A500 with MC68000 at 7.16MHz). Notice that previous tables refer to registered version; the evaluation version supplied in this archive is about 35% slower (but has the same efficiency). Compressdisk.device release 1.6 is backward compatible with 1.5 and 1.0 (partitions created with the previous version will work fine under the new version). For further informations about LZRW3-A refer to the original documentation given by Ross Williams (directory "pub/compression" on the FTP site sirius.itd.adelaide.edu.au [IP=129.127.40.3]). The standard Calgary corpus can be found in the directory "/pub/text.compression.corpus" on the FTP site fsa.cpsc.ucalgary.ca [IP=136.159.2.1]. 6 - Implementation. From a technical viewpoint, compressdisk.device may look a bit strange, since it seems to break the hierarchy of the Amiga OS. In fact it is unusual for a device to use the DOS, since this one is at an higher level in the hierarchy. The hierarchy is purely functional: devices are used for management of physical resources (eventually virtual ones) and handlers for O.S. logical resources. For this reason a device is allowed to implement its primitives using a logical resource, provided that the tasks which use the device are not aware of the implementation of the primitives. The device core is a process which satisfies all I/O requests: whenever it receives a CMD_READ command, it loads and decompresses all the needed tracks; in a similar way, whenever it receives a CMD_WRITE or CMD_FORMAT command, it compresses and stores the tracks. If possible, the process uses the cache both in reading and writing: in reading, if a track is already in the cache, there is no need to reload and decompress it; in writing, tracks are compressed and stored only if there is no space left in the cache. Tracks are updated upon receiving a CMD_UPDATE command; data consistency is thus ensured. Since version 1.5, the user can choose (with CDPrefs) how the device must update the caches in writing; there are three possible ways: Standard: buffers are updated as soon as the device receives a CMD_UPDATE command. This is the standard behaviour of Amiga devices. Smart: the device updates the buffers only if a CMD_UPDATE command has been received and a certain (user-programmable) amount of time has elapsed without receiving other write requests. Safe: in this mode the device works in the same way as in standard mode; the device, however, tries to preserve the virtual partitions consistency by making an uncompressed copy of the cache buffers to disk for each CMD_WRITE command. The compression of tracks is performed only when they must be discarded from the buffers, or when a CMD_UPDATE command is received. In version 1.5 tracks used to be compressed each time they were written to disk, with great loss of performance (indeed, I think it was almost unusable). The only difference is that if a software failure happens while writing to a virtual partition, you will end up with some uncompressed - but still readable! - tracks; however, they will be compressed as soon as the device reads them. The best performance is obtained in SMART mode, that avoids all overheads caused by the repeated CMD_UPDATE requests issued by the standard AmigaDOS file systems. The default timeout the device uses in SMART mode is 5 seconds; you can however change this, according to your needs: the timeout can range from 1 to 60 seconds. If a software or hardware failure happens before the device updates its buffers, you may lose some data; it is therefore advisable to avoid too long timeouts (at least if you don't know what you are doing). The following table shows the time taken to copy the directory containing the standard Calgary corpus from RAM: to a virtual partition; timings refer to compressdisk.device 1.6 registered, using 8 buffers of cache, on a standard A500/A590 20M drive: Mode Time Speed -------------------------------------- Smart (5s) 4m 21s 12.50K/s Standard 4m 33s 11.95K/s Safe 7m 15s 7.51K/s TABLE 2 - Typical device performance (std A500/A590 with 20Mb drive) Since the standard Calgary corpus contains mostly big files, there is little DOS overhead, thus the device performance in "smart" mode is the same as in "standard" mode; for operations with shorter files, however, operations in "smart" mode are considerably faster; the following table reports the time taken to copy the assembler include files v2.0 (about 510K) to a virtual partition: Mode Time Speed -------------------------------------- Smart (5s) 0m 42s 12.35K/s Standard 1m 09s 7.52K/s Safe 1m 50s 4.72K/s TABLE 3 - Typical device performance with small files (A500+/A590). The track buffers can range from 1 to 32. Every track has 64 sectors of 512 bytes each, for a total of 32K. The device needs about 75K of working memory, plus about 32K for each track buffer. Notice that no extra memory is needed by the device for opening the files of a virtual partition. The highest cylinder number currently supported is 65000. The largest possible size of a virtual partition is thus 2031.25M. 7 - Acknowledgements. I wish to thank the following people, in no particular order: - Raffaele Dell'Aversana, for the academic discussions prior to the realization of the device; - Ross N. Williams, for releasing the LZRW* material; - Jan van den Baard, for the (useful!) utility "GadToolsBox"; - Giuseppe Sacco, for the utility "ZeroUno" (see the DISCLAIMER!! ;-); - my sister Letizia, for no particular reason; - all registered users. 8 - Copyright/distribution. Compressdisk.device, Copyright © 1992, 1993 Antonio Schifano. All rights reserved. This program is SHAREWARE; if you use it, send Lit. 15000 if you live in Italy, 20DM if you live in the European Community, or 20USD if you live somewhere else. Registered users will receive the registered version, and will have facilities for updating to future releases of the device. The program can be distributed only in its original archive format; can be included in collection disks of public domain software. It can also be distributed in electronic form via BBS, FTP, and similar. You may not charge more than Fred Fish does for a single library disk. Also no profit is to be made by selling this program. The commercial use and distribution of this program is NOT allowed without written permission from the author. The author cannot be held liable for any damage or data loss directly or indirectly caused by the use or misuse of this program; by deciding to use it, the user assumes all the responsibilities. Suggestions, bug reports and comments are welcome. Send cash, hints, registration forms to Antonio Schifano, at one of the following addresses: via C. Colombo, 28 56125 Pisa, Italy (valid until 30-6-1993) via dell'Oca, 11 91100 Trapani, Italy (always valid, but slower) The FidoNET adddress is no longer valid (I'm sorry for all the people who unsuccessfully tried to reach me via E-MAIL, but my local BBS was off most of the time, and now is definitely gone).