Program(s): PackDev Version: 1.3 Author: Christian Wasner Date: 06-May-95 -i) READ THE SECTION ABOUT PARAMETERS !!! THERE ARE MAJOR CHANGES !!! o) Version $VER: PackDev.doc V1.3 (30-Apr-95) i) Disclaimer The author cannot be held liable for the suitability or accuracy of this manual and/or the program(s) it describes. Any damage directly or indirectly caused by the use or misuse of this manual and/or the program it describes is the sole responsibility of the user her/him self. ii) Copyright/Distribution All files mentioned in iii) are (C) Copyright 1994, 1995 Christian Wasner. All rights reserved. These programs are FREEWARE, so no financial donations are required (but welcome). They may be freely distributed as long as all files remain unchanged and are included with the distribution. Distribution on disks or CDs is permitted only on the disks or CDs from Fred Fish or the Aminet CDs. Electronical distibution (e.g. by Aminet, mailboxes, modems) is allowed. Inclusion into freeware software packages is allowed, inclusion into other packages must be expressly allowed by me in written form. iii) Files The following files should come along: PackDev (23372 bytes) PackDev.doc (21965 bytes) iv) Documentation Program: PackDev Template: FROM/O,TO/O,P=PACK/K/O,XB=XPKBUFSIZE/K/N/O,M=MEMTYPE/K/O, CU=CLRUNUSED/S/O,ETDF=ETDFORMAT/S/O,TDF=TDFORMAT/S/O, ALL/S/O,NV=NOVERBOSE/S/O,Q=QUIET/S/O,NC=NOCONFIRM/S/O, VIEWFILE/K/O,VIEWFILESYS/K/O,TESTFILE/K/O,BLOCKLIST/K/O, PASSWORD/K/O Purpose: Reading/writing data directly from/to filesystems with xpk support. Specification: With this program you can read or write data directly from/to a disk. The blocks of the disk are read and stored into a file. When reading from a DOS disk, only the blocks that are used are archived. When writing a file from a DOS disk to a disk, only the blocks are written that are stored in the archive. Optionally the data read from a disk can be packed with an xpk packer. The program currently only has a shell interface. All this sounds like "Oh no, yet another DMS clone", but this program can handle any device with a filesystem (e.g. DHx:, DFx:, a RAD: of any size etc.), DMS can only handle floppy disks or devices with the size of floppy disks. Even Non-DOS disks can be handled (but then all blocks are read). Another advantage of PackDev is that is doesn't use stolen code like the authors of DMS do (see below). PackDev supports the xpk packer system thus it's much more flexible than DMS. You can use any xpk packer you like, i.e. you can use a packer suited to the type of data on the disk. When comparing size, you see that PackDev is only 21 KB long ! If you pack it with PowerPacker, it would be less than 12 KB ! A disk is read as follows: 1. Read device data from DOS 2. Check if the file system of the disk is supported (currently OFS, FFS, OFS International, FFS International, OFS Directory Dache and FFS Directory Cache are supported) 3. Inhibit device If the filesystem is supported and the ALL keyword is not set: 4. Read root block (contains location of block allocation map = BAM) 5. Read BAM and store it (after packing, if specified) 5. Read used blocks and store them (after packing, if specified) 6. Quit If the filesystem is not supported or the ALL keyword is set: 4. Read all blocks and store them (after packing, if specified) 5. Quit Writing functions similarly. When writing to a filesystem, DevPack will only allow filesystems that are exactly of the same partition size, block size and number of reserved blocks. If the disk should be formatted, the track size of the disk (e.g. 11 blocks for a floppy disk) must currently be equal, too. The following parameters are supported: FROM, TO ======== These parameter specifies the source data and the destination data. The source is specified first and the destination second. Either both or none of these two keywords must be set. If the one parameter is a filesystem the other one must be a file and vice versa. If both parameters are specified along with their keywords, they can be placed anywhere and in any order of the command string. If VIEWFILE, VIEWFILESYS or TESTFILE is set, FROM and TO must not be used. File names will be handled the following way: When writing to the file, a suffix ".pkd" will be added to the file name if it's not present. When reading from it and the file name doesn't contain the ".pkd" suffix, it's checked first if there is a file .pkd. If this file doesn't exist then the original file name is used. Note that copy-protected disks (i.e. with a custom track format) cannot be handled. Examples: DH0: foobar Read data from DH0: and write it to foobar.pkd foobar DH0: Read data from foobar.pkd (foobar if not present) and write it to DH0: TO foobar FROM DH0: Read data from DH0: and write it to foobar.pkd, other parameters can be placed anywhere between, before or after them. PACK (or P) =========== This parameter is optional, but strongly suggested. It may only be specified along with a READ action, because when writing to a disk, the packer type of the archive is automatically recognized. Along with PACK an xpk packer name and the efficiency can be specified (separated by a "."). The best xpk packers for standard data are NUKE and SHRI. NUKE is less efficient than SHRI, but much faster than SHRI. NUKE is 10-20% less efficient than the DMS packing algorithm, but faster. When unpacking, NUKE is much faster than DMS or SHRI. SHRI is 10-20% better than DMS, but slower. A general rule is : The less full a disk is, the slower is DMS because DMS always reads all blocks, even if they are not put into the archive (maybe ParCon is afraid of "lamers" who think, DMS forgets to read these blocks :-). Note that some packer types ignore the efficience value. Examples: PACK NUKE (Use NUKE packer with default efficiency) P SHRI.75 (Use SHRI packer with efficience 75) MEMTYPE (or M) ============== This parameter is optional. It specifies the memory type that is used for device buffers. Some older devices may require chip memory for their buffers. For example, under 1.3 the trackdisk.device needed chipmem because it used the blitter to decode the data (not because the disk DMA functions with chipmem only). Possible values are CHIP, FAST, ANY. Under 2.0+ trackdisk doesn't need chipmem any longer. Default is ANY. Examples: M CHIP (chipmem will be used or program fails) M FAST (fastmem will be used or program fails) M ANY (default: highest-priority memory will be used) XPKBUFSIZE (or XB) ================== This parameter is optional and can only be specified along with a READ action along with an xpk packer. It specifies the number of device blocks to be buffered before packing. If not specified, the default buffer size of the xpk packer is used or blocks with a total size of 65536 bytes when no packer is specified. Some packers have minimum or maximum buffer and some packers are more efficient with a larger size, so this parameter is implemented. If a packer is specified, the default buffer size is the one of the packer, if not, it's 65536 bytes (both will be extended to a multiple of the device block size) Examples: XB 200 (buffer = 200 blocks, 100KB along with a block size of 512 bytes) XB 50 (buffer = 50 blocks) CLRUNUSED (or CU) ================= This optional parameter is allowed only when writing to a disk. If set, all unused blocks are overwritten with zeroes. This has the disadvantage that the write operation becomes slower, but has the advantage that tools like DiskSalv ((C) by Dave Haynie) don't find old file fragments when trying to undelete an accidentally deleted file (deleted after usage of PackDev, of course). This parameter is unset by default. Example: CU (PackDev overwrites unused tracks with zeroes) ETDFORMAT (or ETDF) =================== This parameter is optional. It should be set if you want to write to an unformatted floppy disk. It may not function if you write to a hard disk, because this parameter causes PackDev to use a system format routine that functions with floppy disks (DFx:) but may not function with other devices (like oktagon.device, my scsi device, it took hours to find this out). The advantage of this format routine is that label buffers can be written with it in one row (Filenotes are NOT stored here, I told this in an earlier version of the doc file, ahem...). TDFORMAT will start an extra run for the label buffers (see below). For programmers: This causes PackDev to format with ETD_FORMAT. ETDF is unset by default. Example ETDF (enhanced format routine for floppies activated) TDFORMAT (or TDF) ================= This parameter is optional. It should not be set if you want to write to an unformatted floppy disk. It will function if you write to a hard disk, because this parameter causes PackDev to use the general format routine that should function with any disk device that is supported by DOS. This general system format routine (TD_FORMAT, to be specific) is slower than the format routine stated above, because it needs an extra write run to write the label buffers (don't ask my why, I don't know). I don't suggest to use this parameter for hard disks because they don't need to be Amiga-formatted. It is implemented because there may exist some devices that need formatting but don't support the enhanced routine. TDF is unset by default. Example TDF (standard format routine activated) ALL === This parameter can be used when reading from a device. If set, PackDev reads all blocks from the device, no matter if the filesystem is known or not. This is useful for all these poor demo disks that have a filesystem on it, but also raw data on blocks that are not used by the filesystem. I hesitated before implementing this parameter because I HATE disks of this kind, but SiliconSurfer insisted on it. Come on boys, stop these lame combination of DOS and trackloaders... Example ALL (All blocks are read) NOVERBOSE (or NV) ================= This parameter suppresses the output of the blocks currently worked on. This is useful when logging the output to a file. It's disabled by default. Example NV (no output of blocks currently worked on) QUIET (or Q) ============ If this parameter is set, nothing is written to the standard output and nothing is read from the standard input. This means that Packdev will always abort if there are any problems (read/write error, ^C pressed, write on existing file etc.) and reports no error text. It's disabled by default. Example Q (No input and output) NOCONFIRM (or NC) ================= If this parameter is set, PackDev will never ask for user input, i.e. it will immediately start with the specified action without waiting for confirmation. When errors occur, PackDev will always abort (see also QUIET). This is useful when starting PackDev with a file as standard input or when starting it from another program. It's disabled by default. Example NC (No input) VIEWFILE ======== This keyword will make PackDev output a file's filesystem information. It must be specified along with a file name and nothing else. Example VIEWFILE foo.bar VIEWFILESYS =========== This keyword will make PackDev output some information about a filesystem. It must be specified along with a filesystem name and nothing else. Example VIEWFILESYS DH0: TESTFILE ======== if PackDev is started with this parameter, along with a file name, the complete archive is read for testing purposes. It must be specified along with a file name and nothing else. Example TESTFILE foo.bar BLOCKLIST ========= If this parameter is set along with a file name, an ASCII block list is generated, one line for each block with the block number in decimal nad hexadecimal notation. This parameter cannot be set along with VIEWFILE and VIEWFILESYS. Example BLOCKLIST foobar PASSWORD ======== Setting this parameter allows usage of the XPK packers that are able to crypt the data. It can be used in combination with creating an archive if an xpk packer is used that supports crypting. When extracting or testing a crypted archive you must specify the password or decrypting will fail. Also note that there are countries where crypting data is not allowed or restricted. So if you live e.g. in the Iran, be careful that your head isn't chopped off... Example PASSWORD YouWillNeverGuessThis Examples for usage: =================== Reading the disk in DF0: and storing the data into RAM:disk.pkd, packing it with the SHRI algorithm: PackDev DF0: RAM:disk P SHRI.100 Crypting DF0: to RAM:Secret.pkd, using the IDEA algorithm: PackDev DF0: RAM:disk P IDEA.100 PASSWORD StupidPassword Writing data from DH0:dh1data.pkd to DH1: PackDev DH0:dh1data DH1: Writing data to an unformatted floppy disk (DF1:), not asking for user input, nonstandard parameter order and creating a block list file: PackDev ETDF NI FROM DF1: TO DH0:disk BLOCKLIST ram:disk.blocks Viewing the header of a .pkd file (ram:abc.pkd; not present: ram:abc): PackDev VIEWFILE ram:abc Viewing device information of DH0: PackDev VIEWFILESYS DH0: Testing archive integrity of foobar.pkd, an encrypted archive PackDev TESTFILE foobar PASSWORD YohMan v) XPK packers and their efficiency with PackDev Here is a list of all XPK packers I know. Note that some seem to be buggy (HUFF crashed with older versions of PackDev. It was my fault, sorry) and some don't function correctly with the default buffer size (see above) etc. You will get a short info about them if they are installed on your system when starting PackDev. The entries under "Packing efficiency" are produced with a strongly fragmented floppy disk (880k size, 44% full), using an A500/030/25MHz with 4 MB Fast and 1 MB Chip RAM. This list is not complete because my test disk is lost and I don't want to do all this again. Especially the graphical representation was hard work. ----- Info ------- ----- Packing efficiency ----- Packer Version Pack time File size Speed [seconds] [bytes] [bytes/s] xpkMASH: 1.98 43.56s 181792 9150 xpkCBR1: 1.2 25.34s 336400 15750 xpkCRM2: 1.1 Library cannot be opened xpkACCA: 1.0 25.88s 228472 15450 xpkCRMS: 1.1 Library cannot be opened xpkLHLB: 1.0 81.00s 188620 4900 xpkSQSH: 1.10 46.74s 204340 8550 xpkSMPL: 1.0 27.08s 335356 14750 xpkSHRI: 2.0 87.78s 169400 4550 xpkRLEN: 1.1 26.64s 329140 15000 xpkRDCN: 3.3 25.80s 229836 15450 xpkNUKE: 1.0 33.52s 192960 11900 xpkNONE: 1.0 24.64s 413128 16200 xpkIMPL: 1.0.79 174.26s 191792 2250 xpkIDEA: 1.99 Cryptor, no specs determined xpkHUFF: 0.63 Functions now, no specs determined xpkHFMN: 1.28 26.16s 289696 15250 xpkFEAL: 1.03 Cryptor, no specs determined xpkFAST: 1.06 38.22s 220156 10450 xpkENCO: 1.0 Cryptor, no specs determined xpkDLTA: 0.1 24.84s 412932 16050 xpkDHUF: 0.58 25.00s 412980 15950 xpkCBR0: 1.0 25.32s 336400 15750 xpkBLZW: 3.0 27.18s 242384 14700 xpkRAKE: 1.5 28.60s 190484 13950 Graphical representation of packing efficiency (incomplete) Ratio unpacked/packed [%] ^ | + SHRI | | + LHLB + MASH 200 + + IMPL + NUKE + RAKE | + SQSH | + FAST ACCA, | + RDCN | + BLZW + | | + HFMN | + RLEN | SMPL + + CBR1 100 + | + DHUF, | DLTA, | CBR1, | NONE 50 +---------+---------+---------+---------+--------+----------+--------> 0 5000 10000 15000 Speed [bytes/s] Following the graph, the best packer for me is RAKE when I read from a floppy drive, because gain*bytes has a maximum value for this packer. If I wanted to read from a fast device (RAD:, FFx:, DHx:), one of the packers more to the left (e.g. MASH or SHRI) may be better because the time for reading the disk will become shorter. The more to the left the entries are here, the more they will be shifted to the right in this case and the speed will become less important. Unpack speed is not considered here, this may also affect your (and my) choice. Experiment with SHRI, MASH, NUKE, RAKE, RDCN or ACCA, they seem to be most effective. vi) Compatibility This program needs OS V2.0+ to run. If you want to use xpk packers (very likely), you need the xpkmaster.library and some packer sublibraries (I suggest SHRI, MASH, NUKE, RAKE, RDCN or ACCA). The xpk package and some more sublibraries should be present in any good pd mailbox and in the Aminet. They are not included here because it's much larger than the PackDev package. vii) Bugs If the disk should be formatted, the track size of the disk (e.g. 11 blocks for a floppy disk) must currently be equal to the track size of the disk from which the file is read. This is not necessary, but coding the thing this way is easier and faster. Should you detect a bug, please tell me (email or phone). Be as specific as you can. viii) Future What may be done in the future: GUI (yes, really, I will do it...tomorrow :-)) Device that treats an archive like a disk Built-in packer What will not be done in the future: DMS compatibility (see below) Localization (I hate those zillions of useless files, there is no support for people who cannot speak English. This sounds arrogant, but I think, this tool should not be used by inexperienced users anyway) Versions for each type of processor (I made the experience that doing this causes a negligible speedup that is not worth even writing this sentence, but perhaps there will be a C compiler that can do better...) ix) My personal opinion Note that PackDev will never support the DMS packing algorithm, because the current authors of DMS (ParCon) use stolen code from the original DMS (which was written by someone else who has not given up his copyrights) and release DMS as shareware. In my opinion, they are not allowed to do so. It's not enough to assume that you can do with copyrighted material what you want, only because you cannot contact the real authors. It would be OK to release DMS in the public domain, but making money with someone else's code is called FRAUD and THEFT. It's worse than piracy - shame on you, ParCon !!! Note that you may own pirate software if you bought a "registered" version of DMS that has a version greater than 1.11. An interesting consequence: It may not matter if you use a cracked version of DMS or a "registered" version. If ParCon doesn't have the copyrights for DMS, they cannot sue for cracking it. I've seen mailboxes that put cracked versions of DMS into their PD area for this reason. x) How to contact me Christian Wasner Phone ++49-40-7236349 Email: u241045@niesel.dkrz.d400.de CRISI@BLACKBOX.SHNET.ORG If possible, use email. If you phone me, please do it from 8pm to 10pm and don't forget that I have Mean European Time here, so 8pm for you may not mean 8pm for me ! xi) History Aug-14 1994 V1.0 - Initial release, never released I think... Aug-18 1994 V1.1 - Minor bugs fixed Apr-16 1995 V1.2 - Problems with OFS disks fixed (PackDev didn't know the number of free/used blocks) - ALL, NOVERBOSE, QUIET and NOCONFIRM keyword added - Doc file corrected and improved - Filesystem type is now read from block 0 instead of reading it from the DOS node, because the latter always contains DOS\0 for Amiga floppies - Minor bugfixes Apr-30 1995 V1.3 - If the partition with LIBS: on it is to be handled, PackDev could not open XPK (sub-)libraries, fixed - Minimum XPK buffer size corrected - TESTFILE parameter added - Checksums installed, in case an xpk packer doesn't keep them.. - Argument handling changed (you got me, Christian...) - BLOCKLIST parameter added - PASSWORD parameter added