Cloanto DirDiff v. 5.1


 1.         Introduction

 DirDiff is a powerful tool to scan, compare, synchronize and replicate
 files. It was originally designed for internal use at Cloanto to simplify
 the maintenance of software master disks and for backup management. Over
 the years, new features were implemented, the file comparison engine was
 enhanced and a Workbench interface was added.


 1.1        What can DirDiff do?

 DirDiff allows you to compare (and optionally update) two disks. Several
 options can be set to finely tune this process.

 DirDiff allows you to copy some files from hard disk to floppy, then use
 the floppy on another computer (for example at home), and from time to
 time automatically move the files which have changed to whichever medium
 contains the older version.

 DirDiff allows you to compare bootable disks against new versions of the
 Amiga Workbench disks. It can selectively compare only the files used in
 both disks, skipping icon files if desired. So you get only messages
 regarding the files that count. If you wish, you can upgrade the system
 files on your work disks in the same process.

 When copying tens, or hundreds of Mbytes from one device to another,
 errors are more likely to occur and pass undetected than with normal,
 interactive use. Using DirDiff, at Cloanto we discovered faulty I/O
 configurations which caused only one or two bytes out of 100 million to
 be corrupt. We now always use DirDiff after each step in the preparation
 of master disks, both for floppy disks and for CD-ROMs, to make
 absolutely sure that no errors have occurred. When it finds certain
 errors, DirDiff does an additional check to make sure that the storage
 medium gives consistent results. It even double-checks the RAM locations
 it uses.

 DirDiff is an invaluable tool to back-up a hard disk to a slower device,
 such as a magneto-optical disk, because it can copy only the files which
 have changed. Results can be achieved faster and in a less
 resource-consuming way. To make the process bullet-proof, the source
 directory can be locked, and differing files can be forced from hard disk
 to the optical disk even if they have an older date. Files from older
 backups which become unused can be deleted in the process. The combined
 use of these functions can deliver identical copies at a fraction of the
 time a standard "Copy" procedure would normally require.

 DirDiff can detect even subtle differences between two items, such as a
 directory renamed from "WBstartup" to "Wbstartup". Before DirDiff reports
 that two directories are identical, it checks not only the contents, but
 also the attribute bits, file comments, dates and exact names of all
 items.

 DirDiff can scan a directory or disk and give an electronic signature
 (for example "DD5-A89CE287"), derived from the contents of all files.
 This short code can then be used to refer to a particular disk or product
 release without detailing the entire contents. For example, two distant
 offices which want to verify if they have the same version of a disk can
 execute DirDiff separately, and then check the signature in a phone call
 taking only a few seconds. Or the signature can be used as a reference in
 a licensing agreement, as a confirmation of what is being licensed.

 DirDiff includes additional checksum capabilities, and features such as
 clearing of unused blocks on a device (useful when you do not want others
 to "undelete" data on a medium which you considered safe to give away).


 2.         The user interface

 DirDiff can work in two modes: single directory mode and two-directory
 mode. The first mode is mainly a scanning mode. It can be used to check
 the integrity of files (all data is read from beginning to end), and to
 calculate file checksums and a final signature based on the total data.

 When working with two directories, DirDiff can perform complex comparison
 and update tasks, while remaining very easy to control. For certain tasks
 (as in the "Lock", "Maximum Priority" and "Delete" modes), the first
 directory is considered the primary, or source directory, and the second
 directory is the destination. The first directory can be locked, so that
 data only flows in one direction. The first directory can also be
 assigned "maximum priority", so that data is transferred from the first
 directory to the second even when files in the first directory appear to
 be older than those in the second. Existing files in the second directory
 can optionally be deleted, if they do not exist in the first directory.

 DirDiff can be executed from Workbench or from the Shell. In Workbench
 mode, all options can be set from the program window. Menus allow to do
 additional things such as saving the current settings. The window has a
 status line indicating the current file or action, plus an area showing
 either the settings or the results.

 In Shell mode, a help list containing the keywords associated to the
 program settings is displayed whenever the program is executed without
 parameters (that is, just entering "DirDiff" in the Shell). The execution
 results are printed in the Shell, and upon termination DirDiff exits with
 an AmigaDOS return code ("OK", if no errors or unresolved differences
 were found, or "WARN" otherwise), which can be used for conditional
 branches when DirDiff is executed from scripts.


 2.1        Window Gadgets

 The window displayed by DirDiff incorporates standard Amiga gadgets. The
 Window Close gadget on the upper left corner is equivalent to the Quit
 gadget. The Zoom gadget, on the right, allows the user to quickly expand
 the window size to the maximum possible size. The Depth gadget is located
 on the window title bar next to the Zoom gadget. It is used to bring the
 window to the front or back with respect to other windows. DirDiff
 automatically pops the window to the front of other windows when it
 finishes a job.

 The bottom of the window contains five gadgets: Run, Pause, Abort,
 Iconify and Quit. Run instructs DirDiff to begin a new processing cycle.
 Pause temporarily halts processing, which can be resumed by pressing the
 same gadget again. Abort concludes processing on the current file and
 suspends further processing. Iconify closes the window (but does not
 interrupt processing). A "DirDiff" icon appears on the Workbench to
 indicate the iconified state. A double-click on the icon reopens the
 window. The Quit gadget closes the window and terminates the program
 (aborting any processing in progress, if necessary).


 2.2        The Mode Gadget

 This gadget toggles between single and dual directory mode. By default
 DirDiff works with two directories. If the same directory is indicated
 twice, DirDiff switches to single-directory mode.


 2.3        The Path Gadgets

 These two text gadgets specify the working directories. Icons can be
 dragged from the Workbench screen and dropped over the gadgets. The small
 gadget to the right of the string gadgets activates a file requester
 which can be used to select the path.

 Note: If one of the two devices is much slower than the other, or if it
 has considerably fewer files, and if the processing involves a simple
 comparison and no updates, then DirDiff is faster if the directory with
 fewer files, or a slower device, is indicated as the first directory.
 This is because in the main phase the second directory has to be searched
 for all files found in the first one, and this occurs in an order
 arranged after the first directory.


 2.3        The Page Gadget

 The central area of the program window always contains one of three
 pages: the Settings Page, the Output Page or the Status Page. The Page
 gadget allows the user to cycle through these pages.


 2.4        The Settings Page

 This page contains all options which can be set by the user. A particular
 configuration of options can be saved from the program menus.


 2.4.1      Dates

 By default, whenever DirDiff finds the same file or directory in both the
 first and in the second directory, it also compares the dates. On the
 standard Amiga filing system this date indicates the date of the last
 write to the file or directory (although directory dates are not always
 up-to-date).

 If DirDiff if used to compare two directories, and when it is known in
 advance that the files in a directory had their dates modified during an
 incomplete copying process, this option could be disabled in order to
 leave the list of differences more readable, without excessive messages.

 The date associated by AmigaDOS to each directory entry is accurate to
 1/50 of a second (although fractions of a second are not indicated by the
 Amiga Workbench, nor by the Shell's List command). Some filing systems
 (e.g. MS-DOS, or ISO 9660 as used for CD-ROMs) and archiving software
 (e.g. LhA) store dates with a slightly different precision than the Amiga
 filing system. So it may happen that when a file is copied to a device
 employing a different filing system, the date of the copy shows a small
 difference (usually less than a second).

 The "Dates ± 2 sec" option instructs DirDiff to ignore small differences
 in the file dates, within a range of ± 2 seconds. In Update mode, it also
 informs DirDiff that file dates are not reliable within the ± 2 second
 range.

 When this option is disabled, dates are neither compared nor updated,
 unless a file or directory is being updated for a different reason.

 Note: It is recommended to always preserve the original file date when
 copying files. This is a piece of information which can give precious
 hints about the history and the current status of the file. In case of
 updates, DirDiff may use these dates to determine which item is older and
 should be overwritten. Unfortunately, by default the Amiga Copy command
 of the Shell changes the date of the destination file. To avoid this,
 Copy should be followed by the CLONE option. At Cloanto, we use a "cp"
 command, which is defined as follows in the S:Shell-Startup file:

     Alias cp Copy [] CLONE


 2.4.2      Attributes

 Just as it compares the dates of items found in both directories, DirDiff
 also compares attributes such as the comment associated to that item and
 its status bits (see the Filenote and Protect AmigaDOS commands). Under
 AmigaDOS, matching files could even have slightly different file names,
 because capital and lower case letters are treated as if they were
 equal.

 Some or all of these additional attributes could be altered or even lost
 when files are archived or moved to other filing systems. For example,
 most Amiga implementations of the ISO-9660 CD-ROM filing system do not
 represent the original Amiga file comments and attribute bits. In a
 conversion to the MS-DOS filing system, it is possible that all file
 names are automatically converted to capitals.

 When this option is disabled, attributes (comments, attribute bits and
 exact names) are neither compared nor updated, unless a file or directory
 is being treated as different for another reason.


 2.4.3      Icons

 DirDiff normally also processes the ".info" files which contain
 information about the Amiga Workbench icons associated to files and
 directories.

 This option allows to leave out icon files from this process. This could
 be useful, for example if two disks are known to have similar contents,
 but differing icon positions.


 2.4.4      Quick

 On a reliable medium, it is normally safe to consider that two files
 having the same name, length and date are identical. The Quick option
 allows DirDiff to make this assumption, so that the contents of files
 having the same name, size and date is not analyzed. This is much faster
 than scanning two files from beginning to end. When the dates differ,
 DirDiff scans the files automatically, even if they have the same name
 and length.

 It is recommended to disable the Quick option whenever the reliability of
 a medium or of a copying process is questionable. This causes all files
 to be byte-scanned and compared from beginning to end.


 2.4.5      Couples Only

 This option restricts the set of items being processed to those which
 exist in both the first and in the second directory. This is useful
 whenever a medium contains a base set of files which has to be checked
 against an original, plus some additional items which do not need to be
 checked or updated.

 Activating this option could be useful, for example, to compare a new
 Workbench disk with the Workbench installed on the computer's hard disk.
 In this case, both the hard disk and the Workbench disk could contain
 items which do not exist in the other directory, and which are not
 relevant for a quick difference check.


 2.4.6      Update

 This option controls the part of DirDiff that not only reads, but also
 writes to the directories being processed. When Update is disabled, no
 writes occur. When Update is active, DirDiff will take actions in order
 to resolve the differences between the two directories. These could be:
 copying the entire file and/or setting the date and other attributes.

 Disabling the Update option is also useful to execute a test run of
 DirDiff to see which files would be affected if Update were enabled. If
 the results are acceptable, the Update option can be enabled to run
 DirDiff a second time.


 2.4.7      Lock Directory 1

 This option acts as a "valve", limiting the direction in which the update
 process is allowed to write data. It is normally a reassuring option to
 use when using DirDiff to make backups. In this case, the source
 directory (containing the data to be copied) should be indicated as the
 first directory.

 Regardless of the state of this option, older files are never copied over
 newer ones. This is only possible by activating the "Maximum Priority"
 option.


 2.4.8      Maximum Priority

 This is a potentially "dangerous" option. It forces data to be copied
 from the first directory to the second one whenever two items are
 different, even if this means copying an older item over one having a
 more recent date.

 The Maximum Priority option only causes file to be copied from the first
 directory to the second, and not vice versa. From this point of view, it
 is similar to "Lock Directory 1", only that it is considerably more
 "active".

 Maximum Priority is very useful to restore a directory regardless of the
 file dates, and to make sure that all items in the second directory are
 identical to those in the first one.


 2.4.9      Delete in Directory 2

 This is another potentially "dangerous" option. It extends the update
 process by deleting those files or directories in the second directory
 which do not exist in the first directory.

 This option is normally used together with Maximum Priority. It could be
 employed, for example, to maintain a backup copy of a hard disk
 up-to-date, removing those files which have been removed (or moved, or
 renamed) in the source since the previous backup. The result is that the
 two directories are forced to be identical.


 2.4.10     Fill Directory

 This function can be activated in single-directory mode. It causes empty
 blocks on the storage device to be cleared.

 Normally, when files are deleted it is still possible to recover some
 information with low-level disk tools. When delivering material to third
 parties, this potential security risk can be prevented either by
 formatting a new disk and copying the files to it, or by using a function
 like DirDiff's "Fill Directory", which fills the directory with a file as
 large as allowed by the existing space, and then deletes that file.


 2.5        The Output Page

 This page displays the scrollable output list of DirDiff, which contains
 entries for the directories and files processed, plus errors and other
 messages in chronological order, followed by the final results as
 displayed in the Status Page. The list can be saved to file, printed or
 cleared (useful in case of memory shortage) by selecting the
 corresponding gadgets.

 The output, just as processing, occurs in two phases (in two-directory
 mode). The first phase does a preliminary check to see if the second
 directory contains any items which are not present in the first directory
 (in Delete mode, these are immediately deleted). The second phase scans
 all items in the first directory, comparing them against the second
 directory.

 In single-directory mode, DirDiff lists the size and checksum of each
 file. The checksum is a simple sum of all bytes in the file. At the end
 of the process an electronic signature is also printed. This has the
 format "XXX-YYYYYYYY", where XXX stands for the program version, and the
 Y digits are the actual signature. This is not a simple checksum. A
 change in a single attribute bit of one of hundreds of files can
 completely change the signature.

 Note: The signature method employed by DirDiff does not employ encryption
 technology, thus it does not pretend to offer any protection against the
 possibility that a set of source files can be artificially prepared to
 produce a given signature. By being only 8 characters long (32 bit) the
 signature leaves open the theoretical possibility that two different set
 of files produce the same signature.

 The comparison of two directories produces an output showing the
 differences found and the action being taken, if any. The possible
 differences include: non-existence of an item or differences in contents,
 name, attribute bits, comments and date. When DirDiff finds matching
 items with different names (upper/lower case differences) it shows both
 names. When it finds files with a different content but the same length,
 it shows the offset in which the first difference was found.

 To resolve the differences (in Update mode), DirDiff can take one of the
 following actions, on a case by case basis: copy the entire file, update
 (copy newer over older) or force (copy older over newer) the parts which
 are different, delete the item (directory or file), create a directory.

 At the end of a directory comparison process (two directories), DirDiff
 prints final result, indicating whether it found any errors or
 differences that could not be resolved. If the settings allowed for all
 attributes of all files to be checked, and no differences were found,
 then DirDiff will state that it has found the directories to be
 identical.


 2.6        The Status Page

 This page contains information on the number of files and directories
 which have been processed, updated, created or deleted, and on the total
 number of errors or unresolved differences. This information is updated
 in real time as processing occurs.

 It should be noted that the counters indicating the total items being
 processed are increased once for each item and for each separate access.
 They are not indicators of the numbers of files and directories present
 on the disk. If an item exists in both directories, it is accessed at
 least once in each directory, thus counting as two. Plus, the counters
 are also increased in the first phase.


 2.7        Menus

 The items in the Project menu are standard Amiga application items,
 equivalent to the corresponding gadgets in the window of DirDiff.

 The settings menu allows the user to save and reuse particular
 configurations of settings, in a way similar to the standard menus found
 in the Amiga system Preferences.

 The Save as Defaults command makes the current settings the default
 startup settings.

 Last Saved reapplies the settings which were last saved, whereas Restore
 resets all values to the initial program settings.

 If the Create Icons option is enabled, DirDiff will associate a Workbench
 icon to all settings files written. A double-click on the icon loads
 DirDiff with the settings stored in the selected file.


 2.8        Examples

 The following examples show typical applications of DirDiff, indicating
 which options have to be set in a position different than the default.


 To show missing or different files and directories:

 - Default configuration


 To verify a disk after a copy or optimization:

 - Quick option disabled


 To verify a CD-ROM ISO image before mastering:

 - Dates set to "± 2 sec", Quick and Attributes options disabled


 To compare Workbench files on an application disk:

 - Couples Only option active


 To synchronize hard disk and floppy files:

 - Update and Couples Only options active


 To make or refresh a backup copy (keeping history of "orphan" files):

 - Update and Lock Directory 1 options active


 To make or refresh a backup copy (deleting "orphan" files):

 - Update, Lock Directory 1 and Delete options active


 To make or refresh an exact copy:

 - Update, Maximum Priority and Delete options active


 To check file integrity, show checksums and produce final signature:

 - Single directory mode


 To clear unused blocks on a disk:

 - Single directory mode, Fill Directory option active


 2.9       Program Messages

 Most error messages displayed by DirDiff are self-explanatory. For
 example, simple AmigaDOS errors may occur when accessing (locking,
 opening, examining, deleting, creating, renaming, copying or writing) a
 file or directory. This may occur if an update exceeds the storage
 capacity of the disk, for example. For most messages, an AmigaDOS error
 code is also indicated. "Not enough memory" messages indicate that there
 is insufficient free RAM.

 The following is a list of other messages, in alphabetical order.


 Data length different from file length ...

 This message indicates that the number of bytes which DirDiff was able to
 read from a file was different than the data associated to the file in
 its directory entry. This could be caused by a problem in the filing
 system structure (or also - but not necessarily - by a virus program
 inserting some hidden link code into executable files). At Cloanto we
 noticed this discrepancy to be quite common when processing other
 computers' filing systems on the Amiga.


 Data OK after retry on medium (possible random match)

 After discovering a difference in the contents of two files having the
 same size, DirDiff read the location showing the difference a second
 time, and found no difference between the files. The fact that two reads
 of the same data produced two different results is likely to point to a
 medium or I/O error, while the fact that the second read matched the
 contents of the other file could be caused either by the error being
 (temporarily) recovered, or by another error which caused the results to
 match by pure chance.


 Dates within ± 2 sec range - No update possible

 DirDiff is operating in Update mode, but encountered corresponding
 entries in each directory which required an update, but had dates too
 close to indicate with precision which item is newer. This only happens
 if Dates is set to "± 2 sec".


 Directory entries are of different types

 DirDiff encountered two entries (one in each directory) having the same
 name, but different types, presumably one being a directory and one a
 file. The two entries are skipped, as they can neither be compared, nor
 updated.


 Identical dates - No update possible

 DirDiff is operating in Update mode, but encountered corresponding
 entries in each directory which required an update, but had identical
 dates, making it impossible to determine which file is newer.


 Indirect date change...

 As a side effect of DirDiff updating the files in a directory, the filing
 system "touched" the date of the directory. DirDiff detects this and
 restores the original date.


 Internal error (...)

 These are internal errors which "should not occur". Reporting the error
 condition to Cloanto would be helpful to correct the problem.


 Medium Error (unconfirmed read on ...)

 Data consistency checks are performed routinely by DirDiff after certain
 errors occur, or when unlikely file differences are detected (as in files
 having the same name and length, but different contents). This message
 indicates that DirDiff obtained results which were different from the
 previous read. This could be caused by defects in the medium or in the
 I/O system.


 RAM error
 RAM inconsistency

 DirDiff checks the RAM locations it uses to store its file comparison
 buffers. These messages indicate that an error occurred during one these
 tests. An "inconsistency" is an error which could not be confirmed by a
 second test.


 Read beyond end of file

 This message indicates that DirDiff was able to read more data (before
 reaching the end of file) than the file length information in the
 directory entry indicated. This message is similar to "Data length !=
 File length", only that it occurs in a different context.


 For additional information on this distribution of DirDiff please refer
 to the "DirDiff.readme" text, also in the archive containing this file.