Short: Synchronise two folders automagically! Author: Chris S Handley (cshandley-REMOVETHIS@ANDTHIS_iee.org) Uploader: Chris S Handley (cshandley-REMOVETHIS@ANDTHIS_iee.org) Version: v1.02a Type: util/dir The purpose of FolderSync is to allow one set of files to be shared by two separate people (computers), and occasionally update BOTH people's set of files with the changes that the other has done. Neither person has priority, because FolderSync just 'mirrors' what one person has done on to the other's files, as if both were working on just a single set of files. It does NOT run in the background, but is run by the user when needed. Synchronisation is done at the level of whole files, rather than somehow updating a part of a file (as might happen with Address Book sync s/w). It can sync between an Amiga disk & a FAT (Windows) disk quite happily. I wrote FolderSync to allow me to carry 10,000 of my Amiga's files (500Mb) around with me on my Psion 7 (a 'minature laptop'), without having to worry about remembering what changes I had done on either my Psion or my Amiga, and without having to synchronise every time I switched between Amiga or Psion. It can also be made to backup the changes in one folder to another, which I use to keep an up-to-date backup of my Workench partition (in case I do something silly, like formatting it). Having spent so much effort making FolderSync very fast & reliable, I though I ought to make it publically available :-) I am not aware of any other program that does anything vagely similar, and if there is, then I doubt it is written so well! I am quite proud of how well it works, given the apparent lack of support by filing systems. Note that it is a CLI (Shell) program, and should probably only be used by an 'expert' (someone with some experience of the technical side of computers - which probably covers a lot of the surviving Amiga users!). I have a fairly seriously updated FolderSync, but this will not be made available until I have done a few other things. There is also less motivation since no-one seems to be using FolderSync... Serious Warning +++++++++++++++ If used without care or understanding, FolderSync is a very dangerous program, because it could POTENTIALLY delete or change every single file on your harddrive. The main risk is from YOU using it wrongly, but there is of course a very small chance of a bug. Take this seriously, because I will not be held responsible in any way for ANY loss of files or data from the use of FolderSync. The program is provided for free, and you use it entirely at your own risk. No warranty of merchantability or fitness for a particular purpose is expressed or implied. I personally recommend you do a complete backup before you use FolderSync for the first time, and after that it is up to you to take any other precautions. Computers should be backedup regularly anyway, so this shouldn't be too much of an additional burden. Make sure you fully read & understand this document before using FolderSync. Anything else I say does not invalidate or modify the above warnings! Comments on my Warning ++++++++++++++++++++++ Because I do not want to loose my own precious data, I have gone out of my way to make FolderSync as safe as possible. However, for FolderSync to be useful, it needs to work with little user interaction, and this gives the possibility of it doing something you didn't intend. Specific features which try to minimise the above risks are: 1. The algorithm used will only delete a file or directory if it was seen in the previous synchronisation, and no longer exists in one of the two folders. It remembers the files it has seen, using one special file, which means you would have to do something very strange to make it wrongly delete a file: Deleting this special file will not cause serious problems, and randomly corrupting the file is incredibly unlikely to make it delete anything. 2. The mentioned special file is sanity checked at every synchronisation, so corruption or bugs should not be propogated to further synchronisations. 3. A "trash folder" holds any files that are 'deleted' or updated. But only one file of any name may exist (no sub-folders are used), and eventually I hope to make it automatically delete old files to prevent the trash folder becoming too large. So don't rely on the trash folder, but it will hopefully allow you to mostly undo any mistakes. 4. There is a user-definable "safety limit" of the number of destructive file operations (delete & update), above which it will ask before doing anything. This limit defaults to 1000 files. If the number of destructive operations sounds much larger than you would expect, then I recommend you make a backup before proceeding, as it will not tell you what operations it will perform. I may improve this. 5. Every slightly important file & directory operation is checked for success, and problems will be reported. It will only abort during synchronisation if there is a major error (such as out of memory), or the user presses "Ctrl C". 6. It report useful information at a definable level of detail, so you have some idea of what it has done. Hence, if things go wrong, you will know as early as possible, giving maximum opportunity to undo anything. 7. It has been designed very carefully, written in a professional programing language (AmigaE), and tested reasonably well. In other words, it is NOT a messy nightmare with hundreds of wierd special cases that are only sometimes considered, and isn't written in a language which makes testing & modern design methods difficult or impossible. If you don't want to trust my program, you can ask for the source code, once I decide how to release it (probably GPL v2). Discussion of FolderSync's behaviour ++++++++++++++++++++++++++++++++++++ The words "folder", "directory" & "dir" all refer to the same thing. Although in practice using FolderSync is simple, I can't tell you which one or two small things you need to remember for it to work as expected in your situation. So I must tell you everything, and let you work it out! :-) I.E. FolderSync could be used in many situations in many different ways, because files & directories themselves can be very varied & used in many ways. This means a good understanding of FolderSync's behaviour is necessary, at least if you are to trust FolderSync. This program will compare the contents of two separate folders (& their sub-folders), and the state of the folders that it recorded after the last synchronisation. It will then deduce what changes have occured since then (as best as it can), before finally attempting to make those changes in the mirror folder. Because computer filing systems were not designed with a program like FolderSync in mind, in relatively complex situations where the same files or folders are changed in BOTH folders (between uses of FolderSync), it is not always possible to correctly deduce what the user intended. However, I think such situations are unlikely for normal use, and at worst you are more likely to end up with two copies of the same file, or possibly an older file, rather than a file being deleted. A problem is much more likely if you are synchronising folders on two DIFFERENT filing systems, such a the Amiga's FFS & Window's FAT. In this case, both filing systems have different limitations, and the set of files being synchronised must usually stay withing the limits of BOTH filing systems. I have designed FolderSync with expectation of possibly being used with a long-filename FAT filing system (via the freely available FAT95 v2.17+ software), so the problems with FAT are minor. The only exception I am aware of is that some special Amiga characters are illegal in filenames under Windows (although FAT95 allows them); the simple solution is: After the first synchronisation, you should run a Windows program to correct any errors on the disk, and if any are found, then to rename the relevant files on the Amiga side. Be aware that changes made by the Windows program will be interpreted by FolderSync as if you had made those changes, so deleting a file with a 'corrupt' name will make FolderSync try to do so on the mirror file. There are various solutions, although a simple but hacky one is to delete FolderSync's special file (called "lastsync"), which will cause it to copy any files not found in the other folder, rather than try to delete or rename files. In any case, if the files you are synchronising are valuable to you, then you should understand the next section, so you are aware of unavoidable oddities in how FolderSync operates. Specific details on FolderSync's operation ++++++++++++++++++++++++++++++++++++++++++ The synchonisation of the two folders is BI-DIRECTIONAL, i.e. changes in either or both folders will appear in each other. But try to avoid changing the same file in both folders, because then it MAY be complicated for you to know what FolderSync will decide (although it's decision will be absolutely _logically_ the best given what it knows). The first time two folders are synchronised, IDEALLY the folders should not contain the same files or dirs, although it should be ok. If using different filing systems, then it is highly RECOMMENDED, since there could be some wierd conflicts. The best situation is to start by initially synchronising one folder with an empty folder, as then FolderSync has full control over the mirror files. The contents of files are only updated based on their time stamp. But by the use of heavy magic, FolderSync tries to detect changes to a file's protection bits, comment, or even filename CASE - despite the fact that changes to these aspects of a file do not cause the time stamp to be changed. Correcting changes that aren't notified by a time stamp is called "tweaking". It will even try to avoid synchronising file comments & protections bits if they are not supported by one of the filing systems (this is only known to work with FAT95). This means you will not loose Amiga comments or SRE protection bits if you synchronise an Amiga folder with a one on a FAT disk. Dir entries (not their contents!) are also synchronised, but their timestamp is ignored because this is not usually handled usefully by the OS. I.E. Dirs are only tweaked. Files & dirs that are created or deleted will have the same done to their mirror. Created ones are exact clones (except for a dir's time stamp). Files & dirs that are renamed or moved make the decision harder for FolderSync, because then the main way of identifying a file (it's name & path) has been lost (filing systems were not designed with FolderSync in mind). It will still work, but you are more likely to loose Amiga filing system specific information (such as comment or the SRE protection bits) if the other folder is on a foreign filing system like FAT. File or dir names that are longer than 30 characters (an Amiga OS legacy limitation) will be semi-intelligently renamed shorter. You will be told when this happens, unless you use the TERSE switch. You wouldn't believe how much of a bother this was to code! 'Deleted' files are put in the user-specified trash folder; their original directory is put in their file comment. Currently the file's time stamp is not changed, but when a fixed-size trash folder is implemented, it will be changed to the time of deletion. Currently the trash folder will grow in size forever, unless you manually delete files in it. Directories with their P-bit set ("pure") will be ignored; effectively it is a hidden bit used soley for FolderSync. P maps to the System bit under FAT95. This allows you to flexibly synchronise only certain sub-folders to the mirror folder. For example, I only keep a sub-set of my Amiga folders on my Psion 7. There may be a case for using the S-bit ("script") instead of the the P-bit, as Windows uses the P-bit ("System") for a few directories... Try not to create a directory that is a mirror of one with the P-bit set, because FolderSync will not know about the P-bit one; it won't do anything bad, but it will likely report quite a few warnings & notes! Read EVERYTHING FolderSync outputs, because it only outputs important stuff! Usage of FolderSync +++++++++++++++++++ Finally, I am going to tell you how to use FolderSync (!), now that you understand enough to hopefully use it without getting unexpected problems because of any oddities of your particular situation & how you wish to use it. It's CLI arguments are: FolderA/A, FolderP/A, TrashFolderA/A, TrashFolderP/A, LastSyncFileInFolderP/S, BackupFolderP/S, SafetyLimit/N, Terse/S, Verbose/S A brief description of each argument: FolderA = One of the folders to be synchronised; given the general name "A". FolderP = The other folder to be synchronised; given the general name "P". TrashFolderA = The trash folder for "A", which must be on the same volume. TrashFolderP = The trash folder for "P", which must be on the same volume. LastSyncFileInFolderP = Put special file "lastsync" in "P" rather than "A". BackupFolderP = Do one-way synchronisation, only changing folder "A". SafetyLimit = How many destructive file operations allowed without asking. Terse = Only output warnings. Verbose = Output every file & dir operation performed. Usage examples: 1.Maybe you want to share some Amiga files with your Windows partition? FolderSync Data:StuffToSync Windows:AmigaStuff Data:StuffToSync/Trashcan Windows:Recycled VERBOSE SAFETYLIMIT=90 2.Maybe you want to have an up-to-date backup of your workbench partition on a spare harddrive? FolderSync WbBackUp: Workbench: WbBackup:Trashcan Workbench:Trashcan BACKUPFOLDERP Additional details on the arguments: Although folders "A" & "P" derive their name from Amiga & Psion, there is infact nothing special about either folder. You could interchange them, although as FolderSync stores it's special file in the "A" folder (by default), it won't find this file unless you moved it to the other folder also (or used the switch LastSyncFileInFolderP). The special file is called "lastsync", and should not be deleted or moved (and certainly not edited) unless you know what you are doing. The Trash folders may be inside their respective "A" or "P" folder, without causing problems. This allows the "A" & "P" folders to be the whole volume if you wish. The backup mode enabled by the BackupFolderP switch will not generate or use the special file, because it is not needed. Any differences between the two folders will cause folder "A" to be changed to match, even if this means replacing a new file with an older one. This makes it a very safe way to maintain an up-to-date backup, as long as you don't accidentally swap the backup & source folders! The safety limit defaults to 1000 files. You may change it to anything from 0 to just over 2 billion. I personally would not use the Terse switch, as you may not see useful (but probably non-urgent) information. But you can if you want. I would only use the Verbose switch if you don't have many files to synchronise, and don't completely trust FolderSync yet. If it synchronised 1000 files, then it would need to output at least 1000 lines of text, which would be very slow indeed! You will notice that when it reports a filename or directory, that the volume name will will have a slash after the colon. This is not a legal Amiga path, and the slash must be removed. This is due to the use of my portable file & directory classes, which represents paths a little differently to the Amiga. I could translate the paths back to Amiga format, but it would be rather tedious, obsfucate the source code a lot, and be rather unnecessary IMO. In theory FolderSync should be ideal for use with the AmigaNCP filesystem, because it minimises the filing system calls, however the last (beta) version I tried still had a couple of problems that stopped it being usable. I am hoping these may be fixed, but this is up to the author (who no longer even has a Psion to do tests with). Upgrade Procedure +++++++++++++++++ If you have used an older version of FolderSync for synchronisation (not backup), and the "lastsync" file format has changed, then you will need to follow this upgrade procedure. If you do not, FolderSync will tell you to! 1.Synchronise using the old version of FolderSync. 2.Delete the special "lastsync" file. 3.Synchronise using the new version of FolderSync. This time FolderSync should not make any changes (if you have followed the instruction correctly). History +++++++ v1.02a (30.07.2002) - Added anti-spam stuff to email address :( v1.02 (11.04.2002) - First Aminet release. - Minor tweak to the error handling system. - Forgot verbose report of dir creation. - Removed theoretical problem of possible hidden mirror duplicate filename when reducing a filename length. - Added reporting creation of dir that already exists but was hidden. - Added version (detection) to lastsync file. - Corrected FolderSync's executable version year date! - Added tweaking of name case, attributes & comments. - Added automatic detection of filesystem support for comments & protection bits, thus all protection bits are now supported. v1.01 (19.03.2002) - Minor corrections to the documentation. - Removed minor bug where copied FAT95's timestamp file comment. - Removed a bug which potentially could trash memory (but didn't seem to). - Replaced slow AmigaE memory & E-string functions with my own super fast ones, resulting in a speed up of about 70 times! v1.00 (19.02.2002) - First release. v0.00 (02.11.2001) - I first started seriously looking at programming it. The Future ++++++++++ If you want me to bother uploading any new versions of FolderSync (bug fixes, new features, improved efficiency, etc), then I suggest you tell me that you use FolderSync, so that I have some reason to upload! (no emails are the best way for me to think no-one uses my program...) I do not intend to add many more features, as I would rather be working on my next project. Therefore, don't expect me to be keen on adding major features! However, as I will eventually make the AmigaE source code available, you could theoretically add features yourself. Keep in mind the size of the source code, and the language used though! Current to do list: (no order) - Improve the algorithm, so it detects a rename or move, rather than performing an equivalent (but much slower) copy & delete. - Make the trash folder of fixed size (delete older files as necessary). - If the safety limit is exceeded, output the proposed changes to a text file in RAM:, so the user can better evaluate whether to allow FolderSync to proceed. - Fix bugs, *if* any are found. Things being thought about: - Using the S-bit instead of the P-bit to provide hidden directories. - A tiny program to undelete all files in the trash folder (to the correct location) might be nice. But you should never need it, so... Bugs ++++ None known. But if you find any bugs, I am keen to know, so that I can fix it! MAKE SURE YOU GIVE ENOUGH DETAILS FOR ME TO REPRODUCE THE PROBLEM MYSELF. Ideally you would provide a small LHA archive with the smallest set of files needed to needed to reproduce the problem. Please don't email me a file larger than 50Kb without asking first. Support +++++++ As my time is limited, I do not intend to give much support for FolderSync, above what is specified in Bugs & The Future. I expect that while the program is fairly new, there will be a few issues with my documentation. As such, I won't mind clarifying things, as this should mean fewer support emails to me overall :) However, going by my previous Aminet releases, I expect the feedback to be almost non-existant... Technical Notes +++++++++++++++ For those who are interested, and those who even dare to try to understand the source code, here are a few technical notes I thought were interesting (and maybe even useful!): I started writing it after I had thrown my Towered A1200 PPC in the roof, and replaced it by a lovely new 1500MHz PC with 512Mb RAM - but still running AmigaOS, thanks to Amithlon (part of the AmigaOS XL package). Thus I have no idea of how FolderSync performs on a 'real' Amiga, and would be interested to find out... It is written in the famous AmigaE (v3) language, and uses quite a bit of Object Orentated Programming (which AmigaE makes easy - who needs C++?). The main program (inc. a fast list class) is 2100 lines, and it uses a 2600 line set of classes which abstract file & dir operations in an OS portable manner (and may therefore be initially a little difficult for someone else to understand). (What is the use of OS portable file & dir classes when AmigaE onlys compiles for 68k Amigas? You will have to wait and see... :-) It uses a fast & robust sorted linked-list class (500 lines) to store lists of files & directories; these are used to process the files/dirs, and produce lists of files/dirs for different categories (such as those to be Deleted or Updated). By careful algorithm design, comparisions between two or more lists only takes O(n) time, whereas a naive implementation would take O(n^2) time. Thus FolderSync will scale well (by which I mean, it won't suddenly slow down horribly for large numbers of files). The fairly high complexity of FolderSync meant I came across some limitations in AmigaE: 1.The OstrCmp() function [for Ordered String Comparion] was broken when strings contained some wierd characters (which some of my files did!). I wrote my own replacement, and then re-wrote it when someone showed me a several times more efficient way. 2.Memory allocation & deallocation (especially by the E-string functions) was discovered to consume about 98% of the processing time (which could partly be hidden by slow disk access). This is not really AmigaE's fault, as it is only using the AmigaOS's own memory functions; this doesn't make AmigaOS look very good IMHO, although all OSes have this problem to some extent. Anyway, I wrote my own incredibly fast (and rather INefficient) memory (de)allocation routines, and then duplicated all of AmigaE's own E-string function so they use my new routines - and reduced processing time by about a factor of 70! Thus disk accessing is now the main delay in FolderSync, and I had already minimised this. This means that FolderSync is no-longer suitable for low-memory situations (if it ever was), although on-request I could compile a version without the new memory routines - it will be much slower for very large numbers of files, but memory usage should be at least 2-3 times less, or even more if you have a shallow folder structure. Note that I had written FolderSync for speed, and this meant caching lots of stuff in memory. I had not counted on the memory (de)allocation being so slow, although reducing the memory usage much would probably have resulted in a much greater slow-down overall. You may think 98% of processing time taken by memory (de)allocation is excessive, but this is probably because FolderSync is designed so well that it has very little processing to do! I remember reading that typical programs (probably on Unix) can waste (IIRC) 33% of their time on just memory (de)allocation - so my program is not much worse, and I think the AmigaOS may use a much slower memory (de)allocation algorithm than more modern OSes. Thats about all I can think to say at the moment. The Author & His Soap Box ;-) +++++++++++++++++++++++++ This is my email address: Chris S Handley I live in England, and have been an avid Amiga user since 1991 because of it's elegant design (it is quite simple to use, but powerful & very flexible). With the lack of development of many Amiga programs in recent years, my hardware being unreliable (esp. PPC), and the need for a more portable machine, I have been gradually moving less demanding stuff over to Psion machines. Finally, with Amithlon (part of AmigaOS XL) being released, I have reliable 'Amiga' (x86) hardware again, so I have less need of my Psions, but their portability & extreme ease of use are still very nice to have. With FolderSync, I don't have to choose between Amiga or Psion, and can use both without a thought :-) The design of Symbian OS (aka EPOC on Psions) is mostly unoriginal, but it is so well done that this doesn't matter. And it looks very likely to be the future standard OS for phones (& I'd bet PDAs too), which is *really* pissing off MS! Now, at the risk of offending the majority of computer users (who often don't know better;), I don't use any other OSes, simply because they are crap: Palms are so simple, they are little more than toys (or should that be address books?). Using them to do more than they were designed is just asking for problems. Multitasking? WHAT multitasking?! Think MS-DOS's 640Kb limit is bad - wait till you see PalmOS's limit! PocketPC (WinCE) machines are powerful, unfortunately their user interface is a complete mess, and the OS's design makes them inherantely unreliable - which again relegates them to (expensive) toys. And I need a decent keyboard... Linux is a nice idea, but above the kernel it is a total design mess, and being based on Unix only makes things worse. Also, being open source is a curse, often with at least two 'standards' for doing anything, so that people spend as much time trying to glue stuff together (or arguing over standards) as doing anything useful. I avoid knowing too much about Linux, and so won't be drawn into useless technical details! Windows itself is obviously incredibly powerful. But although it has one design direction (rather than being splintered into factions), MS has insisted on always being backwards compatible (evolution not revolution) & adding marketable features rather than solving real problems, so it is as big a design mess as Linux, but for different reasons. Thus doing anything complex usually causes it to fall over, not to mention user-unfriendly in reality, so I avoid it where ever possible. There are other commercial OSes, but nothing else that is a serious general-purpose computer platform which any chance of general success. Note: AmigaDE is not a general-purpose computer, yet, although it may be eventually. But there are a LOT of research OSes out there that show all current commercial offerings (except maybe BeOS...) are basically crap (even the Amiga): For a start, we should all be using exokernels with SISR protection, a unified address space, and true capability based fine-grain security. Then, maybe, a resiliant persistant file-less system, dynamic portable code generation & transparent network thread migration... Of course, these opinions are my own, but I see no need to append IMHO to every opinion I state, as this should be obvious! :-)