URLUpdat V1.01.19980816. © 1998 Michael J. Maloney BSc (Hons.). Date Sunday 16th August 1998. Syntax: URLUpdat COMMAND FILE_SOURCE_DIRECTORY FILE_DESTINATION_DIRECTORY URL_INPUT_FILE URL_OUTPUT_FILE URL_MODIFIERS_FILE BEGIN_HOURS BEGIN_MINUTES BEGIN_SECONDS PAUSE_SECONDS FAILED_URLS_FILE MOVE_FILES_SHELL_SCRIPT This program will not download files directly from the Internet. It scans a list in ASCII format containing the Universal Resource Location (Internet address) of all the files that you would normally download over a given period of time, moves the relevant downloaded files to a different location altering their filenames and datestamps to user specified values, before updating the original URL list. The updated list of URLs can then be passed to whichever download utility you are using so that it can download the next batch of files from the Internet. The download utility that I use for this purpose is FileHound on the PC, available from 'http://www.windows95.com/apps/98/utilities.html' (or thereabouts at the time of writing). It is shareware, small, simple to use, and will not cripple you if you do not register it. A similar Amiga program should be available from Aminet 'http://www/cucug.org/aminet.html'. The operation of URLUpdat can be best explained by covering the arguments used in its operation: COMMAND e.g. u URLUpdat currently operates in two modes. u - Update mode. Updates URLs and downloaded files as described in blurb. e - Expand mode. If you wish to download a lot of sequential files with similar names you only have to enter the first URL. URLUpdat can force this single URL to undergo inflation, generating an 'expanded' download batch list. E.g. Suppose that you would like to download all the files between 'http://www.files.com/file001.txt' and 'http://www.files.com/file999.txt'. You do not have to enter all 999 entries manually, URLUpdat will do it for you. Simply enter the URL of the first file that you want to download i.e. 'http://www.files.com/file1.txt' and URLUpdat will do the rest. FILE_SOURCE_DIRECTORY e.g. URLUpdat:FilesIn/ This is the source directory from where URLUpdat will look for downloaded files. I recommend that you backup this before running URLUpdat as once a downloaded file has been copied to its destination directory and any alterations to its filename and datestamp have been successfully completed then the original is no longer needed and automatically deleted. FILE_DESTINATION_DIRECTORY e.g. URLUpdat:FilesOut/ This is the destination directory where URLUpdat will copy any modified files. WARNING: as it stands the Amiga Dos Copy command will not check if there is enough room for the copied files on the destination volume. It is incumbent on the user to do so. This should not be a problem as URLUpdat only copies one file at a time, but those of you out there (including myself) who use their Amigas in conjunction with their PCs via the shareware program PC2Amiga (available from Aminet 'comms/net/PC2Am308.lha') will have to make sure that there is enough room for all the copied files on the Amiga if URLUpdat is copying them directly from the PC. Failure to do so will result in checksum errors on the Amiga destination volume once it has run out of room, and several hours spent repairing the damage with DiskSalv. URL_INPUT_FILE e.g. Ram:URL_In.txt This is a simple ASCII file containing a list of the URLs that you would like to download in the order that you would like to downloaded them. NB. Many download utilities download files in parallel. That way if a web site is very slow the other files do not have to be stuck in a queue waiting for one stubborn file to finish downloading. This is great, but does tend to randomize the file order if you are listing them chronologically. URLUpdat sets the datestamps of selected files so that they reflect the date and time according to the position that they appear on this list, and not the date and time that the file was actually downloaded. Each URL should be on a separate line. URLUpdat will scan backwards from the end of this line until it finds a '/' character, signifying the division between the location of the file and the filename itself. For this reason all valid lines must contain a '/' somewhere on them. The Filename itself is then subdivided into an alpha, a numeric and an extension section for processing e.g. 'file', '999', '.txt'. URLUpdat assumes that most regularly updated files will similarly be comprised of an alpha section, followed by a numeric section followed by an extension; filenames that jumble their alphanumeric characters may not be processed correctly by URLUpdat. The first time that URLUpdat encounters '/Monday' it will prompt the user to enter the date of the first Monday from which file downloading was meant to begin (rarely the same as the date that it actually did). The date is entered as a single eight digit number in the form year (4), month (2), day (2) e.g. 19980816. Those using the American system for dates take special care to follow the European convention here; a derivative of this number can later be incorporated into the modified filename allowing files with the same alpha prefix to be listed in the same order whether they are listed alphabetically or chronologically. This does not happen with the US date convention as the month and day are taken out of the years, months, days, hours, minutes and seconds chronological sequence. Following on from the first Monday every time that URLUpdat encounters a valid day it will increase by one the date of files that were supposed to be downloaded on that day, resetting the time value to its default for a new day. The example script is configured for all the files that might be downloaded in one week (this is the time scale that the majority of web masters use when updating their sites), but it could be configured for one day or one month or more, there is no upper limit, as long as no days are skipped. URLs encountered before the first '/Monday' will retain there original datestamps, even if their filenames are altered. If COMMAND "e" is being used i.e. URLUpdat is in expand mode then any single URL can be forced to undergo inflation. E.g. If you wish to download all files between 'http://www.files.com/file001.txt' and 'http://www.files.com/file999.txt' it is necessary to enter only the FIRST URL, i.e. 'http://www.files.com/file001.txt'. In expand mode URLUpdat will fill in the blanks for you. This option is best served as a precursor to a batch download. Execute URLUpdat in 'e' mode to generate a batch list for download, feed this list to your download utility, then execute URLUpdat in its normal 'u' mode to process the successfully downloaded files, including those downloaded from URLs that underwent inflation. I recommend backing up this file as URLUpdat will probably automatically overwrite it with an updated list of URLs once all modifications to the current crop of downloaded files are completed successfully cf. URL_OUTPUT_FILE. URL_OUTPUT_FILE e.g. Ram:URL_Out.txt Once all the downloaded files from the original ASCII list of URLs have been processed then an updated list of URLs is prepared and saved as an ASCII file. This argument should normally have the same value as the URL_INPUT_FILE argument, ensuring that the old list of URLs is overwritten after it has served its purpose, but here and in the IconX example script both files are held separately in RAM: before the old list of URLs is overwritten by the new. URL_MODIFIERS_FILE e.g. Ram:URL_Mods.txt This is a standard delimited ASCII file containing the desired modifications to the downloaded files and the URL list that was used to download them. Discrete data segments are separated by commas; strings are enclosed within double quotation marks. Every line holds a separate modification instruction. A typical modification instruction is broken down into five elements: "http://www.files.com/","file",7,"new","date" The first part of the instruction outlines the full Internet address of the downloaded file excluding the filename. E.g. if the full URL were 'http://www.files/file999.txt' all that would be required is 'http://www.files.com/' not the 'file999.txt file' part. If you will it is the 'directory' containing the file. The second part is the alpha part of the filename. E.g. if the full URL were again 'http://www.files.com/file999.txt' the filename would be 'file999.txt' and all that would be required is 'file'. Only one instruction is needed for URLs that are identical up to this part in their structure. E.g. in this case one instruction would process all the URLs between 'http://www.files.com/file1.txt' and 'http://www.files.com/file999.txt' and beyond. Upon successfully downloading a file from the ASCII list of URLs the corresponding URL is incremented by a set number. This ensures that when the updated list of URLs is again fed to a download utility it will not download the same file again, but the next in sequence. The third part of the instruction is the number by which the successful URL is incremented (failed URLs are fed back as are). Setting this value to 0 means that the URL will not be altered (some files at the end of URLs are changed on a regular basis even though there URLs are not). The fourth part of the instruction determines what the alpha part of the filename of the modified file will be. E.g. suppose the original downloaded file was called file999.txt. Setting the fourth parameter to 'new' means that once the file has been processed it will have the filename 'new999.txt'. By setting the fifth parameter of the instruction to 'date' the modified file will incorporate the file's date into its modified filename between the alpha and numeric segments. This makes sure that URLs that are supposed to remain constant from download to download will nevertheless generate files with unique filenames that when listed alphabetically will retain the same characteristics as when listed chronologically. In this case, the downloaded file 'file999.txt' will be renamed 'new19980816999.txt' assuming it was supposed to be downloaded on Sunday 16th August 1998. If you wish to disable this option then set the parameter to any other string value than 'date'; 'no' works quite well. If this parameter is set to an integer value then it doubles as the upper expansion limit for a given URL undergoing inflation when URLUpdat is in 'e' mode. Consequently in 'u' mode URLUpdat will not increment the URL even after its file has been downloaded successfully and its increment modifier signals that it should be icremented; URLUpdat assumes that after a given URL undergoes inflation it does not require further incrementing. E.g. if the full modification instruction for URLUpdat were "http://www.files/","file",1,"new","999" and the URL_INPUT_FILE contained http://www.files.com/file001.txt then URLUpdat would add http://www.files.com/file002.txt http://www.files.com/file003.txt and all the URLs up to and including http://www.files.com/file999.txt to its designated URL_OUTPUT_FILE. This expanded list of URLs can then be passed to a download utility so that all the files can be downloaded without the user having to enter them manually. When URLUpdat is later executed in 'u' mode all the files between file001.txt and file999.txt will be processed as normal, but their corresponding URLs will not be incremented; here the '999' value in the 'date' parameter signals that the increment modifier of '1' is used only for this specific case of URL inflation, not for routine URL increments. BEGIN_HOURS e.g. 03 This argument comprises a two digit number between 00 and 23 representing the 24 hour clock. When a new valid day is encountered in the ASCII list of URLs the timestamp of the next file processed is set to the default value for a new day, determined by this and the following two arguments. E.g. if this argument is set to 18 and the following two arguments are set to 30 and 15 then every time the list of URLs calls for a new day the timestamp of the first file that was supposed to be downloaded on that day will be set to 18:30:15. NB. Try not to use 00 here; this is the value that I use and if ever I get any files from someone using this program it will mess up my file order (hey, if you are not paying for this program then this is the least that you can do). BEGIN_MINUTES e.g. 00 This argument comprises a two digit number between 00 and 59 representing the minutes in an hour. BEGIN_SECONDS e.g. 00 This argument comprises a two digit number between 00 and 59 representing the number of seconds in a minute. PAUSE_SECONDS e.g. 20 The timestamps of sequentially processed files will be incremented by a number of seconds equal to the numeric value set here. E.g. if this is set to 90 then when the processed files are examined it will appear as if every file took one and a half minutes to download. This allows the later insertion of files into the 'gaps'. If this argument is set to 0 then the modified files will retain there original datestamps. A word of caution before you choose this value. If the interval is so great that it causes the clock to go from 23:59 to 00:00 the date will not increase, and new files will have earlier datestamps than older ones. Only a valid day encountered by URLUpdat in the list of URLs can trigger a date change. FAILED_URLS_FILE e.g. Ram:URL_Fail.txt This is an ASCII file containing a list of URLs that failed to download (when it comes time to retry downloading failed URLs depending on which download utility you use it is easier to feed into it a list of failed URLs rather than the entire list). Generated only in 'u' mode. MOVE_FILES_SHELL_SCRIPT e.g. Ram:URL_Move.txt URLUpdat does not modify files directly. It creates a shell script listing desired modifications to downloaded files that the user must execute (the example IconX script does this for you automatically). This argument specifies the name of the ASCII file containing this script. Having a shell script allows you to see exactly what actions will be carried out on your downloaded files before they are carried out; it also allows anyone with a working knowledge of word processors and A- Rexx macros to further modify the actions to be performed on their downloaded files without an in depth knowledge of 'C'. A file will only be included in this script if it already exists in the source directory, matches a valid URL in the ASCII list of URLs and its modified filename would not overwrite an existing file in the destination directory. Generated only in 'u' mode. Known bugs: This is not really a bug, just a word of caution. Because PCs put a carriage return as well as a line feed to signal the end of a line in an ASCII file, you should only use scripts that have been prepared on an Amiga, not a PC. When I was writing this program I initially used the list of URLs I had prepared using Notepad and FileHound on the PC. When URLUpdat tried to process this it worked fine the first time, but not the second. URLs are case sensitive, and they are handled as such by URLUpdat, so take care when entering information into the URLs and Mods script files. History: URLUpdat v1.01.19980816 default update mode in first release now activated by using COMMAND u. COMMAND e added. Spaces now allowed in directory structures and filenames , (I know Amiga users know better than to use spaces indiscriminately, but PC Users do not, nor do some PC operating system developers). No corresponding URL_Mods looping error caused by outdated library fixed. URLUpDat v1.00.19980518 first release. Disclaimer: The author accepts no liability for any loss or damage incurred through use or misuse of this program. This program is shareware; if you find that it spares you from endless tedious hours of keeping your URL addresses and downloaded files in good order then please spare a thought for the author and send the equivalent of £10 sterling to the postal address below. As the author does not believe in crippleware, you will receive no benefit for doing so other than knowing that you have paid good money for good service and helped in the development of this and future products. The Amiga is dead; long live the Amiga. The example URLs in the Scripts directory are exactly that. The author does not bear any responsibility for the content at the end of these links, nor does the author encourage users of this program to follow them. Users do so at their own risk. There is nothing to stop anyone from including their own list of URLs with this program. Mr. M. J. Maloney BSc (Hons.) 53 Vale Road COLWICK Nottingham NG4 2GL United Kingdom. Bug reports; messages of praise; offers of marriage / employment etc. to: avatar_uk@hotmail.com