tcpdl V2.2 distribution August 1997 CONTENTS ======== This archive consists of the following files: tcpdl the tcpdl http download program tcpdlpp the post processor for downloaded pages tcpdl.doc this document urllist example urllist file tcpdl.config example tcpdl.config file tcpdlpp.config example tcpdlpp.config file INTRODUCTION ============ tcpdl is a program to download files from http hosts. Files to be downloaded are listed in a file called "tcpdldir:urllist". tcpdlpp is a program to post process files that have been downloaded by tcpdl. This amends the URLs within each html file so that the downloaded pages may be used locally, while any URLs that have not been downloaded refer to their full URL. COPYRIGHT ========= tcpdl is Copyright © 1996 Patrik Nilsson Copyright © 1997 Ramjam Consultants Ltd tcpdlpp is Copyright © 1997 Ramjam Consultants Ltd This archive is freely distributable, but may not be included in any commercial software collection other than Aminet without prior permission. REQUIREMENTS ============ To use tcpdl you need a TCP/IP stack and a connection to a http host. Both tcpdl and tcpdlpp should work on any Amiga system which supports a TCP stack. In use tcpdl typically requires approx 1Mb of RAM plus 500-1000 bytes for each file to be downloaded, and tcpdlpp requires about 100K plus 100 bytes per file to be processed. Disk space equivalent to about twice the size of the files to be downloaded is also required. tcpdl tries to use Xen font for the status window. If this is not available it will use the default font, but this may lead to some data not being readable. SUPPORT ======= Please report any problems, and send any suggestions or comments to: support@ramjam.demon.co.uk The most recent version will always be available from http://www.ramjam.demon.co.uk/software Major versions may also be available from Aminet. There is no requirement to register use of this program, but anyone doing so will be kept informed of updates. Both programs have been optimised for the 68000 processor in order to be as generic as possible. Versions compiled specifically for other processors (e.g. 68030, 68040) are available upon request. KNOWN PROBLEMS ============== tcpdl: - doesn't support the automatic download of URLs referenced by the name attribute of the object tag. This is because the name attribute does not always apply to a URL. Since the object tag is only supported by IE3, and the name attribute refers to a URL that is likely to be applicable only to IE3, this is not seen as a significant restriction. - doesn't handle URLs containing invalid Amiga filenames very well - uses forbid/permit, which limits the program's compatibility with future versions of AmigaDOS tcpdlpp: - No reported problems. DISCLAIMER ========== A great deal of effort has gone into making these programs as reliable as possible. However, there is no guarantee that they will perform as described in all cases. These programs are used entirely at the user's own risk. No liability can be accepted for loss of data resulting from the use of these programs. AKNOWLEDGEMENTS =============== Thanks to Patrik Nilsson for the original version of tcpdl. CHANGES FROM PREVIOUS RELEASE ============================= tcpdl ----- V2.1 -> V2.2: - added tcpdl.config V2.0 -> V2.1: - fixed problem where URLs in the urllist file which were not followed by at least one space or tab character were ignored. tcpdlpp ------- V1.2 -> V1.3: - minor optimizations V1.1 -> V1.2: - improved the handling of ".." in URLs - added optional translation of characters in URLs, by means of a new configuration file "tcpdldir:tcpdlpp.config" USING TCPDL AND TCPDLPP ====================== OVERVIEW: --------- Both tcpdl and tcpdlpp expect the assign tcpdldir: to refer to a directory. This directory is the work area for both programs. The urllist file, containing the list of URLs to be downloaded is expected to be in this directory. The optional configuration files, tcpdl.config and tcpdlpp.config should also be in this directory. When tcpdl downloads URLs it will create two directories below tcpdldir:, DATA and HTTP. Beneath each of these, one directory will be created for each host, and beneath each of these will the directories and files which are downloaded. Example: the following directory tree shows the structure that might result from use of the example URLs given in the "THE TCPDLDIR:URLLIST FILE" section below. tcpdldir: | | +------- urllist | | +------- HTTP | | | | | +------- www.ramjam.demon.co.uk | | | | | | | | +------- index.html | | | | | | | | +------- amiga | | | | | | | | | | ... ... | | | | | +------- www.cu-amiga.co.uk | | | | | | | | +------- index.html | | | | | | | ... ... | | +------- DATA | | ... Thus, once a file has been downloaded, it appears within a directory that identifies the host from which it came. The DATA directory is used by tcpdl during the download and update process. The final files appear in the HTTP directory. The HTML files downloaded by tcpdl will have references to URLs replaced by a reference to a local file, e.g. http://www.ramjam.demon.co.uk/index.html will become file://localhost/tcpdldir:http/www.ramjam.demon.co.uk/index.html The post processor, tcpdlpp, processes all files within the tcpdldir:http directory and converts references to other files that are present in this file hierarchy to relative URLs, and converts all other references back to absolute URLs. This allows the files within the tcpdldir:http directory to be browsed offline, while allowing links to other URLs to be followed if the user happens to be online. By downloading your favourite pages, you can browse the web much faster, while still being able to follow links to other sites. The downloaded pages may be updated periodically using tcpdl with the UPDATE flag specified in the urllist file, and the links fixed by running tcpdlpp again. TO START: --------- 1. assign "tcpdldir:" to the directory which is to contain the downloaded files (this directory should exist). (e.g. "assign tcpdldir: work:tcpdldir") 2. Edit the file tcpdldir:urllist such that it contains the files to be downloaded. 3. edit tcpdldir:tcpdl.config as required 4. edit .tcpdlpp.config as required 5. check that there is enough disk space for the pages you intend to download 6. connect to the Internet 7. run "tcpdl" from a shell 8. if required, run "tcpdlpp" from a shell (can be done offline). If on checking the output of tcpdlpp there are a lot of references to a non-local URL you may want to use tcpdl to download that URL. After downloading it, re-run tcpdlpp to change all links to that URL to refer to the local file. TCPDL TASK STATUS WINDOW: ------------------------- The tcpdl status window is updated approx. once per second (Note that not every change in status will have a chance to appear in the window). tcpdl will download up to 15 files at once. There is one line in the status window for each of these tasks. The fields on each line are described below: Status: one of the following values: "Connecting" Trying to connect a host. "ERROR" An error detected. "Sending" Sending request. "Header" Receiving header. "Updating" Requesting using "If-Modified-Since" from a host or loading data from tcpdldir:data/ "OK" File downloaded successfully "Receiving" Receiving data. "Wait. html" A limit of 512k html-data to process. Processing will continue when the amount outstanding falls below the limit. "Proc. html" Processing html. "Copying" Copying html-file from t: to tcpdldir:http/ Time: elapsed time since trying to connect CPS: the current download rate achieved for this file CSize: the size of data received FSize: the size of data, if available in header. Request: the request string. The top line of the status window also contains an overall progress indicator (DONE: TOTAL:): the number of files downloaded so far the number of files listed in memory The bottom line of the status window gives some overall performance figures: Total time: the elapsed time since tcpdl started execution Total bytes: the total number of bytes downloaded so far Average cps: the average number of characters per second downloaded OTHER NOTES: ------------ tcpdl uses "t:" to store temporary files while processing html. Execution may be terminated using CTRL+C. The program exits as fast as is possible safely. It can take a little while if it is processing large htmlfiles at the time. Any part of a filename that is not a valid amigados-name is replaced with a crc-checksum. e.g.: http://www.cs.uit.no/~kjelli/thor.html would become tcpdldir:http/www.cs.uit.no/_kjelli_CRC_BDF5F321/thor.html The program reties every request 5 times, each of them have a timeout of about 20 seconds until connected and then 60 seconds. If a download fails the request is not repeated. Futher, if a particular host times out on more than 5 occasions, no further attempts are made to download any files from that host. This avoids wasting time attempting to connect to a host that is down. THE FILE "TCPDLDIR:urllist": ---------------------------- The urllist file should contain one or more URLs which are to be downloaded. Each URL should start on a new line, and be followed by the appropriate flags, separated by spaces. The supported flags are: D1, D2, ..., D19 downloading n levels of text/html DEFAULT 255 (!) H0, H1, ..., H5 If another host is refered by a HREF max number of levels is set to n. DEFAULT H0, current host only. P0, P1, ..., P5 If path is other than given in your urllist max number of levels is set to n. DEFAULT Pn, where n is the same as for Dn IMG download IMG-files (in fact, any non-html files) This option is of limited use, and may be removed in a future version. TEXT only download text files ALL download all types of files. This is the default if none of IMG, TEXT, ALL are specified. (Note that this will not download files with types that appear in IGNORE lines within tcpdl.config) UPDATE If this flag is not specified, the program does not request the file if it has already been downloaded to the harddisk. If "UPDATE" is specified and the file has already been downloaded then the file is downloaded again only if it has been modified since it was previously downloaded. e.g. http://www.cu-amiga.co.uk/index.html D2 H3 TEXT will download 2 levels of html files referenced by the specified file from www.cu-amig.co.uk, and 3 levels of links to any other host. http://www.ramjam.demon.co.uk/amiga.html D5 H0 ALL UPDATE will download 5 levels of files referenced by the specified file from www.ramjam.demon.co.uk, updating any that have previously been downloaded, but will not download any files that are referenced on any other host. http://www.ramjam.demon.co.uk/ will download all text files referenced by index.html from the host www.ramjam.demon.co.uk. THE FILE "TCPDLDIR:tcpdl.config": --------------------------------- The tcpdl.config file is optional. If present, it will be read in and the contents will be used in determining what file types will be downloaded. Any line with a hash ('#') in column 1 will be ignored as a comment. Currently only one configuration command is supported. IGNORE where is a file suffix which should not be downloaded. Note that such files mentioned explicitly in urllist will be downloaded, but any such files referenced within html will not. Note that may contain any characters except white space, but will only be matched against the end of a file name. White space is ignored, and the commands are not case-sensitive. e.g. The following is an example of what may appear in tcpdl.config # # File suffixes to be ignored # # ignore lha archives IGNORE .lha # ignore zip archives IGNORE .zip # ignore .wav sound files IGNORE .wav THE FILE "TCPDLDIR:tcpdlpp.config": ----------------------------------- The tcpdlpp.config file is optional. If present, the character translations defined in it are applied to each URL processed. Note that the actual file names are NOT changed, but only the URLs within each html file. Each line of this file consists of a character literal to be converted and a character literal or string that should replace it. Character literals should be enclosed by single quotes ('), and strings should be enclosed by double quotes ("). White spaces (spaces and tabs) are ignored unless inside a string. Any line where the first non-whitespace character is a hash (#) is treated as a comment and ignored. Certain escape characters are allowed in character literals and strings: \a = bell \b = backspace \f = formfeed \n = newline \r = carriage return \t = horizontal tab \v = vertical tab \\ = backslash \' = single quote \" = double quote \nnn = character with octal value nnn \xnnn = character with hexadecimal value nnn e.g. the following line will convert MS-DOS style backslashes into the AmigaDOS & Unix style forward slashes: '\\' '/' the following line will convert tilde into the HTML character entity equivalent: '~' "~" Each character is translated using a single rule - even if the end result includes a character which would have been translated by some other rule. This allows two characters to be swapped over. TCPDLPP LISTING --------------- The tcpdlpp program post processes the files downloaded by tcpdl. It expects the same tcpdldir: assign as tcpdl does. A listing is sent to stdout, which consists of 3 sections: - a list of wach file processed or skipped. Since only html files will contain URLs to be updated, all other files are skipped. - a list of all local files. The number of references to each is given. If a file has no references to it, either it is a top level html file, or it is simply not referenced. If you are building a browsable copy of your favourite sites, you may want to delete any unreferenced files to save disk space. - a list of all non-local URLs. As for the local files, the number of references made to each by the local files is given. If a particular URL has a lot of references, you may want to download that URL too. Each time you add or remove files from the tcpdldir:http directory, you should re-run tcpdlpp to adjust any links that require amendment. Files which are present, but which have had the URLs which reference them modified in some way by the translations in the tcpdlpp.config file will be listed as non-local unless the file name has been modified also. If such files are listed, then the file names should be changed, and tcpdlpp re-run to identify those files as local. WARNING ------- Note that unless explicitly stated to the contrary, the copyright of all files on the WWW is held by the owners of the appropriate site. If you intend to redistribute *any* files downloaded from the WWW please ensure that you have the permission of the copyright holder to do so.