tcpdl V2.3b distribution January 1998 CONTENTS ======== This archive consists of the following files: tcpdl 2.3 the tcpdl http download program tcpdlpp 1.4 the post processor for downloaded pages tcpdl.doc this document urllist example urllist file tcpdl.config example tcpdl.config file tcpdlpp.config example tcpdlpp.config file INTRODUCTION ============ tcpdl is a program to download files from http hosts. Files to be downloaded are listed in a file called "tcpdldir:urllist". tcpdlpp is a program to post process files that have been downloaded by tcpdl. This amends the URLs within each html file so that the downloaded pages may be used locally, while any URLs that have not been downloaded refer to their full URL. COPYRIGHT ========= tcpdl is Copyright © 1996 Patrik Nilsson Copyright © 1997 Ramjam Consultants Ltd tcpdlpp is Copyright © 1997 Ramjam Consultants Ltd This archive is freely distributable, but may not be included in any commercial software collection other than Aminet without prior permission. REQUIREMENTS ============ To use tcpdl you need a TCP/IP stack and a connection to a http host. Both tcpdl and tcpdlpp should work on any Amiga system which supports a TCP stack. In use tcpdl typically requires approx 1Mb of RAM plus 500-1000 bytes for each file to be downloaded, and tcpdlpp requires about 100K plus 100 bytes per file to be processed. Disk space equivalent to about twice the size of the files to be downloaded is also required. tcpdl tries to use Xen font for the status window. If this is not available it will use the default font, but this may lead to some data not being readable. SUPPORT ======= Please report any problems, and send any suggestions or comments to: support@ramjam.demon.co.uk The most recent version will always be available from http://www.ramjam.demon.co.uk/software Major versions may also be available from Aminet. There is no requirement to register use of this program, but anyone doing so will be kept informed of updates. Both programs have been optimised for the 68000 processor in order to be as generic as possible. KNOWN PROBLEMS ============== tcpdl: - doesn't handle URLs which use Host Masking. - doesn't allow access to sites which require a userid/password. - doesn't support the automatic download of URLs referenced by the name attribute of the object tag. This is because the name attribute does not always apply to a URL. Since the object tag is only supported by IE3, and the name attribute refers to a URL that is likely to be applicable only to IE3, this is not seen as a significant restriction. - doesn't handle URLs containing invalid Amiga filenames as well as it could tcpdlpp: - No current reported problems. DISCLAIMER ========== A great deal of effort has gone into making these programs as reliable as possible. However, there is no guarantee that they will perform as described in all cases. These programs are used entirely at the user's own risk. No liability can be accepted for loss of data resulting from the use of these programs. AKNOWLEDGEMENTS =============== Thanks to Patrik Nilsson for the original version of tcpdl. Thanks also to all the users of tcpdl for their comments and suggestions. CHANGES FROM PREVIOUS RELEASE ============================= tcpdl ----- V2.3a -> V2.3b: - tcpdl sent http commands that some hosts didn't like - it should now manage to talk to hosts using any version of HTTP. - The UPDATE flag wasn't correctly propogated to all the URLs referenced by the files marked with UPDATE in the urllist. This was a fundamental design flaw, and so the UPDATE flag has been removed and replaced by an UPDATE command line option which applies to all URLs in urllist. - The IMG flag didn't quite do what was advertised, so it has been amended slightly: TEXT means all URLs that are referenced by a "href" attribute IMG means all URLs that are referenced by an attribute other than "href" ALL means all URLs that are referenced - added a configuration option USER, which allows the mail address to be used in the HTTP commands to be specified. By default tcpdl uses the username and host name used by the tcp/ip stack. If the USER option is specified with no mail address, then no From: header will be sent in HTTP commands. - tcpdl used to send the user's realname to the http host - there is no need to do this, so now it doesn't. - fixed the DISK-ERR message, which looked untidy. V2.3 -> V2.3a: - made it possible to exit tcpdl when run from Workbench - fixed enforcer hits when urllist contains a blank line V2.2 -> V2.3: - added support for proxy servers - added configuration options for connection, and htpp transfer timeouts - added configuration option for the number of retries for failed transfers - tcpdl now uses memory pools for its memory allocation. This improves the allocation times slightly, but has a dramatic effect on the deallocation time. It also reduces the risk of memory "leakage" when tcpdl is interrupted. - the User Agent name has been changed to conform to RFC2068 - the tcpdl.config file is now closed as soon as it has been read - the environment variable USERNAME is now used if USER is not set - changed the ERROR status indicator to be more specific about the cause of the error V2.1 -> V2.2: - added tcpdl.config to allow the specification of file types that are not to be downloaded. V2.0 -> V2.1: - fixed problem where URLs in the urllist file which were not followed by at least one space or tab character were ignored. tcpdlpp ------- V1.3 -> V1.4: - changed the handling of URLs with a leading '/'. All URLs are now converted to relative form, so all local links should work when browsing off-line. V1.2 -> V1.3: - minor optimizations V1.1 -> V1.2: - improved the handling of ".." in URLs - added optional translation of characters in URLs, by means of a new configuration file "tcpdldir:tcpdlpp.config" USING TCPDL AND TCPDLPP ====================== OVERVIEW: --------- Both tcpdl and tcpdlpp expect the assign tcpdldir: to refer to a directory. This directory is the work area for both programs. The urllist file, containing the list of URLs to be downloaded is expected to be in this directory. The optional configuration files, tcpdl.config and tcpdlpp.config should also be in this directory. When tcpdl downloads URLs it will create two directories below tcpdldir:, DATA and HTTP. Beneath each of these, one directory will be created for each host, and beneath each of these will be the directories and files which are downloaded. The HTTP directory contains the actual files that are downloaded, while the DATA directory contains files holding information about each file downloaded. Example: the following directory tree shows the structure that might result from use of the example URLs given in the "THE TCPDLDIR:URLLIST FILE" section below. tcpdldir: | | +------- urllist | | +------- HTTP | | | | | +------- www.ramjam.demon.co.uk | | | | | | | | +------- index.html | | | | | | | | +------- amiga | | | | | | | | | | ... ... | | | | | +------- www.cu-amiga.co.uk | | | | | | | | +------- index.html | | | | | | | ... ... | | +------- DATA | | ... Thus, once a file has been downloaded, it appears within a directory that identifies the host from which it came. The DATA directory is used by tcpdl during the download and update process. The final files appear in the HTTP directory. The HTML files downloaded by tcpdl will have references to URLs replaced by a reference to a local file, e.g. http://www.ramjam.demon.co.uk/index.html will become file://localhost/tcpdldir:http/www.ramjam.demon.co.uk/index.html The post processor, tcpdlpp, processes all files within the tcpdldir:http directory and converts references to other files that are present in this file hierarchy to relative URLs, and converts all other references back to absolute URLs. This allows the files within the tcpdldir:http directory to be browsed offline, while allowing links to other URLs to be followed if the user happens to be online. By downloading your favourite pages, you can browse the web much faster, while still being able to follow links to other sites. The downloaded pages may be updated periodically using tcpdl with the UPDATE option, and then running tcpdlpp again to adjust any amended references. TO START: --------- 1. assign "tcpdldir:" to the directory which is to contain the downloaded files (this directory should exist). (e.g. "assign tcpdldir: work:tcpdldir") 2. Edit the file tcpdldir:urllist such that it contains the files to be downloaded. 3. edit tcpdldir:tcpdl.config as required 4. edit .tcpdlpp.config as required 5. check that there is enough disk space for the pages you intend to download 6. connect to the Internet 7. run "tcpdl" from a shell 8. if required, run "tcpdlpp" from a shell (this can be done offline). If on checking the output of tcpdlpp there are a lot of references to a non-local URL you may want to use tcpdl to download that URL. After downloading it, re-run tcpdlpp to change all links to that URL to refer to the local file. TCPDL TASK STATUS WINDOW: ------------------------- The tcpdl status window is updated approx. once per second (Note that not every change in status will have a chance to appear in the window). tcpdl will download up to 15 files at once. There is one line in the status window for each of these tasks. The fields on each line are described below: Status: one of the following values: "Connecting" Trying to connect a host. "Sending" Sending request. "Header" Receiving header. "Updating" Requesting using "If-Modified-Since" from a host or loading data from tcpdldir:data/ "OK" File downloaded successfully "Receiving" Receiving data. "Wait. html" A limit of 512k html-data to process. Processing will continue when the amount outstanding falls below the limit. "Proc. html" Processing html. "Copying" Copying html-file from t: to tcpdldir:http/ "LIB-ERR" Unable to open bsdsocket.library "HOST-ERR" Unknown host "SOCKET-ERR" Unable to open socket "CON-ERR" Unable to connect to host "HDR-ERR" Failed to download header "RECV-ERR" Failed while receiving data "FILE-ERR" Failed to open output file "DISK-ERR" Failed while writing to output file "ERROR" Some other error occurred Time: elapsed time since trying to connect CPS: the current download rate achieved for this file CSize: the size of data received FSize: the size of data, if available in header. Request: the request string. The top line of the status window also contains an overall progress indicator (DONE: TOTAL:): the number of files downloaded so far the number of files listed in memory The bottom line of the status window gives some overall performance figures: Total time: the elapsed time since tcpdl started execution Total bytes: the total number of bytes downloaded so far Average cps: the average number of characters per second downloaded OTHER NOTES: ------------ tcpdl uses "t:" to store temporary files while processing html. Execution may be terminated using CTRL+C. The program exits as fast as is possible safely. It can take a little while if it is processing large htmlfiles at the time. Any part of a filename that is not a valid amigados-name is replaced with a crc-checksum. e.g.: http://www.cs.uit.no/~kjelli/thor.html would become tcpdldir:http/www.cs.uit.no/_kjelli_CRC_BDF5F321/thor.html If a particular host times out on more than 5 occasions, no further attempts are made to download any files from that host. This avoids wasting time attempting to connect to a host that is down. THE FILE "TCPDLDIR:urllist": ---------------------------- The urllist file should contain one or more URLs which are to be downloaded. Each URL should start on a new line, and be followed by the appropriate flags, separated by spaces. The supported flags are: D1, D2, ..., D19 downloading n levels of text/html DEFAULT 255 (!) H0, H1, ..., H5 If another host is referenced by a HREF max number of levels is set to n. DEFAULT H0, current host only. P0, P1, ..., P5 If path is other than given in your urllist max number of levels is set to n. DEFAULT Pn, where n is the same as for Dn TEXT download files referenced by a "href" attribute. These will commonly, but not exclusively, be HTML files. IMG download files referenced an attribute other than "href". These are commonly, but not exclusively, images. ALL download all types of files. This is the default if none of IMG, TEXT, ALL are specified. (Note that this will not download files with types that appear in IGNORE lines within tcpdl.config) (UPDATE This flag has been removed - use the UPDATE command line option instead) e.g. http://www.cu-amiga.co.uk/index.html D2 H3 TEXT will download 2 levels of html files referenced by the specified file from www.cu-amig.co.uk, and 3 levels of links to any other host. http://www.ramjam.demon.co.uk/amiga.html D5 H0 ALL will download 5 levels of files referenced by the specified file from www.ramjam.demon.co.uk, but will not download any files that are referenced on any other host. http://www.ramjam.demon.co.uk/ will download all text files referenced by index.html from the host www.ramjam.demon.co.uk. THE FILE "TCPDLDIR:tcpdl.config": --------------------------------- The tcpdl.config file is optional. If present, it will be read in and the contents will be used in determining what file types will be downloaded. Any line with a hash ('#') in column 1 will be ignored as a comment. White space is ignored, and the commands are not case-sensitive. Currently the following configuration commands are suported: IGNORE where is a file suffix which should not be downloaded. Note that such files mentioned explicitly in urllist will be downloaded, but any such files referenced within html will not. Note that may contain any characters except white space, but will only be matched against the end of a file name. PROXY this specifies that all http requests shpould be sent via the specified server. If the port number is omitted, then a default of 8080 is used. By specifying your ISP's proxy server you can improve download speeds significantly - especially for busy sites. A proxy will also be required for connections via a firewall. CONTIMEOUT this specifies the initial timeout for each connection in seconds. The default is 20. The timeout must be within the range 10 to 600. HTTPTIMEOUT this specifies the timeout for each http request in seconds. The default is 60. The timeout must be within the range 10 to 600. RETRIES this specifies the number of attempts that will be made to download each file. The default is 5. The value must be within the range 1 to 100. USER the mail address is sent to the HTTP server as the address that mail can be sent to if there are problems caused by tcpdl's requests. If the USER option is specified without a mail address, then HTTP requests will not include any mail address (this can help maintain your anonymity). If the USER option is not present in the config file, then your current user id and host name are used. e.g. The following is an example of what may appear in tcpdl.config # # Specify the Demon Internet proxy server # PROXY www-cache.demon.co.uk:8080 # # Specify the timeouts - small since we're using a proxy # CONTIMEOUT 10 HTTPTIMEOUT 30 # # Specify the number of attempts for each file # RETRIES 2 # # Specify that no mail address is to be sent to the server # USER # # Specify the file suffixes not to be downloaded # # ignore lha archives IGNORE .lha # ignore zip archives IGNORE .zip # ignore .wav sound files IGNORE .wav # ignore MS-DOS executables IGNORE .exe THE FILE "TCPDLDIR:tcpdlpp.config": ----------------------------------- The tcpdlpp.config file is optional. If present, the character translations defined in it are applied to each URL processed. Note that the actual file names are NOT changed, but only the URLs within each html file. Each line of this file consists of a character literal to be converted and a character literal or string that should replace it. Character literals should be enclosed by single quotes ('), and strings should be enclosed by double quotes ("). White spaces (spaces and tabs) are ignored unless inside a string. Any line where the first non-whitespace character is a hash (#) is treated as a comment and ignored. Certain escape characters are allowed in character literals and strings: \a = bell \b = backspace \f = formfeed \n = newline \r = carriage return \t = horizontal tab \v = vertical tab \\ = backslash \' = single quote \" = double quote \nnn = character with octal value nnn \xnnn = character with hexadecimal value nnn e.g. the following line will convert MS-DOS style backslashes into the AmigaDOS & Unix style forward slashes: '\\' '/' the following line will convert tilde into the safe "%xx" equivalent: '~' "%7E" Each character is translated using a single rule - even if the end result includes a character which would have been translated by some other rule. This allows two characters to be swapped over. TCPDLPP LISTING --------------- The tcpdlpp program post processes the files downloaded by tcpdl. It expects the same tcpdldir: assign as tcpdl does. A listing is sent to stdout, which consists of 3 sections: - a list of each file processed or skipped. Since only html files will contain URLs to be updated, all other files are skipped. This acts as a progress indicator. - a list of all local files. The number of references to each is given. If a file has no references to it, either it is a top level html file, or it is simply not referenced. If you are building a browsable copy of your favourite sites, you may want to delete any unreferenced files to save disk space. - a list of all non-local URLs. As for the local files, the number of references made to each by the local files is given. If a particular URL has a lot of references, you may want to download that URL too. Each time you add or remove files from the tcpdldir:http directory, you should re-run tcpdlpp to adjust any links that require amendment. Files which are present, but which have had the URLs which reference them modified in some way by the translations in the tcpdlpp.config file will be listed as non-local unless the file name has been modified also. If such files are listed, then the file names should be changed, and tcpdlpp re-run to identify those files as local. WARNING ------- Note that unless explicitly stated to the contrary, the copyright of all files on the WWW is held by the owners of the appropriate site. If you intend to redistribute *any* files downloaded from the WWW please ensure that you have the permission of the copyright holder to do so.