Short: v1.02, Fast dl of http. Use for offline reading. (amitcp) Uploader: patrik.nilsson@mbox3.swipnet.se (Patrik Nilsson) Author: patrik.nilsson@mbox3.swipnet.se (Patrik Nilsson) Type: comm/tcp Replaces: tcpdl.lha Distribution: Aminet tcpdl.doc v1.02 COPYRIGHT © 1996 PATRIK NILSSON =============================== This program is copyrighted to Patrik Nilsson. Everybody may use this program as long as: * Sending me an email at "patrik.nilsson@mbox3.swipnet.se" telling that you are using this program, when you find it useful. (I do not promise to reply...) * You are using it at your own risk. SHORT: ===== tcpdl is a program to download files from http hosts. Files to be downloaded is listed in a file called "tcpdldir:urllist". There are an example coming with this package. LONG: ===== TO START: ======== 1. You have to assign "tcpdldir:" anywhere. (i.e. "assign tcpdldir: work:tcpdldir" 2. Make sure that tcpdldir:urllist exists. 3. Run "startnet" (amitcp) 4. run "tcpdl" from a shell see also: "testtcpdl", a script coming with this package. WINDOW: ====== The window is updated every 3rd second (downloading has got a higher priority). --cut-- Status Time CPS CSize FSize Request (2685/2927) --cut-- WINDOW/Status: ============== What is going on? "Connecting" Trying to connect a host. "ERROR" An error detected. "Sending" Sending request. "Header" Receiving header. "Updating" Requesting using "If-Modified-Since" from a host or loading data from tcpdldir:data/. "OK" Okej. "Receiving" Receiving data. "Wait. html" A limit of 512k html-data to process. Will continue when below limit. "Proc. html" Processing html. "Copying" Copying html-file from t: to tcpdldir:http/ WINDOW/Time: ============ Time since trying to connect. WINDOW/CPS: =========== CPS WINDOW/CSize: ============= Size of data received. WINDOW/FSize: ============= Size of data, if available in header. WINDOW/Request: =============== Request string. WINDOW/(2685/2927): =================== example: 2685 is number of request done. 2927 is number of requests in memory. OTHER NOTES: =========== It uses "t:" for temporary files. (When processing html-files.) The program requires about one megabyte to run. You can always break the program using CTRL+C. The program exits as fast as possible, breaking all downloadings. (It can be a while if processing large htmlfiles - those processes are not broken.) Filenames that is not a valid amigados-name is replaced with a crc-checksum. i.e.: http://www.cs.uit.no/~kjelli/thor.html would be tcpdldir:http/www.cs.uit.no/_kjelli_CRC_BDF5F321/thor.html The program reties every request 5 times, each of them have a timeout of about 10 seconds until connected and then 60 seconds. If the download fails the request is not repeated. THE "TCPDLDIR:URLLIST"-FILE: =========================== The file contains: a request-string (url) at left, one (or more) space(s) and then flags for the download. http://host/path/ D2 H3 IMG TEXT http://www.sds.se D5 H0 IMG the flags can be: D1, D2, ..., D19 downloading n levels of text/html H0, H1, ..., H5 If an other host is refered by a HREF max number of levels is set to n. DEFAULT H0; current host only. P0, P1, ..., P5 If path is other than given in your urllist max number of levels is set to n. DEFAULT IMG download IMG-files (pictures at html) TEXT download text-files (accept: text/*) ALL download all types of files (accept: */*) UPDATE If this flag does not exists, the program does not request this file if at harddisk. If "UPDATE" exists and file is at harddisk the request contains "If-Modified-Since". (Date sent, when using this, is the same coming with header. see "tcpdldir:data/http/" ) example: ======= You have found kjelli's homepage (thor). You would download everything he has got at his page. http://www.cs.uit.no/~kjelli/thor.html D19 P0 IMG ALL This will download his html-file for two levels and get his fiends homepages. http://www.cs.uit.no/~kjelli/thor.html D2 P1 IMG OFFLINE-READING (MOSAIC): ======================== run "amitcp:amitcp" as command double click at mosaic icon and then open menu "local host". Get a file from i.e. "tcpdldir:http/www.cs.uit.no/_kjelli_CRC_01234567/thor.html Thanks: ====== This will be edit as soon as possible. HISTORY: ======= v1.00 First public release. --- v1.01 Bugs: * TAB-problem, fixed. * Enforcer hits fixed. New: * Didn't know about "port" in "http://host:port/path" * The program reties every request 4 times, each of them have a timeout of about 10 seconds until connected and then 30 seconds. If the download fail the request is not repeated. --- v1.02 * Now using xen.font 8 if available at disk, else using default font meaning the text wouldn't fit in its window. Bug: * "Port" v1.01 didn't work. I hope this will. I didn't test it:( * More TAB-problems, fixed. New: * For safe: if a host fails more than 10 times this host is not called any more. This task has a priority of -2, which means that everything that is more important is done before this. changed: The program reties every request 5 times, each of them have a timeout of about 10 seconds until connected and then 60 seconds. If the download fail the request is not repeated.