Short: V1.04, Fast dl of http. Use for offline reading. prg+src Author: patrik.nilsson@mbox3.swipnet.se (Patrik Nilsson) Uploader: patrik.nilsson@mbox3.swipnet.se (Patrik Nilsson) Type: comm/tcp Replaces: tcpdl.lha Distribution: Aminet tcpdl.doc v1.04 COPYRIGHT © 1996 PATRIK NILSSON =============================== This program is copyrighted to Patrik Nilsson. Everybody may use this program as long as: * Sending me an email at "patrik.nilsson@mbox3.swipnet.se" telling that you are using this program, when you find it useful. (I do not promise to reply...) * You are using it at your own risk. SHORT: ===== tcpdl is a program to download files from http hosts. Files to be downloaded is listed in a file called "tcpdldir:urllist". There are an example coming with this package. LONG: ===== TO START: ======== 1. You have to assign "tcpdldir:" anywhere. (i.e. "assign tcpdldir: work:tcpdldir" 2. Make sure that tcpdldir:urllist exists. 3. Run "startnet" (amitcp) 4. run "tcpdl" from a shell see also: "testtcpdl", a script coming with this package. WINDOW: ====== The window is updated every 3rd second (downloading has got a higher priority). --cut-- Status Time CPS CSize FSize Request (2685/2927) --cut-- WINDOW/Status: ============== What is going on? "Connecting" Trying to connect a host. "ERROR" An error detected. "Sending" Sending request. "Header" Receiving header. "Updating" Requesting using "If-Modified-Since" from a host or loading data from tcpdldir:data/. "OK" Okej. "Receiving" Receiving data. "Wait. html" A limit of 512k html-data to process. Will continue when below limit. "Proc. html" Processing html. "Copying" Copying html-file from t: to tcpdldir:http/ WINDOW/Time: ============ Time since trying to connect. WINDOW/CPS: =========== CPS WINDOW/CSize: ============= Size of data received. WINDOW/FSize: ============= Size of data, if available in header. WINDOW/Request: =============== Request string. WINDOW/(2685/2927): =================== example: 2685 is number of request done. 2927 is number of requests in memory. OTHER NOTES: =========== It uses "t:" for temporary files. (When processing html-files.) The program requires about one megabyte to run and 500-1000 bytes for each request. You can always break the program using CTRL+C. The program exits as fast as possible, breaking all downloadings. (It can be a while if processing large htmlfiles - those processes are not broken.) Filenames that is not a valid amigados-name is replaced with a crc-checksum. i.e.: http://www.cs.uit.no/~kjelli/thor.html would be tcpdldir:http/www.cs.uit.no/_kjelli_CRC_BDF5F321/thor.html The program reties every request 5 times, each of them have a timeout of about 20 seconds until connected and then 60 seconds. If the download fail the request is not repeated. THE "TCPDLDIR:URLLIST"-FILE: =========================== The file contains: a request-string (url) at left, one (or more) space(s) and then flags for the download. http://host/path/ D2 H3 IMG TEXT http://www.sds.se D5 H0 IMG the flags can be: D1, D2, ..., D19 downloading n levels of text/html H0, H1, ..., H5 If an other host is refered by a HREF max number of levels is set to n. DEFAULT H0; current host only. P0, P1, ..., P5 If path is other than given in your urllist max number of levels is set to n. DEFAULT IMG download IMG-files (pictures at html) TEXT download text-files (accept: text/*) ALL download all types of files (accept: */*) UPDATE If this flag does not exists, the program does not request this file if at harddisk. If "UPDATE" exists and file is at harddisk the request contains "If-Modified-Since". (Date sent, when using this, is the same coming with header. see "tcpdldir:data/http/" ) example: ======= You have found kjelli's homepage (thor). You would download everything he has got at his page. http://www.cs.uit.no/~kjelli/thor.html D19 P0 IMG ALL This will download his html-file for two levels and get his fiends homepages. http://www.cs.uit.no/~kjelli/thor.html D2 P1 IMG OFFLINE-READING (MOSAIC): ======================== run "amitcp:amitcp" as command double click at mosaic icon and then open menu "local host". Get a file from i.e. "tcpdldir:http/www.cs.uit.no/_kjelli_CRC_01234567/thor.html Thanks: ====== This will be edit as soon as possible. HISTORY: ======= v1.00 First public release. --- v1.01 Bugs: * TAB-problem, fixed. * Enforcer hits fixed. New: * Didn't know about "port" in "http://host:port/path" * The program reties every request 4 times, each of them have a timeout of about 10 seconds until connected and then 30 seconds. If the download fail the request is not repeated. --- v1.02 * Now using xen.font 8 if available at disk, else using default font meaning the text wouldn't fit in its window. Bug: * "Port" v1.01 didn't work. I hope this will. I didn't test it:( * More TAB-problems, fixed. New: * For safe: if a host fails more than 10 times this host is not called any more. This task has a priority of -2, which means that everything that is more important is done before this. changed: The program reties every request 5 times, each of them have a timeout of about 10 seconds until connected and then 60 seconds. If the download fail the request is not repeated. --- v1.03 Rewrote loop "Infotoprocess". changed: The program retries every request 5 times, each of them have a timeout of about 20 seconds until connected and then 60 seconds. If the download fails the request is not repeated. The program requires about one megabyte to run and 500-1000 bytes for each request. --- v1.04 Now to check about "HTTP/1.0 200 " in the header. (If the request is not alright the "tcpdldir:data/http/" is not written.) ============================= Archive contents ============================= Original Packed Ratio Date Time Name -------- ------- ----- --------- -------- ------------- 594 285 52.0% 02-Jun-96 13:17:18 +Compile 891 266 70.1% 04-Feb-96 14:33:02 +Compile.info 860 295 65.6% 25-May-96 23:35:52 +copyfile.c 3768 683 81.8% 25-May-96 23:35:52 +createfinalurl.c 531 235 55.7% 24-Feb-96 09:58:30 +createpath.c 21608 3507 83.7% 01-Jun-96 21:02:50 +dotcpline.c 6215 1164 81.2% 26-May-96 01:14:18 +getfilenumber.c 1792 540 69.8% 01-Jun-96 21:11:00 +InfoToProcess.c 2393 732 69.4% 30-Apr-96 15:23:36 +makefilename.c 5219 903 82.6% 26-May-96 22:54:50 +phrasestring.c 12663 800 93.6% 29-May-96 17:36:02 +process.c 2012 447 77.7% 06-May-96 13:20:32 +readdataconfig.c 8091 1332 83.5% 29-May-96 21:52:34 +readhtml.c 8439 1067 87.3% 21-May-96 20:13:18 +readurlfile.c 1639 397 75.7% 30-May-96 22:11:10 +safe.c 10789 3011 72.0% 30-May-96 21:59:44 +tcpdl.c 4198 1345 67.9% 01-Jun-96 20:55:30 +tcpdl.h 6546 2668 59.2% 02-Jun-96 13:18:36 +tcpdl.readme 3255 690 78.8% 01-Jun-96 16:12:52 +testahref.c 2401 637 73.4% 29-May-96 17:26:54 +testimg.c 1198 452 62.2% 25-May-96 23:35:54 +testtitle.c 442 215 51.3% 02-Jun-96 12:26:10 testtcpdl 2656 724 72.7% 02-Jun-96 12:25:16 urllist 39280 18402 53.1% 02-Jun-96 13:18:50 tcpdl 6546 2668 59.2% 02-Jun-96 13:18:36 tcpdl.readme -------- ------- ----- --------- -------- 154026 43465 71.7% 02-Jun-96 14:16:44 25 files