IIIInnnnssssttttaaaalllllllliiiinnnnggg g ````````CCCC NNNNeeeewwwwssss'''''''' NNNNeee ttttwwwwoooorrrrkkkk NNNNeeeewwwwssss SSSSoooof fffttttwwwwaaaarrrreeee Geoff Collyer Department of Statistics University of Toronto This document describes the flow of network news within and between machines, what each component of C news does, and how to install C news. IIIInnnnttttrrrroooodddduuuuccccttttiiiioooon nn Network news (or _n_e_t_n_e_w_s for short) consists of a collection of messages formatted similarly to ARPAnet mail (see ARPA Inter- net RFC 1036 for details), widely spread. The logical network, imposed on top of various real networks, formed by the set of all interconnected sites exchanging network news is called ``Usenet'' and was formed in 1979, radiating out from Duke University. Net- news is propagated between cooperating machines by a flooding algorithm, with some loop prevention heuristics: each machine sends its neighbours news that the neighbours have (probably) not yet seen. Flow of netnews between machines may be achieved by use of any network which can transmit an arbitrary stream of (at least 7-bit) ASCII code, unmodified. If a network cannot transmit ASCII intact (e.g. BITNET), it is possible to encode netnews before transmission across the network and decode it upon recep- tion from the network. Since one cannot be certain that one's network neighbours will be up and reachable at all times, outgo- ing netnews must be queued, at least in the case of network trou- ble. To date, at least these networks, protocols and media have been used to transmit netnews correctly: UUCP, RS232, NNTP, Eth- ernet(Reg), the ARPA Internet, Datakit(Reg), ACSnet, magnetic tape, SMTP, and BITNET, though at least the last two require some form of encapsulation to avoid corruption of articles; SMTP because some common implementations get the newline-CRLF mappings wrong, thus throwing off byte counts in batches, and BITNET because of its Procrustean chopping, expanding, mapping, bashing and smashing of all data sent through it (sending lines of 80 or fewer characters of letters of either case and digits and plus and minus appears to be safe). - 2 - Netnews arrives via some network, which causes a command to be executed upon arrival (e.g. _r_n_e_w_s). _r_n_e_w_s typically pools the inbound netnews for later processing. Eventually (often within the hour or at the end of the business day), the input queue is run and the netnews is stored locally, typically under /_u_s_r/_s_p_o_o_l/_n_e_w_s, and queued for transmission to netn ws neigh- bours. Once stored on disk, netnews may be read by any of a col- lection of unprivileged news readers, including _c_a_t(1). _E_x_p_i_ _e is run typically each night to remove old netnews from disk. C News was originally written to provide a much faster and smaller, more robust, reliable and correct implementation of net- news software than B 2.11 news. We believe that C News is also faster, smaller and more robust than B 3.0 news. B 3.0 news has many more features; take that as you will. CCCC NNNNeeeewwwwssss CCCCoooommmmppppoooonnnn eeennnnttttssss.... _R_n_e_w_s invokes the input subsystem, which spools incoming netnews in its original form, as received, typically in compressed batches. Periodically, the input queue should be run by invoking _n_e_w_s_r_u_n, thus uncompressing any compressed input and feeding it to _r_e_l_a_y_n_e_w_s. _R_e_l_a_y_n_e_w_s files incoming netnews as articles on disk and initiates spooled transmission to other machines, often by simply writing the names of the disk files containing the articles on the ends of `batch' files of file names. Quite a bit of policy from RFC 1036 is embedded in _r_e_l_a_y_n_e_w_s. _i_n_e_w_s s a complex front-end for _r_e_l_a_y_n_e_w_s which implements much of the per- ite policy on news posting. _i_n_e_w_s is a fairly-slow shell script which is adequately fast for most sites, but it will need to be replaced by something more streamlined for sites which gateway mailing lists into netnews, for example. The output _b_a_t_c_h_e_r reads lists of file names and generates news batches (see RFC 1036 or _n_e_w_s(5) for the format) of prescribed sizes and queues the batches for transmission to other sites. The batcher is run asynchronously with _r_e_l_a_y_n_e_w_s, t pi- cally once an hour outside of business hours. _E_x_p_i_r_e is generally run once per night to remove from disk news articles older than a few days. _E_x_p_i_r_e can use different expiry criterion for different newsgroups and can archive arti- cles instead of removing them. _E_x_p_i_r_e also runs asynchronously with _r_e_l_a_y_n_e_w_s. There are many news readers. C News comes with a limited news reader (_r_e_a_d_n_e_w_s by Michael Rourke) sufficient to repla e long-winded /_e_t_c/_m_o_t_ds but you will want a heavy-duty news reader if you plan to read more than local news groups. We recommend _r_n by Larry Wall, available from your netnews neighbours or your nearby ccccoooommmmpppp....ssssoooouuuurrrrcccce eessss....uuuunnnniiiixxxx archive site. There are others: _v_n and _v_n_e_w_s are two. - 3 - PPPPrrrreeeeppppaaaarrrraaaattttiiiioooonnnn f fffoooorrrr IIIInnnnssssttttaaaallllllllaaaatt tiiiioooonnnn Netnews consumes a lot of disk space and often a lot of transmission time. Here are some relevant measurements regarding a full netnews feed as of the time of writing (January 1989), taken from _n_e_w_s._l_i_s_t_s: a day's netnews is about 3Mb and 1 400 articles in 450 newsgroups. Years ago, sites often kept 14 days of netnews on disk, but now many sites keep news for 3 to 5 days, thus allowing for the occasional long weekend. Thus a full news feed expired after 4 days will consume about 12Mb. Some people feel that news volume is doubling roughly every 16 months. If this is true, we can expect volume to increase by a factor of 10 in about 4 years to 30Mb per day or 115Mb for 4 days. It is thus wise planning to set aside a lot of disk space for netnews. There are two ways to cope with ever-increasing volumes of net- news: refuse to accept more newsgroups, or expire news after shorter intervals on disk. A current full feed takes just over 7 hours to transmit at 1200 baud, so for dial-up connections one clearly wants the fastest modems one can afford. Clearly, transmitting a full news feed is a non-trivial com- mitment of resources, so you may have some difficulty in finding a site willing to supply one. Such a site may in turn expect you to feed yet other sites. You will need to agree with your pros- pective netnews neighbour(s) upon transfer media, protocols and networks. Before proceeding to install C News, you should read this document through to the end, probably read the companion document _T_h_e _I_n_t_e_r_f_a_c_e _B_e_t_w_e_e_n _C _N_e_w_s _ _n_d _T_h_e _O_u_t_s_i_d_e _W_o_r_l_d, and possibly read selected items in the _C _N_e_w_s _I_m_p_l_e_m_e_n_t_o_r' s _N_o_t_e_b_o_o_k and the manual pages. You will need to assign a user id and group id to netnews (often new ids called ``news''); initialise these directories: NEWSCTL (typically ////uuuussssrrrr////lllliiiibbbb// //nnnneeeewwwwssss), NEWSBIN (////uuuussssrrrr//// llliiiibbbb////nnnneeeewwwwssssbbbbiiiinnnn), and NEWSARTS (////uuuussssrrrr////ssssppppoooooooo lll////nnnneeeewwwwssss); and install each subsystem. NEWSCTL and NEWSARTS are logically one subtree, defining a news data base, but are split for backward compatibility with older news software, particularly old news readers. NEWSBIN contains programs and shell scripts which might be common amongst machines sharing a common architecture (e.g. Sun 3's); NEWSCTL////bbbbiiiin nn may override these. The goal is to install the subsystems, integrate them into a working news system, and configure the news system to communicate with other news systems. There are a few key files that must exist before any serious attempt may be made to operate the news software. NEWSCTL////aaaacccc tttiiiivvvveeee is the list of newsgroups that this site knows (is willing to accept or individually reject), and must be owned by the NEWS userid (the userid that owns NEWSBIN////rrrreeeellllaaaayyyy ///rrrreeeellllaaaayyyynnnneeeewwwwssss, typicall _n_e_w_s). You will probably want to get your initial aaaacccctt ttiiiivvvveeee file from your upstream feed and edit it to suit the set of groups you wish to receive. Be sure to make the second field more than five - 4 - digits wide, by adding leading zeroes (ten digits is a conserva- tive width). NEWSCTL////ssssyyyyssss defines the newsgroups that this site is willing to accept and describes how they are to be forwarded to its neighbours. NEWSCTL////sssseeeerrrrvvvveeeerrrr c ntains the hostname of your file server, if you have multiple machines sharing news via a network file system. NEWSCTL////wwwwhhhhooooaaaammmmiiii similarly contains the name by which a cluster of machines sharing news is to be known for purposes of news. NEWSCTL////mmmmaaaaiiiillllnnnnaaaamm mmeeee is optional and contains the full (possibly dotted) name by which your cluster is known for purposes of mail. NEWSCTL////oooorrrrggggaaaannnniiiiss ssaaaattttiiiioooonnnn (or NEWSCTL////oooorrrrgggga aannnniiiizzzzaaaattttiiiioooonnnn if you insist) contains the name of your organisation, which will be copied into the OOOOrrrrggggaaaannnniiiizzzzaaaa tttiiiioooonnnn:::: header of articles posted locally, by default. NEWSCTL////mmmmaaaaiiiillllppppaaaat ttthhhhssss defines mail routes to machines which contain aliases for postings to moderated news- groups. NEWSCTL////lllloooogggg, NEWSCTL////eeeerrr rrrrrlllloooogggg, NEWSCTL////hhhhiiiissssttttoo oorrrryyyy, NEWSCTL////hhhhiiiissssttttoooorrrryyyy....ddddi iirrrr, and NEWSCTL////hhhhiiiissssttttoooorrrryyy y....ppppaaaagggg must exist and be owned by the NEWS userid. Tentative versions of all these files are built by the installation procedures, but it is quite likely that you'll have to edit some of them. CCCC NNNNeeeewwwwssss IIIInnnnssssttttaaaallll lllaaaattttiiiioooonnnn Proceed to the ccccoooonnnnffff directory of the distribution. There is a major shell file here, named bbbbuuuuiiiilllldddd, that will interrogate you at length and construct shell files to do the real work. You may need to do ``chmod +x build'' before running it, to make it exe- cutable. You will probably need your system's manuals on hand to answer bbbbuuuuiiiilllldddd's questions. Another terminal (or another window, on a bitmap display) would also be useful. You'd best be prepared to take notes, also, as bbbbuuuuiiiilllldddd will occa ionally suggest that something be checked when you're done. BBBBuuuuiiiilllldddd itself does not alter any files or per orm any instal- lation chores: all it does is create shell files in the ccccoooonn nffff directory. If you already know something about news software, or are merely suspicious that bbbbuuuuiiiilllldddd hasn't done e erything right, you should probably read the shell files before running them. There are four of them: ddddooooiiiitttt....rrrroooo oootttt, ddddooooiiiitttt....bbbbiiiinnnn, dddd oooiiiitttt....nnnneeeewwwwssss, and aaaaggggaaaaiiiinnnn....rrrrooooooootttt. DDDDooooiiiitttt....rrrrooooooootttt sets up th major directories for news, and sets their ownerships correctly. It typically will have to be run as _r_o_o_t to have adequate permissions for all of this. It is brief. DDDDooooiiiitttt....bbbbiiiinnnn does most of the ork of building and installing the programs. It should be run as the user who owns the distri- bution directories and will own the executable programs under NEWSBIN. DDDDooooiiiitttt....nnnneeeewwwwssss does some ther small chores and installs control files. If any of the control files already exist, it will com- plain and refuse to overwrite them, as a safety precaution. It - 5 - should be run as the owner of the news files. Since many of the files it is installing are built by ddddooooiiiitttt....bbbbi iiinnnn, that should be run first. Finally, aaaaggggaaaaiiiinnnn....rrrroooooooot ttt tends to ownership and permission changes on a few programs that need to run set-userid. It requires the ability to change ownerships and to set permissions on the files afterwards, which usually means running it as _r_o_o_t. It too is brief. There are undoubtedly strange systems out there that bbbbuuuuiii illlldddd and friends are not smart enough to cope with. In such cases it will be necessary to edit the shell files before running them, or to use them as guides and do the work by hand. In particular, systems that require strange options in MMMMaaaakkkkeeeeffffii illlleeees will need to have those inserted by hand. If you want to test pieces of C News without installing them, some (not all) of the subsystems have a ``make r'' feature that runs a regression test. Note: almost all of these require that NEWSCTL////bbbbiiiinnnn////ccccoooonnnnffffii iigggg, or its local equivalent, be in place already so that shell files can find out what PATH (etc.) they should be using. Note that it is easy to build a test news system which is completely independent of other existing news systems on the same machine, or to build one which shares its NEWSBIN with another C news system, or shares input of articles (e.g. running an old B news system and a new C news system off the same stream of input until you are confident that your new C news system is operating to your satisfaction). See _s_u_b_s_t(1) for the mechanism which per- mits quickly changing all the places that know the location of NEWSCTL////bbbbiiiinnnn////ccccoooonnnnffffiiiig gg.... After that, edit this news system's NEWSCTL////bbbbiiiinnnn////ccccoooonnnnffffiiiig gg and things should be set up for separate existence. FFFFiiiirrrrsssstttt NNNNeeeewwwwssss When you arrange to get a news feed from your neighbor, you should also ask him to send you the current set of articles in the newsgroup nnnneeeewwwwssss....aaaannnnnnnnoooou uunnnncccceeee....nnnneeeewwwwuuuusssseeeerrr ssss. Several of these are very important reading for people who are new to the net. UUUUnnnnuuuussssuuuuaaaallll SSSSyyyysssstttte eeemmmmssss We believe that C News runs fine on 16-bit machines, but it hasn't been tested very thoroughly on them lately. It will not perform quite as well with the more limited space. Machines with very old compilers can be a headache. There are some hooks in hhhh////nnnneeeewwwwssss....hhhh for doing without ``void'' and ``unsigned long'', two particular problem areas, but they have to be arranged manually; bbbbuuuuiiiilllldddd does not know about t em. - 6 - Some very old systems cannot do _s_e_t_u_i_d(_g_e_t_e_u_i_ ()), which makes it impossible for a daemon to make directories and get the ownership right. We provide a small program, sssseeeettttnnnneee ewwwwssssiiiiddddssss, to run setuid-root. RRRReeeellllaaaayyyynnnneeeewwwwssss knows about it and invokes it if _s_e_t_u_i_d(_g_e_t_e_u_i_d()) fails; it then sets permissions co rectly and re-invokes rrrreeeellllaaaayyyynnnneeeewwwwssss. he code is short enough to be read and understood in full, so that the suspicious system administrator can be sure that this setuid-root program is not going to do something awful.