BRIK(1) RISC/os Reference Manual BRIK(1) NAME brik - calculate 32-bit CRC SYNOPSIS brik -h brik -gcGCbafvsqWHT _f_i_l_e ... DESCRIPTION _B_r_i_k generates and verifies 32-bit CRC values (checksums). It is designed to generate CRCs for text files that are the same on all computer systems that use the ASCII character set, provided each text file is in the usual text file for- mat for each system. _B_r_i_k will also optionally use binary CRCs calculated using every byte in a file. Such binary CRCs are portable across systems for binary files that are moved from system to system without any newline conversion. _B_r_i_k can be asked to decide by examining each file whether to calculate a text mode or binary mode CRC for it. Changes from version 1.0 are summarized at the end of this document. The general usage format is: brik -gcGCbafvsqWHT [ _f_i_l_e ] ... The brackets mean that _f_i_l_e, which is the name of a file, is optional. The three dots indicate that more than one filename may be typed (separated by blanks). Exactly one of the options -c, -C, -g, -G, or -h, is required. The -h option gives a help screen. In addition to -h, the _B_r_i_k options available (as they appear on the help screen) are: -g look for _C_h_e_c_k_s_u_m: header, generate CRC for rest of file -c get CRC from header, verify CRC of rest of file -G generate CRC for entire file (add -b for binary files) -C verify all file CRCs from output of -G (-f is not needed) -b use binary mode -- read file byte by byte, not line by line -a automatically decide whether each file is text or binary -f read filenames (wildcards ok) from specified files Printed 0/0/0 Jul 8, 1989 1 BRIK(1) RISC/os Reference Manual BRIK(1) -v be verbose, report all results (else only errors are reported) -s be silent, say nothing, just return status code -q be quiet, don't print header for -G -W after generating CRC with -g, write it to original header -H after generating CRC with -g, print header to stdout -T include trailing empty lines, normally ignored (text mode only) VERIFYING CRC HEADERS The primary intended use of _B_r_i_k is to verify _C_h_e_c_k_s_u_m: headers in Usenet postings and in C and Pascal source files. A _C_h_e_c_k_s_u_m: header looks like this: Checksum: 9485762645b (verify with "brik") The word _C_h_e_c_k_s_u_m: must occur at the beginning of a line. It is followed by a colon, an optional blank, a ten-digit decimal 32-bit CRC, and any arbitrary optional string such as the comment shown above. The CRC value may be followed by a one-character suffix identifying the type of the CRC. These suffixes are described later. When _B_r_i_k is invoked with the syntax brik -c _f_i_l_e ... it will search for the _C_h_e_c_k_s_u_m: header in each specified _f_i_l_e, read the CRC value from that header, calculate the CRC-32 for all lines in the file (except trailing empty lines) that occur *after* the header line, and report an error if the two values do not match. CALCULATING CRC HEADERS The command brik -g _f_i_l_e ... tells _B_r_i_k to look for the _C_h_e_c_k_s_u_m: header in each speci- fied _f_i_l_e, calculate the CRC of the lines in the file fol- lowing the header, and print the CRC and filename without changing the file in any way. You can also ask _B_r_i_k to update the _C_h_e_c_k_s_u_m: header with the following command: Printed 0/0/0 Jul 8, 1989 2 BRIK(1) RISC/os Reference Manual BRIK(1) brik -gW _f_i_l_e ... This causes _B_r_i_k to update the _C_h_e_c_k_s_u_m: header in the file with the newly-calculated CRC. If there is not enough space for the CRC in the existing header, _B_r_i_k reports an error and does not change the existing header. To initially add a _C_h_e_c_k_s_u_m: header to a file, insert a line of the following form into the file with a text editor: Checksum: XXXXXXXXXX -- this is an optional comment The word _C_h_e_c_k_s_u_m must be at the beginning of a line. The ten `X' characters shown can be any ten characters. They will be later replaced by the calculated CRC value. They do not need to be `X'. The comment following them is optional and can be any arbitrary string of characters after the CRC field, separated from it by blanks. As an example, in a C source file called _m_a_i_n._c, you might have: /* Checksum: XXXXXXXXXX (verify or update with brik) */ Once a header like this exists in the file, invoke _B_r_i_k as follows: brik -gW main.c This will cause the ten `X' characters to be replaced with the calculated checksum, giving something like this: /* Checksum: 1922837484 (verify or update with brik) */ Later you can use the command brik -c main.c to verify that the contents of the file following the header have the same CRC as the stored value. If _B_r_i_k is invoked with the -c or -g options and it cannot find a _C_h_e_c_k_s_u_m: header in a file, it prints a row of ques- tion marks. If it is invoked with the -gW option and it does not find enough character positions after the _C_h_e_c_k_s_u_m: string to hold the CRC, it does not fill in the CRC and prints an error message. _B_r_i_k can be asked to generate a complete _C_h_e_c_k_s_u_m: header but print it to standard output instead of writing it to the file. Do this by adding the -H option. If both -W and -H Printed 0/0/0 Jul 8, 1989 3 BRIK(1) RISC/os Reference Manual BRIK(1) are given, the _C_h_e_c_k_s_u_m: header will be written both to the file and to standard output. WHOLE-FILE CRCS A "whole-file" CRC calculation is done for the entire con- tents of a file, without looking for a _C_h_e_c_k_s_u_m: header. The command brik -G _f_i_l_e ... asks _B_r_i_k to do this calculation for all specified files. The output from this command is a list of files and their whole-file CRCs. It is sent to the standard output stream, which in most cases is your screen. The output should be saved in a file by redirecting it. For example, the command brik -G *.h *.c > crc.lst on an MS-DOS system might yield the following in the output file _c_r_c._l_s_t: # CRC-32 filename # ------ -------- 2566277976 getopt.c 100594287 brik.c 1151475469 vms.c 3929503738 turboc.c 2424271922 addbfcrc.c 1943472856 assert.h 2601923477 brik.h The output of the -G option is in free format. The output file may be edited by hand. Empty lines and lines beginning with '#' will be ignored by _B_r_i_k when it is later asked to read this file. This list of filenames and whole-file CRCs may be verified by a command like this: brik -C crc.lst This makes _B_r_i_k read the file _c_r_c._l_s_t, get the CRCs and filenames from it, do the CRC calculation again for each file, and report an error if it does not match the CRC recorded inside _c_r_c._l_s_t. **IX and MS-DOS users (and others who can pipe the output of one command into another) could use a command like the fol- lowing to see if _B_r_i_k'_s -G and -C options are working: Printed 0/0/0 Jul 8, 1989 4 BRIK(1) RISC/os Reference Manual BRIK(1) brik -G file ... | brik -C This invokes "brik -G" on some files, sending the output to a pipe where it becomes the input of "brik -C". If no filename is specified, _B_r_i_k reads from standard input, so "brik -C" will read from the pipe and concurrently verify all the CRCs that are being generated by "brik -G". Whole-file CRCs are normally generated in text mode, in which a file is treated as lines of text. You can also gen- erate whole-file CRCs in binary mode. See below. BINARY VERSUS TEXT MODE _B_r_i_k can work in two different modes. The most common mode, used unless you specify otherwise, is text mode. In this mode _B_r_i_k reads all files line by line, and forces each line of text to be terminated by a newline character of constant value before doing a CRC calculation. Thus, no matter what newline character is used by your system, the CRC calculation will be unaffected. This means that whether your operating system uses linefeeds to terminate lines (e.g. **IX), or a carriage return and a linefeed (e.g. MS- DOS) or a carriage return only (e.g. Macintosh) or nothing but a record length field (e.g. VAX/VMS), the CRC calcula- tion in text mode gives the same results. If a file is not intended to be a text file, the text mode CRC calculated by _B_r_i_k may not be reliable and may be dif- ferent on different systems. If _B_r_i_k is calculating a text mode CRC on a file that appears to contain binary data, it still calculates the text-mode CRC but adds a "*" character after the CRC to indicate to warn the user. _B_r_i_k can be asked to operate in binary mode by adding a -b option. Binary mode is applicable only to the -G command, which acts on whole files. Thus brik -G _f_i_l_e ... finds whole-file CRCs of all specified files in text mode, while brik -Gb _f_i_l_e ... does the same in binary mode. In binary mode _B_r_i_k simply reads and calculates the CRC for all bytes in a file. If a file is moved from one system to another without any newline conversion or any other changes, _B_r_i_k should calculate the same binary mode CRC for it on both systems. Printed 0/0/0 Jul 8, 1989 5 BRIK(1) RISC/os Reference Manual BRIK(1) The output from "brik -Gb" includes a trailing b suffix in each CRC, so "brik -C" will correctly use binary mode for such CRCs and it is never necessary to type "brik -Cb" instead. In its auto-check mode, _B_r_i_k will itself examine each file and determine whether it is a text or binary file and calcu- late a CRC accordingly. To do this, use the -a option. Although _B_r_i_k can determine the type of each file with a high degree of reliability, it is still possible for it to come to a wrong conclusion about some files. The output from "brik -Ga" will include a trailing b charac- ter in the CRC for those files that appeared to be binary to _B_r_i_k. You may find both text and binary CRCs in the output. TRAILING EMPTY LINES The normal behavior of _B_r_i_k in text mode is to ignore any trailing empty lines in a file. An empty line is a line that contains nothing, not even blanks or tabs. (Just hit- ting a carriage return when you are editing a text file usu- ally produces an empty line.) If manipulating a text file causes trailing empty lines to be added or deleted, the CRC calculated by _B_r_i_k will be unaffected. You can override this if necessary with the -T option, which makes _B_r_i_k include trailing empty lines in the CRC calculation. For example, brik -gWT will update the _C_h_e_c_k_s_u_m: header with a CRC that includes all lines in a text file. Similarly brik -GT will find whole-file CRCs that include all lines in a text file. When _B_r_i_k is given the -T option, it adds a T suffix to each generated CRC. Then, when the CRC is verified with -c or -C, _B_r_i_k will correctly include trailing empty lines when needed without having to be explicitly told to do so. In binary mode _B_r_i_k always reads all bytes in a file, so it does not ignore trailing empty lines, and the -T switch is not needed. The effects of the -T and -b options are mutually exclusive. If both are given, whichever is used later overrides the first. So -bT is equivalent to -T and -Tb is equivalent to -T. Printed 0/0/0 Jul 8, 1989 6 BRIK(1) RISC/os Reference Manual BRIK(1) FILENAME CONVENTIONS Under MS-DOS and VAX/VMS, wildcards are allowed on the com- mand line and will be expanded by _B_r_i_k. Under **IX, wild- cards are expected to be expanded by the shell and are not expanded by _B_r_i_k. If no filename is given, _B_r_i_k reads its standard input. By default this is keyboard input, but it is expected that input will usually be redirected to come from a file or a pipe. Also, if a filename is explicitly specified as a dash ("-"), it stands for standard input. The following five commands (valid under the **IX operating system) brik -c myfile # "myfile" brik -c < myfile # stdin = "myfile" cat myfile | brik -c # stdin = a pipe brik -c - < myfile # "-" = stdin = "myfile" cat myfile | brik -c - # "-" = stdin = a pipe all have the effect of verifying the _C_h_e_c_k_s_u_m: header in the file _m_y_f_i_l_e. Standard input can also be read when using the -C option. For example, suppose we have already given the command brik -G *.c *.h > crc.lst to create a file called "crc.lst" that contains a list of whole-file CRCs. We can now verify the integrity of these files with any of the following commands: brik -C crc.lst # "crc.lst" brik -C < crc.lst # stdin = "crc.lst" brik -C - < crc.lst # "-" = stdin = "crc.lst" cat crc.lst | brik -C # stdin = a pipe cat crc.lst | brik -C - # "-" = stdin = a pipe INDIRECT FILE NAMING The -f option allows you to have filenames in a list rather than on the command line. For example, suppose you want to generate whole-file CRCs of files whose names are selected by some other program. You could use _f_i_n_d (under **IX) or _S_t_u_f_f (under MS-DOS) to generate a list of filenames. When the -f option is given, _B_r_i_k reads filenames, one per line, from the file(s) specified. Thus you could do the follow- ing: find . -mtime +3 -print > flist brik -Gf flist > crc.lst Printed 0/0/0 Jul 8, 1989 7 BRIK(1) RISC/os Reference Manual BRIK(1) The first command asks _f_i_n_d to generate a list of names of all files that were modified more than 3 days ago, and sends the output to the file _f_l_i_s_t. The contents of _f_l_i_s_t might now look like this, as an example: ./sez.doc ./brik.doc ./stuff.doc ./looz.doc The second command above asks _B_r_i_k to read the file called _f_l_i_s_t, get a list of filenames from it, one per line, and generate the whole-file CRC of each of these files. We additionally redirect the output of "brik -Gf" to the file called _c_r_c._l_s_t. Given the above contents of _f_l_i_s_t, the fol- lowing two commands are exactly equivalent: brik -Gf flist >crc.lst brik -G ./sez.doc ./brik.doc ./stuff.doc ./looz.doc >crc.lst The advantage of the -f option is that once you have filenames in a file, you need not type them again at the command line. The -f option also allows you to feed _B_r_i_k through a pipe filenames generated by another program. Con- sider this command: find . -mtime +3 -print | brik -Gf - > crc.lst Under **IX this concurrently invokes both _f_i_n_d and _B_r_i_k. As _f_i_n_d generates a filename and sends it to its standard out- put (a pipe), _B_r_i_k reads the filename from its standard input and generates its whole-file CRC. The following pipe- line find . -mtime +3 -print | brik -Gf - | brik -C - invokes _f_i_n_d to generate filenames, _B_r_i_k to print the whole-file CRCs of those files, and _B_r_i_k again to immedi- ately verify them. Under MS-DOS and VAX/VMS (and any other operating system under which _B_r_i_k has been installed to expand wildcards itself), a file specified by the -f option can itself con- tain wildcards in the filenames. For example, suppose the file "wildfile" contains the following lines: /bin/*.exe /bin/*.com /doc/*.doc Now if we invoke _B_r_i_k with the command Printed 0/0/0 Jul 8, 1989 8 BRIK(1) RISC/os Reference Manual BRIK(1) brik -Gf wildfile it will read filespecs from "wildfile," expand wildcards in each filespec, and generate whole-file CRCs for all matching files. If you are checking whole-file CRCs with the -C option, you do not normally need to use the -f option. When used with -C, the -f option introduces an additional level of file naming indirection. For example, the command brik -C crc.lst takes a list of CRCs and filenames from "crc.lst" and veri- fies them. However, the command brik -Cf master.lst does not do the same thing. Instead, it takes a list of files from "master.lst," looks inside each of _t_h_o_s_e files for a list of CRCs and filenames, and verifies them. As an example of the use of -Cf, consider this sequence: brik -Gb /bin/*.exe > exelist brik -Gb /bin/*.com > comlist brik -GT /doc/*.doc > doclist brik -G /doc/*.man > manlist Now we have four files called "exelist," "comlist," "doclist," and "manlist" containing whole-file CRCs of vari- ous types. We can test the integrity of files listed in these in four separate commands like this: brik -C exelist brik -C comlist brik -C doclist brik -C manlist But we could also do this in a single step by first creating a file called "biglist" that contains the names of these four files: exelist comlist doclist manlist and then use -Cf thus: brik -Cf biglist Printed 0/0/0 Jul 8, 1989 9 BRIK(1) RISC/os Reference Manual BRIK(1) This causes _B_r_i_k to read filenames from "biglist," look inside each of those files ("exelist," "comlist," "doclist," and "manlist") and check the CRCs found there. A **IX exam- ple to do the same thing in a more compact way might be: cat exelist comlist doclist manlist | brik -Cf - The above examples are somewhat contrived, however. We could also use the following command: brik -C exelist comlist doclist manlist SILENT VERSUS VERBOSE VERSUS QUIET _B_r_i_k accepts options -s, -q, and -v that control the degree of verbosity. Normally _B_r_i_k prints a message only if it detects an error. For example, if you give the command brik -c *.c *.h and _B_r_i_k finds that all _C_h_e_c_k_s_u_m: headers contain CRCs that match the calculated value, it prints nothing. The -v switch makes _B_r_i_k print a message for each file tested that contains the word "ok" if stored and calculated CRCs matched and "BAD" if they did not. In all messages reporting file CRCs, _B_r_i_k prints the actual CRC it calculated, or a row of question marks if it could not calculate one. _B_r_i_k fails to calculate a CRC if it is trying to calculate a header-based CRC (commands -c and -g) but does not find a _C_h_e_c_k_s_u_m: header in a file. The -s switch tells _B_r_i_k to be silent. This will cause nothing to be printed if _B_r_i_k does not find a _C_h_e_c_k_s_u_m header or if CRCs don't match. However, the status code on exit is still set. The -s option is useful for invoking _B_r_i_k from within a shell script (**IX) or a batch file (MS- DOS) and later testing the error code it returned when it exited. The -s switch does not suppress the error message printed if _B_r_i_k is given a filename and the file cannot be found. For example, if the command brik -cs myfile is given but "myfile" does not exist, _B_r_i_k will print an error message even though the -s option was given. Printed 0/0/0 Jul 8, 1989 10 BRIK(1) RISC/os Reference Manual BRIK(1) The -q switch makes _B_r_i_k slightly quieter. Its only effect is to suppress the introductory comments that are otherwise generated by the -C option. VAX/VMS BUGS o Under VAX/VMS, file manipulation is unpredictable and the VAX/VMS C runtime library in particular is very unreliable. One result is that under VAX/VMS, the -gW option will work only for stream-LF files, which are the only type of file that VAX/VMS seems to handle correctly. Fortunately, the -c option will correctly work for VAX/VMS standard text files. o The VAX/VMS implementation of _B_r_i_k simply ignores filenames that correspond to nonexistent files instead of giving an error message. VMS users are invited to fix this bug (probably somewhere in the _n_e_x_t_f_i_l_e() function in the file _v_m_s._c) and send me the fix. o To avoid annoying error messages, _B_r_i_k under VAX/VMS always exits with a status code of 1. o Due to a problem in the VAX/VMS command interpreter, if any uppercase option characters are typed (e.g. -C, -G, -T), _B_r_i_k will not recognize them as uppercase unless they are enclosed in double quotes. An example of a correct command under VAX/VMS: brik "-GT" *.* An example of a command that won't work: brik -GT *.* USAGE HINTS This section discusses some ways of using _B_r_i_k. - _B_r_i_k is currently used to create and verify checksums of articles posted on the Usenet newsgroup comp.binaries.ibm.pc. While reading Usenet news with _r_n, for example, you can verify the _C_h_e_c_k_s_u_m: header of such an article with the command: | brik -cv This feeds the current article to _B_r_i_k as standard input and asks it to verbosely verify the checksum. - C and Pascal source files that are being distributed can be given their own _C_h_e_c_k_s_u_m: headers. The Printed 0/0/0 Jul 8, 1989 11 BRIK(1) RISC/os Reference Manual BRIK(1) advantage of this over listing CRCs separately is that each file contains its checksum no matter where it goes. The recipient can easily verify that each source file was received intact. This is especially valuable when source files are being sent by electronic mail through an IBM mainframe-based network, such as BITNET, that can cause corruption of text files. Create the _C_h_e_c_k_s_u_m header with a text editor and ini- tialize its value with the command: brik -gW file ... The recipient can verify the checksum with the command: brik -cv file ... - To keep track of any unexpected filesystem corruption or unauthorized changes to system files, _B_r_i_k can be used to create a list of checksums. This should be done using the -b option (all binary checksums) or the -a option (use text or binary as appropriate). Under **IX this can be done with a command like this one: find /bin /usr/bin /lib /usr/lib /usr/new \ -type f -print | \ grep -v '^/usr/bin/beta' | \ grep -v '^/usr/new/lib/news' | \ brik -Gbf - > crc.list This example uses _f_i_n_d to print the pathnames of cer- tain files, _g_r_e_p to filter out certain directory sub- trees, and then _B_r_i_k to print checksums of all selected files. Periodically a background job can be run to compare the file list against current files and report any discrepancies: brik -C crc.lst | mail -s 'brik output' postmaster Under MS-DOS, create a list of checksums with the help of _S_t_u_f_f, a _f_i_n_d-like file finder program that is dis- tributed with _z_o_o 2.01: stuff /bin /msdos /turboc | brik -Gaf - > crc.list This is a simple mechanism that can be used to detect unexpected file changes caused by hardware problems or malicious programs such as viruses and worms. Be warned, however, that since _B_r_i_k uses a published CRC algorithm, very clever deviant programs are possible that could change a file without affecting its CRC. Printed 0/0/0 Jul 8, 1989 12 BRIK(1) RISC/os Reference Manual BRIK(1) To avoid too many false alarms, only files that do not change much (such as system binaries) should be selected. CHANGES Changes from version 1.0 are as follows. - The CRC calculation has been recoded in 8086 assembly code suitable for Turbo C/MS-DOS. As a result _B_r_i_k _2._0 under MS-DOS is much faster than version 1.0. - The new -a flag makes _B_r_i_k automatically determine for each file whether it is text or binary and calculate the CRC accordingly - When _B_r_i_k is asked to calculate a text mode CRC but the file appears to be binary, it now does not print a warning message but instead adds a "*" suffix to the calculated CRC to indicate that the CRC may not be reliable. - The new -H flag makes _B_r_i_k print the formatted _C_h_e_c_k_- _s_u_m: header to standard output. - Detection of binary files is now more reliable under MS-DOS. If a control Z character occurs near the beginning of a binary file, _B_r_i_k will not be fooled by the apparent end-of-file but will in most cases reli- ably detect that the file is binary. - The new -q flag suppresses the introductory comments from the output of the -G command. BUGS _B_r_i_k is designed to work with computer systems that use the 7-bit ASCII character set and 8-bit bytes with the eighth (parity) bit set to zero. _B_r_i_k has not been tested with computer systems that use ASCII characters with the eighth bit set or those that use EBCDIC characters. Although it will calculate a CRC on such machines, the probability of this CRC being the same as the one calculated on machines that use 7-bit ASCII is approxi- mately 0.00000000023. DIAGNOSTICS Error messages are intended to be self-explanatory. The exit status returned is 1 if _B_r_i_k was invoked with incorrect arguments, else it is the number of files found with missing or invalid CRCs, but not greater than an operating-system-dependent maximum value. Printed 0/0/0 Jul 8, 1989 13 BRIK(1) RISC/os Reference Manual BRIK(1) COPYRIGHT Both this documentation and _B_r_i_k are Copyright 1989 Rahul Dhesi, all rights reserved. Permission is granted to copy, use, and distribute for any commercial or noncommercial pur- pose in accordance with the requirements of version 1.0 of the _G_N_U _G_e_n_e_r_a_l _P_u_b_l_i_c _l_i_c_e_n_s_e. Note: This software has not been endorsed by the Free Software Foundation, the creator of the GNU license, and I am not affiliated with that organization. AUTHOR Rahul Dhesi UUCP: {iuvax,pur-ee}!bsu-cs!dhesi Internet: dhesi@bsu-cs.bsu.edu Printed 0/0/0 Jul 8, 1989 14