This is Info file gawk.info, produced by Makeinfo version 1.67 from the input file /ade-src/fsf/gawk/doc/gawk.texi. INFO-DIR-SECTION Programming Languages START-INFO-DIR-ENTRY * Gawk: (gawk.info). A Text Scanning and Processing Language. END-INFO-DIR-ENTRY This file documents `awk', a program that you can use to select particular records in a file and perform operations upon them. This is Edition 1.0.2 of `The GNU Awk User's Guide', for the 3.0.2 version of the GNU implementation of AWK. Copyright (C) 1989, 1991, 92, 93, 96 Free Software Foundation, Inc. Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies. Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one. Permission is granted to copy and distribute translations of this manual into another language, under the above conditions for modified versions, except that this permission notice may be stated in a translation approved by the Foundation.  File: gawk.info, Node: Group Functions, Next: Library Names, Prev: Passwd Functions, Up: Library Functions Reading the Group Database ========================== Much of the discussion presented in *Note Reading the User Database: Passwd Functions., applies to the group database as well. Although there has traditionally been a well known file, `/etc/group', in a well known format, the POSIX standard only provides a set of C library routines (`' and `getgrent') for accessing the information. Even though this file may exist, it likely does not have complete information. Therefore, as with the user database, it is necessary to have a small C program that generates the group database as its output. Here is `grcat', a C program that "cats" the group database. /* * grcat.c * * Generate a printable version of the group database * * Arnold Robbins, arnold@gnu.ai.mit.edu * May 1993 * Public Domain */ #include #include int main(argc, argv) int argc; char **argv; { struct group *g; int i; while ((g = getgrent()) != NULL) { printf("%s:%s:%d:", g->gr_name, g->gr_passwd, g->gr_gid); for (i = 0; g->gr_mem[i] != NULL; i++) { printf("%s", g->gr_mem[i]); if (g->gr_mem[i+1] != NULL) putchar(','); } putchar('\n'); } endgrent(); exit(0); } Each line in the group database represent one group. The fields are separated with colons, and represent the following information. Group Name The name of the group. Group Password The encrypted group password. In practice, this field is never used. It is usually empty, or set to `*'. Group ID Number The numeric group-id number. This number should be unique within the file. Group Member List A comma-separated list of user names. These users are members of the group. Most Unix systems allow users to be members of several groups simultaneously. If your system does, then reading `/dev/user' will return those group-id numbers in `$5' through `$NF'. (Note that `/dev/user' is a `gawk' extension; *note Special File Names in `gawk': Special Files..) Here is what running `grcat' might produce: $ grcat -| wheel:*:0:arnold -| nogroup:*:65534: -| daemon:*:1: -| kmem:*:2: -| staff:*:10:arnold,miriam,andy -| other:*:20: ... Here are the functions for obtaining information from the group database. There are several, modeled after the C library functions of the same names. # group.awk --- functions for dealing with the group file # Arnold Robbins, arnold@gnu.ai.mit.edu, Public Domain # May 1993 BEGIN \ { # Change to suit your system _gr_awklib = "/usr/local/libexec/awk/" } function _gr_init( oldfs, oldrs, olddol0, grcat, n, a, i) { if (_gr_inited) return oldfs = FS oldrs = RS olddol0 = $0 FS = ":" RS = "\n" grcat = _gr_awklib "grcat" while ((grcat | getline) > 0) { if ($1 in _gr_byname) _gr_byname[$1] = _gr_byname[$1] "," $4 else _gr_byname[$1] = $0 if ($3 in _gr_bygid) _gr_bygid[$3] = _gr_bygid[$3] "," $4 else _gr_bygid[$3] = $0 n = split($4, a, "[ \t]*,[ \t]*") for (i = 1; i <= n; i++) if (a[i] in _gr_groupsbyuser) _gr_groupsbyuser[a[i]] = \ _gr_groupsbyuser[a[i]] " " $1 else _gr_groupsbyuser[a[i]] = $1 _gr_bycount[++_gr_count] = $0 } close(grcat) _gr_count = 0 _gr_inited++ FS = oldfs RS = oldrs $0 = olddol0 } The `BEGIN' rule sets a private variable to the directory where `grcat' is stored. Since it is used to help out an `awk' library routine, we have chosen to put it in `/usr/local/libexec/awk'. You might want it to be in a different directory on your system. These routines follow the same general outline as the user database routines (*note Reading the User Database: Passwd Functions.). The `_gr_inited' variable is used to ensure that the database is scanned no more than once. The `_gr_init' function first saves `FS', `RS', and `$0', and then sets `FS' and `RS' to the correct values for scanning the group information. The group information is stored is several associative arrays. The arrays are indexed by group name (`_gr_byname'), by group-id number (`_gr_bygid'), and by position in the database (`_gr_bycount'). There is an additional array indexed by user name (`_gr_groupsbyuser'), that is a space separated list of groups that each user belongs to. Unlike the user database, it is possible to have multiple records in the database for the same group. This is common when a group has a large number of members. Such a pair of entries might look like: tvpeople:*:101:johny,jay,arsenio tvpeople:*:101:david,conan,tom,joan For this reason, `_gr_init' looks to see if a group name or group-id number has already been seen. If it has, then the user names are simply concatenated onto the previous list of users. (There is actually a subtle problem with the code presented above. Suppose that the first time there were no names. This code adds the names with a leading comma. It also doesn't check that there is a `$4'.) Finally, `_gr_init' closes the pipeline to `grcat', restores `FS', `RS', and `$0', initializes `_gr_count' to zero (it is used later), and makes `_gr_inited' non-zero. function getgrnam(group) { _gr_init() if (group in _gr_byname) return _gr_byname[group] return "" } The `getgrnam' function takes a group name as its argument, and if that group exists, it is returned. Otherwise, `getgrnam' returns the null string. function getgrgid(gid) { _gr_init() if (gid in _gr_bygid) return _gr_bygid[gid] return "" } The `getgrgid' function is similar, it takes a numeric group-id, and looks up the information associated with that group-id. function getgruser(user) { _gr_init() if (user in _gr_groupsbyuser) return _gr_groupsbyuser[user] return "" } The `getgruser' function does not have a C counterpart. It takes a user name, and returns the list of groups that have the user as a member. function getgrent() { _gr_init() if (++gr_count in _gr_bycount) return _gr_bycount[_gr_count] return "" } The `getgrent' function steps through the database one entry at a time. It uses `_gr_count' to track its position in the list. function endgrent() { _gr_count = 0 } `endgrent' resets `_gr_count' to zero so that `getgrent' can start over again. As with the user database routines, each function calls `_gr_init' to initialize the arrays. Doing so only incurs the extra overhead of running `grcat' if these functions are used (as opposed to moving the body of `_gr_init' into a `BEGIN' rule). Most of the work is in scanning the database and building the various associative arrays. The functions that the user calls are themselves very simple, relying on `awk''s associative arrays to do work. The `id' program in *Note Printing Out User Information: Id Program., uses these functions.  File: gawk.info, Node: Library Names, Prev: Group Functions, Up: Library Functions Naming Library Function Global Variables ======================================== Due to the way the `awk' language evolved, variables are either "global" (usable by the entire program), or "local" (usable just by a specific function). There is no intermediate state analogous to `static' variables in C. Library functions often need to have global variables that they can use to preserve state information between calls to the function. For example, `getopt''s variable `_opti' (*note Processing Command Line Options: Getopt Function.), and the `_tm_months' array used by `mktime' (*note Turning Dates Into Timestamps: Mktime Function.). Such variables are called "private", since the only functions that need to use them are the ones in the library. When writing a library function, you should try to choose names for your private variables so that they will not conflict with any variables used by either another library function or a user's main program. For example, a name like `i' or `j' is not a good choice, since user programs often use variable names like these for their own purposes. The example programs shown in this chapter all start the names of their private variables with an underscore (`_'). Users generally don't use leading underscores in their variable names, so this convention immediately decreases the chances that the variable name will be accidentally shared with the user's program. In addition, several of the library functions use a prefix that helps indicate what function or set of functions uses the variables. For example, `_tm_months' in `mktime' (*note Turning Dates Into Timestamps: Mktime Function.), and `_pw_byname' in the user data base routines (*note Reading the User Database: Passwd Functions.). This convention is recommended, since it even further decreases the chance of inadvertent conflict among variable names. Note that this convention can be used equally well both for variable names and for private function names too. While I could have re-written all the library routines to use this convention, I did not do so, in order to show how my own `awk' programming style has evolved, and to provide some basis for this discussion. As a final note on variable naming, if a function makes global variables available for use by a main program, it is a good convention to start that variable's name with a capital letter. For example, `getopt''s `Opterr' and `Optind' variables (*note Processing Command Line Options: Getopt Function.). The leading capital letter indicates that it is global, while the fact that the variable name is not all capital letters indicates that the variable is not one of `awk''s built-in variables, like `FS'. It is also important that *all* variables in library functions that do not need to save state are in fact declared local. If this is not done, the variable could accidentally be used in the user's program, leading to bugs that are very difficult to track down. function lib_func(x, y, l1, l2) { ... USE VARIABLE some_var # some_var could be local ... # but is not by oversight } A different convention, common in the Tcl community, is to use a single associative array to hold the values needed by the library function(s), or "package." This significantly decreases the number of actual global names in use. For example, the functions described in *Note Reading the User Database: Passwd Functions., might have used `PW_data["inited"]', `PW_data["total"]', `PW_data["count"]' and `PW_data["awklib"]', instead of `_pw_inited', `_pw_awklib', `_pw_total', and `_pw_count'. The conventions presented in this section are exactly that, conventions. You are not required to write your programs this way, we merely recommend that you do so.  File: gawk.info, Node: Sample Programs, Next: Language History, Prev: Library Functions, Up: Top Practical `awk' Programs ************************ This chapter presents a potpourri of `awk' programs for your reading enjoyment. Many of these programs use the library functions presented in *Note A Library of `awk' Functions: Library Functions.. * Menu: * Clones:: Clones of common utilities. * Miscellaneous Programs:: Some interesting `awk' programs.  File: gawk.info, Node: Clones, Next: Miscellaneous Programs, Prev: Sample Programs, Up: Sample Programs Re-inventing Wheels for Fun and Profit ====================================== This section presents a number of POSIX utilities that are implemented in `awk'. Re-inventing these programs in `awk' is often enjoyable, since the algorithms can be very clearly expressed, and usually the code is very concise and simple. This is true because `awk' does so much for you. It should be noted that these programs are not necessarily intended to replace the installed versions on your system. Instead, their purpose is to illustrate `awk' language programming for "real world" tasks. The programs are presented in alphabetical order. * Menu: * Cut Program:: The `cut' utility. * Egrep Program:: The `egrep' utility. * Id Program:: The `id' utility. * Split Program:: The `split' utility. * Tee Program:: The `tee' utility. * Uniq Program:: The `uniq' utility. * Wc Program:: The `wc' utility.  File: gawk.info, Node: Cut Program, Next: Egrep Program, Prev: Clones, Up: Clones Cutting Out Fields and Columns ------------------------------ The `cut' utility selects, or "cuts," either characters or fields from its standard input and sends them to its standard output. `cut' can cut out either a list of characters, or a list of fields. By default, fields are separated by tabs, but you may supply a command line option to change the field "delimiter", i.e. the field separator character. `cut''s definition of fields is less general than `awk''s. A common use of `cut' might be to pull out just the login name of logged-on users from the output of `who'. For example, the following pipeline generates a sorted, unique list of the logged on users: who | cut -c1-8 | sort | uniq The options for `cut' are: `-c LIST' Use LIST as the list of characters to cut out. Items within the list may be separated by commas, and ranges of characters can be separated with dashes. The list `1-8,15,22-35' specifies characters one through eight, 15, and 22 through 35. `-f LIST' Use LIST as the list of fields to cut out. `-d DELIM' Use DELIM as the field separator character instead of the tab character. `-s' Suppress printing of lines that do not contain the field delimiter. The `awk' implementation of `cut' uses the `getopt' library function (*note Processing Command Line Options: Getopt Function.), and the `join' library function (*note Merging an Array Into a String: Join Function.). The program begins with a comment describing the options and a `usage' function which prints out a usage message and exits. `usage' is called if invalid arguments are supplied. # cut.awk --- implement cut in awk # Arnold Robbins, arnold@gnu.ai.mit.edu, Public Domain # May 1993 # Options: # -f list Cut fields # -d c Field delimiter character # -c list Cut characters # # -s Suppress lines without the delimiter character function usage( e1, e2) { e1 = "usage: cut [-f list] [-d c] [-s] [files...]" e2 = "usage: cut [-c list] [files...]" print e1 > "/dev/stderr" print e2 > "/dev/stderr" exit 1 } The variables `e1' and `e2' are used so that the function fits nicely on the screen. Next comes a `BEGIN' rule that parses the command line options. It sets `FS' to a single tab character, since that is `cut''s default field separator. The output field separator is also set to be the same as the input field separator. Then `getopt' is used to step through the command line options. One or the other of the variables `by_fields' or `by_chars' is set to true, to indicate that processing should be done by fields or by characters respectively. When cutting by characters, the output field separator is set to the null string. BEGIN \ { FS = "\t" # default OFS = FS while ((c = getopt(ARGC, ARGV, "sf:c:d:")) != -1) { if (c == "f") { by_fields = 1 fieldlist = Optarg } else if (c == "c") { by_chars = 1 fieldlist = Optarg OFS = "" } else if (c == "d") { if (length(Optarg) > 1) { printf("Using first character of %s" \ " for delimiter\n", Optarg) > "/dev/stderr" Optarg = substr(Optarg, 1, 1) } FS = Optarg OFS = FS if (FS == " ") # defeat awk semantics FS = "[ ]" } else if (c == "s") suppress++ else usage() } for (i = 1; i < Optind; i++) ARGV[i] = "" Special care is taken when the field delimiter is a space. Using `" "' (a single space) for the value of `FS' is incorrect--`awk' would separate fields with runs of spaces, tabs and/or newlines, and we want them to be separated with individual spaces. Also, note that after `getopt' is through, we have to clear out all the elements of `ARGV' from one to `Optind', so that `awk' will not try to process the command line options as file names. After dealing with the command line options, the program verifies that the options make sense. Only one or the other of `-c' and `-f' should be used, and both require a field list. Then either `set_fieldlist' or `set_charlist' is called to pull apart the list of fields or characters. if (by_fields && by_chars) usage() if (by_fields == 0 && by_chars == 0) by_fields = 1 # default if (fieldlist == "") { print "cut: needs list for -c or -f" > "/dev/stderr" exit 1 } if (by_fields) set_fieldlist() else set_charlist() } Here is `set_fieldlist'. It first splits the field list apart at the commas, into an array. Then, for each element of the array, it looks to see if it is actually a range, and if so splits it apart. The range is verified to make sure the first number is smaller than the second. Each number in the list is added to the `flist' array, which simply lists the fields that will be printed. Normal field splitting is used. The program lets `awk' handle the job of doing the field splitting. function set_fieldlist( n, m, i, j, k, f, g) { n = split(fieldlist, f, ",") j = 1 # index in flist for (i = 1; i <= n; i++) { if (index(f[i], "-") != 0) { # a range m = split(f[i], g, "-") if (m != 2 || g[1] >= g[2]) { printf("bad field list: %s\n", f[i]) > "/dev/stderr" exit 1 } for (k = g[1]; k <= g[2]; k++) flist[j++] = k } else flist[j++] = f[i] } nfields = j - 1 } The `set_charlist' function is more complicated than `set_fieldlist'. The idea here is to use `gawk''s `FIELDWIDTHS' variable (*note Reading Fixed-width Data: Constant Size.), which describes constant width input. When using a character list, that is exactly what we have. Setting up `FIELDWIDTHS' is more complicated than simply listing the fields that need to be printed. We have to keep track of the fields to be printed, and also the intervening characters that have to be skipped. For example, suppose you wanted characters one through eight, 15, and 22 through 35. You would use `-c 1-8,15,22-35'. The necessary value for `FIELDWIDTHS' would be `"8 6 1 6 14"'. This gives us five fields, and what should be printed are `$1', `$3', and `$5'. The intermediate fields are "filler," stuff in between the desired data. `flist' lists the fields to be printed, and `t' tracks the complete field list, including filler fields. function set_charlist( field, i, j, f, g, t, filler, last, len) { field = 1 # count total fields n = split(fieldlist, f, ",") j = 1 # index in flist for (i = 1; i <= n; i++) { if (index(f[i], "-") != 0) { # range m = split(f[i], g, "-") if (m != 2 || g[1] >= g[2]) { printf("bad character list: %s\n", f[i]) > "/dev/stderr" exit 1 } len = g[2] - g[1] + 1 if (g[1] > 1) # compute length of filler filler = g[1] - last - 1 else filler = 0 if (filler) t[field++] = filler t[field++] = len # length of field last = g[2] flist[j++] = field - 1 } else { if (f[i] > 1) filler = f[i] - last - 1 else filler = 0 if (filler) t[field++] = filler t[field++] = 1 last = f[i] flist[j++] = field - 1 } } FIELDWIDTHS = join(t, 1, field - 1) nfields = j - 1 } Here is the rule that actually processes the data. If the `-s' option was given, then `suppress' will be true. The first `if' statement makes sure that the input record does have the field separator. If `cut' is processing fields, `suppress' is true, and the field separator character is not in the record, then the record is skipped. If the record is valid, then at this point, `gawk' has split the data into fields, either using the character in `FS' or using fixed-length fields and `FIELDWIDTHS'. The loop goes through the list of fields that should be printed. If the corresponding field has data in it, it is printed. If the next field also has data, then the separator character is written out in between the fields. { if (by_fields && suppress && $0 !~ FS) next for (i = 1; i <= nfields; i++) { if ($flist[i] != "") { printf "%s", $flist[i] if (i < nfields && $flist[i+1] != "") printf "%s", OFS } } print "" } This version of `cut' relies on `gawk''s `FIELDWIDTHS' variable to do the character-based cutting. While it would be possible in other `awk' implementations to use `substr' (*note Built-in Functions for String Manipulation: String Functions.), it would also be extremely painful to do so. The `FIELDWIDTHS' variable supplies an elegant solution to the problem of picking the input line apart by characters.  File: gawk.info, Node: Egrep Program, Next: Id Program, Prev: Cut Program, Up: Clones Searching for Regular Expressions in Files ------------------------------------------ The `egrep' utility searches files for patterns. It uses regular expressions that are almost identical to those available in `awk' (*note Regular Expression Constants: Regexp Constants.). It is used this way: egrep [ OPTIONS ] 'PATTERN' FILES ... The PATTERN is a regexp. In typical usage, the regexp is quoted to prevent the shell from expanding any of the special characters as file name wildcards. Normally, `egrep' prints the lines that matched. If multiple file names are provided on the command line, each output line is preceded by the name of the file and a colon. The options are: `-c' Print out a count of the lines that matched the pattern, instead of the lines themselves. `-s' Be silent. No output is produced, and the exit value indicates whether or not the pattern was matched. `-v' Invert the sense of the test. `egrep' prints the lines that do *not* match the pattern, and exits successfully if the pattern was not matched. `-i' Ignore case distinctions in both the pattern and the input data. `-l' Only print the names of the files that matched, not the lines that matched. `-e PATTERN' Use PATTERN as the regexp to match. The purpose of the `-e' option is to allow patterns that start with a `-'. This version uses the `getopt' library function (*note Processing Command Line Options: Getopt Function.), and the file transition library program (*note Noting Data File Boundaries: Filetrans Function.). The program begins with a descriptive comment, and then a `BEGIN' rule that processes the command line arguments with `getopt'. The `-i' (ignore case) option is particularly easy with `gawk'; we just use the `IGNORECASE' built in variable (*note Built-in Variables::.). # egrep.awk --- simulate egrep in awk # Arnold Robbins, arnold@gnu.ai.mit.edu, Public Domain # May 1993 # Options: # -c count of lines # -s silent - use exit value # -v invert test, success if no match # -i ignore case # -l print filenames only # -e argument is pattern BEGIN { while ((c = getopt(ARGC, ARGV, "ce:svil")) != -1) { if (c == "c") count_only++ else if (c == "s") no_print++ else if (c == "v") invert++ else if (c == "i") IGNORECASE = 1 else if (c == "l") filenames_only++ else if (c == "e") pattern = Optarg else usage() } Next comes the code that handles the `egrep' specific behavior. If no pattern was supplied with `-e', the first non-option on the command line is used. The `awk' command line arguments up to `ARGV[Optind]' are cleared, so that `awk' won't try to process them as files. If no files were specified, the standard input is used, and if multiple files were specified, we make sure to note this so that the file names can precede the matched lines in the output. The last two lines are commented out, since they are not needed in `gawk'. They should be uncommented if you have to use another version of `awk'. if (pattern == "") pattern = ARGV[Optind++] for (i = 1; i < Optind; i++) ARGV[i] = "" if (Optind >= ARGC) { ARGV[1] = "-" ARGC = 2 } else if (ARGC - Optind > 1) do_filenames++ # if (IGNORECASE) # pattern = tolower(pattern) } The next set of lines should be uncommented if you are not using `gawk'. This rule translates all the characters in the input line into lower-case if the `-i' option was specified. The rule is commented out since it is not necessary with `gawk'. #{ # if (IGNORECASE) # $0 = tolower($0) #} The `beginfile' function is called by the rule in `ftrans.awk' when each new file is processed. In this case, it is very simple; all it does is initialize a variable `fcount' to zero. `fcount' tracks how many lines in the current file matched the pattern. function beginfile(junk) { fcount = 0 } The `endfile' function is called after each file has been processed. It is used only when the user wants a count of the number of lines that matched. `no_print' will be true only if the exit status is desired. `count_only' will be true if line counts are desired. `egrep' will therefore only print line counts if printing and counting are enabled. The output format must be adjusted depending upon the number of files to be processed. Finally, `fcount' is added to `total', so that we know how many lines altogether matched the pattern. function endfile(file) { if (! no_print && count_only) if (do_filenames) print file ":" fcount else print fcount total += fcount } This rule does most of the work of matching lines. The variable `matches' will be true if the line matched the pattern. If the user wants lines that did not match, the sense of the `matches' is inverted using the `!' operator. `fcount' is incremented with the value of `matches', which will be either one or zero, depending upon a successful or unsuccessful match. If the line did not match, the `next' statement just moves on to the next record. There are several optimizations for performance in the following few lines of code. If the user only wants exit status (`no_print' is true), and we don't have to count lines, then it is enough to know that one line in this file matched, and we can skip on to the next file with `nextfile'. Along similar lines, if we are only printing file names, and we don't need to count lines, we can print the file name, and then skip to the next file with `nextfile'. Finally, each line is printed, with a leading filename and colon if necessary. { matches = ($0 ~ pattern) if (invert) matches = ! matches fcount += matches # 1 or 0 if (! matches) next if (no_print && ! count_only) nextfile if (filenames_only && ! count_only) { print FILENAME nextfile } if (do_filenames && ! count_only) print FILENAME ":" $0 else if (! count_only) print } The `END' rule takes care of producing the correct exit status. If there were no matches, the exit status is one, otherwise it is zero. END \ { if (total == 0) exit 1 exit 0 } The `usage' function prints a usage message in case of invalid options and then exits. function usage( e) { e = "Usage: egrep [-csvil] [-e pat] [files ...]" print e > "/dev/stderr" exit 1 } The variable `e' is used so that the function fits nicely on the printed page. Just a note on programming style. You may have noticed that the `END' rule uses backslash continuation, with the open brace on a line by itself. This is so that it more closely resembles the way functions are written. Many of the examples use this style. You can decide for yourself if you like writing your `BEGIN' and `END' rules this way, or not.  File: gawk.info, Node: Id Program, Next: Split Program, Prev: Egrep Program, Up: Clones Printing Out User Information ----------------------------- The `id' utility lists a user's real and effective user-id numbers, real and effective group-id numbers, and the user's group set, if any. `id' will only print the effective user-id and group-id if they are different from the real ones. If possible, `id' will also supply the corresponding user and group names. The output might look like this: $ id -| uid=2076(arnold) gid=10(staff) groups=10(staff),4(tty) This information is exactly what is provided by `gawk''s `/dev/user' special file (*note Special File Names in `gawk': Special Files.). However, the `id' utility provides a more palatable output than just a string of numbers. Here is a simple version of `id' written in `awk'. It uses the user database library functions (*note Reading the User Database: Passwd Functions.), and the group database library functions (*note Reading the Group Database: Group Functions.). The program is fairly straightforward. All the work is done in the `BEGIN' rule. The user and group id numbers are obtained from `/dev/user'. If there is no support for `/dev/user', the program gives up. The code is repetitive. The entry in the user database for the real user-id number is split into parts at the `:'. The name is the first field. Similar code is used for the effective user-id number, and the group numbers. # id.awk --- implement id in awk # Arnold Robbins, arnold@gnu.ai.mit.edu, Public Domain # May 1993 # output is: # uid=12(foo) euid=34(bar) gid=3(baz) \ # egid=5(blat) groups=9(nine),2(two),1(one) BEGIN \ { if ((getline < "/dev/user") < 0) { err = "id: no /dev/user support - cannot run" print err > "/dev/stderr" exit 1 } close("/dev/user") uid = $1 euid = $2 gid = $3 egid = $4 printf("uid=%d", uid) pw = getpwuid(uid) if (pw != "") { split(pw, a, ":") printf("(%s)", a[1]) } if (euid != uid) { printf(" euid=%d", euid) pw = getpwuid(euid) if (pw != "") { split(pw, a, ":") printf("(%s)", a[1]) } } printf(" gid=%d", gid) pw = getgrgid(gid) if (pw != "") { split(pw, a, ":") printf("(%s)", a[1]) } if (egid != gid) { printf(" egid=%d", egid) pw = getgrgid(egid) if (pw != "") { split(pw, a, ":") printf("(%s)", a[1]) } } if (NF > 4) { printf(" groups="); for (i = 5; i <= NF; i++) { printf("%d", $i) pw = getgrgid($i) if (pw != "") { split(pw, a, ":") printf("(%s)", a[1]) } if (i < NF) printf(",") } } print "" }  File: gawk.info, Node: Split Program, Next: Tee Program, Prev: Id Program, Up: Clones Splitting a Large File Into Pieces ---------------------------------- The `split' program splits large text files into smaller pieces. By default, the output files are named `xaa', `xab', and so on. Each file has 1000 lines in it, with the likely exception of the last file. To change the number of lines in each file, you supply a number on the command line preceded with a minus, e.g., `-500' for files with 500 lines in them instead of 1000. To change the name of the output files to something like `myfileaa', `myfileab', and so on, you supply an additional argument that specifies the filename. Here is a version of `split' in `awk'. It uses the `ord' and `chr' functions presented in *Note Translating Between Characters and Numbers: Ordinal Functions.. The program first sets its defaults, and then tests to make sure there are not too many arguments. It then looks at each argument in turn. The first argument could be a minus followed by a number. If it is, this happens to look like a negative number, so it is made positive, and that is the count of lines. The data file name is skipped over, and the final argument is used as the prefix for the output file names. # split.awk --- do split in awk # Arnold Robbins, arnold@gnu.ai.mit.edu, Public Domain # May 1993 # usage: split [-num] [file] [outname] BEGIN { outfile = "x" # default count = 1000 if (ARGC > 4) usage() i = 1 if (ARGV[i] ~ /^-[0-9]+$/) { count = -ARGV[i] ARGV[i] = "" i++ } # test argv in case reading from stdin instead of file if (i in ARGV) i++ # skip data file name if (i in ARGV) { outfile = ARGV[i] ARGV[i] = "" } s1 = s2 = "a" out = (outfile s1 s2) } The next rule does most of the work. `tcount' (temporary count) tracks how many lines have been printed to the output file so far. If it is greater than `count', it is time to close the current file and start a new one. `s1' and `s2' track the current suffixes for the file name. If they are both `z', the file is just too big. Otherwise, `s1' moves to the next letter in the alphabet and `s2' starts over again at `a'. { if (++tcount > count) { close(out) if (s2 == "z") { if (s1 == "z") { printf("split: %s is too large to split\n", \ FILENAME) > "/dev/stderr" exit 1 } s1 = chr(ord(s1) + 1) s2 = "a" } else s2 = chr(ord(s2) + 1) out = (outfile s1 s2) tcount = 1 } print > out } The `usage' function simply prints an error message and exits. function usage( e) { e = "usage: split [-num] [file] [outname]" print e > "/dev/stderr" exit 1 } The variable `e' is used so that the function fits nicely on the screen. This program is a bit sloppy; it relies on `awk' to close the last file for it automatically, instead of doing it in an `END' rule.  File: gawk.info, Node: Tee Program, Next: Uniq Program, Prev: Split Program, Up: Clones Duplicating Output Into Multiple Files -------------------------------------- The `tee' program is known as a "pipe fitting." `tee' copies its standard input to its standard output, and also duplicates it to the files named on the command line. Its usage is: tee [-a] file ... The `-a' option tells `tee' to append to the named files, instead of truncating them and starting over. The `BEGIN' rule first makes a copy of all the command line arguments, into an array named `copy'. `ARGV[0]' is not copied, since it is not needed. `tee' cannot use `ARGV' directly, since `awk' will attempt to process each file named in `ARGV' as input data. If the first argument is `-a', then the flag variable `append' is set to true, and both `ARGV[1]' and `copy[1]' are deleted. If `ARGC' is less than two, then no file names were supplied, and `tee' prints a usage message and exits. Finally, `awk' is forced to read the standard input by setting `ARGV[1]' to `"-"', and `ARGC' to two. # tee.awk --- tee in awk # Arnold Robbins, arnold@gnu.ai.mit.edu, Public Domain # May 1993 # Revised December 1995 BEGIN \ { for (i = 1; i < ARGC; i++) copy[i] = ARGV[i] if (ARGV[1] == "-a") { append = 1 delete ARGV[1] delete copy[1] ARGC-- } if (ARGC < 2) { print "usage: tee [-a] file ..." > "/dev/stderr" exit 1 } ARGV[1] = "-" ARGC = 2 } The single rule does all the work. Since there is no pattern, it is executed for each line of input. The body of the rule simply prints the line into each file on the command line, and then to the standard output. { # moving the if outside the loop makes it run faster if (append) for (i in copy) print >> copy[i] else for (i in copy) print > copy[i] print } It would have been possible to code the loop this way: for (i in copy) if (append) print >> copy[i] else print > copy[i] This is more concise, but it is also less efficient. The `if' is tested for each record and for each output file. By duplicating the loop body, the `if' is only tested once for each input record. If there are N input records and M input files, the first method only executes N `if' statements, while the second would execute N`*'M `if' statements. Finally, the `END' rule cleans up, by closing all the output files. END \ { for (i in copy) close(copy[i]) }  File: gawk.info, Node: Uniq Program, Next: Wc Program, Prev: Tee Program, Up: Clones Printing Non-duplicated Lines of Text ------------------------------------- The `uniq' utility reads sorted lines of data on its standard input, and (by default) removes duplicate lines. In other words, only unique lines are printed, hence the name. `uniq' has a number of options. The usage is: uniq [-udc [-N]] [+N] [ INPUT FILE [ OUTPUT FILE ]] The option meanings are: `-d' Only print repeated lines. `-u' Only print non-repeated lines. `-c' Count lines. This option overrides `-d' and `-u'. Both repeated and non-repeated lines are counted. `-N' Skip N fields before comparing lines. The definition of fields is similar to `awk''s default: non-whitespace characters separated by runs of spaces and/or tabs. `+N' Skip N characters before comparing lines. Any fields specified with `-N' are skipped first. `INPUT FILE' Data is read from the input file named on the command line, instead of from the standard input. `OUTPUT FILE' The generated output is sent to the named output file, instead of to the standard output. Normally `uniq' behaves as if both the `-d' and `-u' options had been provided. Here is an `awk' implementation of `uniq'. It uses the `getopt' library function (*note Processing Command Line Options: Getopt Function.), and the `join' library function (*note Merging an Array Into a String: Join Function.). The program begins with a `usage' function and then a brief outline of the options and their meanings in a comment. The `BEGIN' rule deals with the command line arguments and options. It uses a trick to get `getopt' to handle options of the form `-25', treating such an option as the option letter `2' with an argument of `5'. If indeed two or more digits were supplied (`Optarg' looks like a number), `Optarg' is concatenated with the option digit, and then result is added to zero to make it into a number. If there is only one digit in the option, then `Optarg' is not needed, and `Optind' must be decremented so that `getopt' will process it next time. This code is admittedly a bit tricky. If no options were supplied, then the default is taken, to print both repeated and non-repeated lines. The output file, if provided, is assigned to `outputfile'. Earlier, `outputfile' was initialized to the standard output, `/dev/stdout'. # uniq.awk --- do uniq in awk # Arnold Robbins, arnold@gnu.ai.mit.edu, Public Domain # May 1993 function usage( e) { e = "Usage: uniq [-udc [-n]] [+n] [ in [ out ]]" print e > "/dev/stderr" exit 1 } # -c count lines. overrides -d and -u # -d only repeated lines # -u only non-repeated lines # -n skip n fields # +n skip n characters, skip fields first BEGIN \ { count = 1 outputfile = "/dev/stdout" opts = "udc0:1:2:3:4:5:6:7:8:9:" while ((c = getopt(ARGC, ARGV, opts)) != -1) { if (c == "u") non_repeated_only++ else if (c == "d") repeated_only++ else if (c == "c") do_count++ else if (index("0123456789", c) != 0) { # getopt requires args to options # this messes us up for things like -5 if (Optarg ~ /^[0-9]+$/) fcount = (c Optarg) + 0 else { fcount = c + 0 Optind-- } } else usage() } if (ARGV[Optind] ~ /^\+[0-9]+$/) { charcount = substr(ARGV[Optind], 2) + 0 Optind++ } for (i = 1; i < Optind; i++) ARGV[i] = "" if (repeated_only == 0 && non_repeated_only == 0) repeated_only = non_repeated_only = 1 if (ARGC - Optind == 2) { outputfile = ARGV[ARGC - 1] ARGV[ARGC - 1] = "" } } The following function, `are_equal', compares the current line, `$0', to the previous line, `last'. It handles skipping fields and characters. If no field count and no character count were specified, `are_equal' simply returns one or zero depending upon the result of a simple string comparison of `last' and `$0'. Otherwise, things get more complicated. If fields have to be skipped, each line is broken into an array using `split' (*note Built-in Functions for String Manipulation: String Functions.), and then the desired fields are joined back into a line using `join'. The joined lines are stored in `clast' and `cline'. If no fields are skipped, `clast' and `cline' are set to `last' and `$0' respectively. Finally, if characters are skipped, `substr' is used to strip off the leading `charcount' characters in `clast' and `cline'. The two strings are then compared, and `are_equal' returns the result. function are_equal( n, m, clast, cline, alast, aline) { if (fcount == 0 && charcount == 0) return (last == $0) if (fcount > 0) { n = split(last, alast) m = split($0, aline) clast = join(alast, fcount+1, n) cline = join(aline, fcount+1, m) } else { clast = last cline = $0 } if (charcount) { clast = substr(clast, charcount + 1) cline = substr(cline, charcount + 1) } return (clast == cline) } The following two rules are the body of the program. The first one is executed only for the very first line of data. It sets `last' equal to `$0', so that subsequent lines of text have something to be compared to. The second rule does the work. The variable `equal' will be one or zero depending upon the results of `are_equal''s comparison. If `uniq' is counting repeated lines, then the `count' variable is incremented if the lines are equal. Otherwise the line is printed and `count' is reset, since the two lines are not equal. If `uniq' is not counting, `count' is incremented if the lines are equal. Otherwise, if `uniq' is counting repeated lines, and more than one line has been seen, or if `uniq' is counting non-repeated lines, and only one line has been seen, then the line is printed, and `count' is reset. Finally, similar logic is used in the `END' rule to print the final line of input data. NR == 1 { last = $0 next } { equal = are_equal() if (do_count) { # overrides -d and -u if (equal) count++ else { printf("%4d %s\n", count, last) > outputfile last = $0 count = 1 # reset } next } if (equal) count++ else { if ((repeated_only && count > 1) || (non_repeated_only && count == 1)) print last > outputfile last = $0 count = 1 } } END { if (do_count) printf("%4d %s\n", count, last) > outputfile else if ((repeated_only && count > 1) || (non_repeated_only && count == 1)) print last > outputfile }