------------------------------------------ WORDFREQ: Frequency of Words in a Document © By Martin Kurtz ------------------------------------------ WORDFREQ analyzes a textfile and reports on the number of times specific words are used in a document. Such a system is best used in the analysis of writing style. All of us tend to fasten on pet words and overuse them. With WORDFREQ you can get a screen report of the number of times each word appears. Every unique word is mentioned in the report, whether it is used one or more. To help you try this method, I have put Abraham Lincoln's Gettysburg Address within the WORDFREQ document. Start the program. At the prompt to name the textfile to be analyzed, type: j:wordfreq/gettys A line of asterisks appears. This is the number of lines in the file. The asterisks turn color to indicate the line-by line analysis has begun. Next comes a "quiet" screen whose duration is set by the complexity and length of the file being checked. Then begins a check to determine if any word or words occurs 99 times, then works its way down to one occurrence. The report is given on a series of screens. Pressing any key continues the report to the next screen. Lincoln's brief masterpiece contains 266 words. Of these, nearly 100 are used only once. The most frequently used word is "that". Perhaps Mr. Lincoln could have improved his speech had he replaced or eliminated some of those occurrences. HOW TO ANALYZE THE RESULTS Unique words within a document are generally not a problem. You can all but ignore them. Similarly, the most often used words are usually those neutral words from which one wouldn't wish to escape -- the, that, on, to, a, and, it. In my own text analysis, I look for fairly unusual words I have used four or five times. In exposing this article to WORDFREQ, I discovered I had used the word "program" 17 times. That was too many, so I reworked these words to take out and replace them in many places. As things stand now, the word has been used seven times. I have let stand another word form I feel I have used too much -- "analysis," "analyze," and "analyzed." I left in these to illustrate my frailty and to let you find them and consider how I might have reduced these repetitions. The point is, I wrote this article without any clue I was overusing the word. WORDFREQ revealed my lapse. WHEN IT DOESN'T WORK No matter how much memory your Amiga has, WORDFREQ will not work on large text files. As rule of thumb, I suggest you restrict file size to about 15,000 bytes -- perhaps 200 lines of 76-character type. The program measures the file you load. If it is possibly too long to handle, it notes that fact and gives you the options or continuing or exiting. Even this measurement cannot always be correct, for memory is used based on a number of strings that are generated but never shown on screen. In short, you must approach things gingerly, starting with smaller files, then working upward to determine how large a file your system and its memory can accommodate. Sooner or later, you will run out of memory. The system is not to be faulted. It does as well as it can within the sizes of files you load for analysis. INTERNAL WORKS Most of the program's analysis is performed invisibly. In essence, the program loads the textfile one line at a time. It renders that line's text as upper case, then saves only spaces and letters from A to Z. While it eliminates punctuation marks, numerals and extra spaces, it does strive to preserve hyphenated constructions, as in "memory-intensive" and "five-day". It also attempts to reassemble words hyphenated from one line to the next, though it eliminates the hyphen and may in some negligible cases render a hyphenated construction as an unbroken word. As each line is prepared for analysis, it is assigned to one of 26 strings, for the letters of the alphabet. If the string does not contain a newly encountered word, it is attached to the end, along with a number to indicate its occurrences, as in: 001 AARDVARK The next time the same word is found, the entry becomes: 002 AARDVARK ... and so on. While up to 999 occurrences of a word can theoretically be recorded, the program only checks from 99 to 1 occurrences of each word. It would be quite rare for a document to contain more than 20 repeats of the same word. The program uses memory according to the number of new words found. Thus, a very large file with few unique words might be handled correctly, while a shorter file with many unique words might run out of heap memory. This is why the program cannot exactly anticipate size of files it can handle. Files which are PowerPacked (this one is) must be decrunched before they can be worked. As example of how to do this, open a Shell or CLI and therein type: j:util/decrunch j:wordfreq/wordfreq.doc ram:wordfreq.doc 2 Then run WORDFREQ and type: ram:wordfreq.doc ... as the file to be called. END OF TEXT