Special Shareware version of "An Electronic Holmes Companion" (This file copyright 1987-1993 by Robert J. Stek. All rights reserved) The following text is taken from the commercial version of "An Elec- tronic Holmes Companion" available from PsyLogic Systems, Box 315, Tolland, CT 06084 for $59.95 plus $3.00 shipping and handling. The complete version has all four novels and fifty-six short stories (over 3.7 megabytes of text), including the stories still protected by copyright (this Shareware version has only stories in the public domain). In addition, the complete version has many public domain, shareware, and licensed programs which can be used to analyze the text in various ways (these are described later in this file). A 40 page printed manual and over 200 pages of documentation on disk are provided as well as a listing of the 250+ world-wide Holmesian scion societies, a basic reading list of 100 Holmesian references, the Con- stitution and Buy-Laws of the Baker Street Irregulars, information on Sher- lockian journals and vendors, a buyer's guide for commercial text indexing and retrieval software, and an invitation to dial up Sherlocktron, a remote bulletin-board system devoted to the Sherlockian world. Introduction Sherlock Holmes "Elementary, my dear Watson!" "The game is afoot!" "Who do you think I am - Sherlock Holmes?" How many millions of people the world over have used those phrases! How many millions instantly recognize the combination of pipe, magnifying glass, deerstalker cap, and Inverness cape? How many more people know the address 221B Baker Street than the address 10 Downing Street? Few fic- tional characters have made a greater impression on the world's collective psyche than Sherlock Holmes. According to one source, the only book which has more copies in circulation than the Holmes stories is the Bible. (By the way, true Sherlockians insist that Holmes was not fictional; Sir Arthur Conan Doyle was merely John Watson's literary agent!) As I write these words in December of 1987, the centenary celebration of the introduction of the world's first consulting detective is coming to a close. In the 1887 Beeton's Christmas Annual, the novel A Study in Scar- let by a little known physician-author was first published. Almost in- stantly, late Victorian England had a hero whose career was to extend for several decades in terms of published adventures, and who now is entering his second century of general popularity. Even when his creator tried to kill him off in The Final Problem, he refused to die. Doyle was forced to resurrect Holmes several years later as the public demanded to read more of the adventures of this amazing detective and his friend and chronicler, John H. Watson, M.D. As you will see in the Appendix "A Basic Holmesian Library," much has been written about Sherlock Holmes over the years. And more will be writ- ten in years to come. There is an undying fascination with the Holmes character and with Doyle himself. Not only the printed word, but movies, television shows, the theater, cartoons, audio cassettes, radio plays, even computer games - almost all types of media have been used to portray the adventures of Holmes and Watson. But until now, the Canon (as Sherlock- ians are wont to call the collected works, a.k.a the Sacred Writings) has not appeared in computer readable form. Why would anyone want to read the adventures of Sherlock Holmes on their computer, when a handsome leather-bound volume seems so much more ap- propriate? I am can only quote Sherlock himself: "Data! Data! I must have data!" While it may be nice for collectors of Sherlockiana to have yet an- other edition, I personally wish to use my computer to add to the constant- ly increasing accumulation of knowledge about the Master and his literary agent. Much gold has been mined already; check your library for back is- sues of The Baker Street Journal, Baker Street Miscellania, and The Sher- lock Holmes Journal (only the better libraries carry them!). For years Holmes fans have poured over the stories, trying to explain in- consistencies, put the stories in chronological order, and out-deduce Holmes himself. For example, while Holmes' birthday is known, the year of his birth is never explicitly mentioned. By using information contained in the Canon regarding his years at university (Cambridge, Oxford, or both?), his early adventures, casual remarks made to Watson, etc., it is possible to make some educated guesses. (Those of you wondering why anyone would want to do this have already missed the point and should donate your copy of the Companion to a true Sherlockian.) By having the Canon in computer readable text, it is now possible to use the power of the computer to analyze the stories for more "data" on which to base further hypotheses - some serious and many tongue-in-cheek (a member of my local scion society once devoted 18 pages to determining whether or not Holmes played tennis). With the software programs provided in the Companion it is now possible to instantly find any word or phrase in the Canon. Tools are provided to create a printed concordance to the stories and to compare some characteristics of Watson's (Doyle's?) writing style to writers of the same and later eras. As is the case of tools in general, their use is limited only by the imagination of the craftsman using them. Computers and Text Analysis Although computers are usually associated with science and engineering and with business data processing, even humanists and literary types have been making use of computers for the last two decades. Literary analysis involves the comprehension of the intellectual, thematic and structural continuities which make a work an artistic whole. The complete text must be available to the student of literature, but the unassisted human mind cannot completely comprehend a long and involved text. So the scholar is forced to scan the text in a laborious page-by-page manner - a slow, dull process. This process has been augmented with the development of concordance-generating or full-text retrieval systems on computers. A concordance has been described as a general purpose working tool for the study of literature. It takes a text, isolates the words, sorts them in alphabetical order or in order of frequency, and lists each word in some context - generally the rest of the line in which the word appeared. A Key Word In Context (KWIC) concordance displays all words centrally with a context before and after each word. A Key Word Out of Context (KWOC) con- cordance displays its keywords at the left or right margin with the context either following or preceding the keyword. Concordance programs have grad- ually become more sophisticated, and many permit selective concording (word-inclusion and word-exclusion lists), automatic sorting of words of different spellings under a "root" form, variable length contexts, and col- location searching (words within a certain distance of other words). Concordances can reveal an author's vocabulary and use of idioms. An author may be biased towards some words and phrases (one of the least "singular" words used in the Canon is the word "singular"). Omissions of words in common use can also be revealing. An author may give an idiosyn- cratic meaning to a word or phrase, or may even coin new words. Any in- sight into the author's word usage increases our understanding of what he is saying. Concordances are also valuable in stylistic analysis. A text may be concorded in a phonetic form with markers for the metrical stress of each word. In this way the distinctive rhythms of a poet may be revealed. By sorting the words alphabetically from right to left, a reverse concordance will order all rhyming words together. Words may be marked with syntactic identifiers which after concording can reveal characteristic phrase and sentence structures. Even "pictorial" concordances can be constructed by proponents of structural stylistics. That is, graphs can be constructed based on the number of times a word or phrase or concept appears in a given context. Literary structures such as "five-act", "four-part", "linear", "circular", "inverted", "rising and falling", or "suspended" styles may be found. With the advent of the personal computer, concording programs which had resided only on mainframe computers migrated downward. As evidenced by the programs included in the Companion, tools which were affordable only by large universities are now readily available at minimal costs. While the tools provided here are rather basic, and some are limited as to the size of the text with which they can work, they nevertheless provide a glimpse into the world of computer assisted literary analysis. For further reading consult: Abercrombie, John. Computer Programs for Literary Analysis. Philadel- phia: Univ. of Pennsylvania Press, 1984. Aitken, A.J., R.W. Bailey and N. Hamilton-Smith. The Computer and Literary Studies. Edinburgh: Edinburgh Univ. Press, 1973. Burke, E. ed. Computers in Humanistic Research. Englewood Cliffs, N.J.: Prentice-Hall, 1967. Day, A. Colin. Text Processing. Cambridge: Cambridge Univ. Press, 1984. Hockey, Susan. A Guide to Computer Applications in the Humanities. Balti- more: Johns Hopkins Univ. Press, 1980. Howard-Hill, T.H. Literary Concordances: A Guide to the Preparation of Manual and Computer Concordances. Oxford: Pergamon Press, 1979. Kernighan, B.W. and P.J. Plaugher. Software Tools in Pascal. Reading, Mass.: Addison-Wesley, 1981. Mitchell, J.L., ed. Computers in the Humanities. Edinburgh: Edinburgh Univ. Press, 1974. Oakman, Robert. Computer Methods for Literary Research. Columbia, S.C.: Univ. of South Carolina Press, 1980. Patton, P.C. and R.A. Holoien. Computing in the Humanities. Lexington, Mass.: Lexington, 1981. Wisbey, R.A., ed. The Computer in Literary and Linguistic Research. Cam- bridge: Cambridge Univ. Press, 1971. Holmes Meets the Micro: A SCANdal in Technologia In the fall of 1986 I recognized that the centenary year of Sherlock Holmes was about to begin. Although I had been a Sherlockian for a great number of years, I had been content to read and re-read the stories and to attend local scion society meetings for the fellowship and good cheer. I decided that I wished to undertake a project which would in some way mark the centenary and would also personalize my own involvement with the Hol- mesian world. Besides being an avid Sherlockian, I am also a computer hobbyist. Though I have worked with computers in my professional life for a number of years, I was also one of those tinkerers who assembled a computer from a kit back in 1977. It seemed natural to somehow link the Sherlockian world with the computer age. At first I thought I would write an article showing how Sherlock has shown up in the computing world in a variety of ways. For example, there are two books about programming in BASIC and Pascal which are presented as the logically-minded Holmes giving lessons to Dr. Watson. A number of "adventure" games involving Holmes and Watson also exist. And several "high-tech" companies advertise their products using images of Holmes and Watson. I had heard that a Sherlockian from Toronto had spent several years typing the stories into a computer for the purpose of publishing a con- cordance to the Canon. I was also aware of a new generation of optical scanners which could "read" typeset material and turn it into computer readable format. One "angle" for the story suggested itself: a comparison of the time involved in entering the stories manually versus having them scanned. Then it occurred to me that having the text available on disk could also be an end in itself. The Problems of Scanning Text Many people express amazement at how many things computers can't do. Unless you have a general understanding of the limitations of computers, you may blithely assume (as we are encouraged to do by television and film writers) that computers can do just about anything. HAL, the talking com- puter of the film 2001: A Space Odyssey, was on screen back in the late sixties. In Being There, it was taken for granted that we had the technol- ogy to construct a word processor which understood spoken language. What could be so difficult about having a computer read? Didn't banks have those peculiar looking characters on the bottom of checks for their com- puters to read? Optical character reader (OCR) technology has been available for a number of years. Banks used those peculiar characters because it has been extremely difficult to get a computer to understand ordinary printed or written communication. Proportional spacing of typeset material printed in a variety of font styles and sizes create a real headache for the relative- ly inexpensive (under $3000) scanners available today. These scanners are adequate for graphic images and can recognize a limited number of mono- spaced typewriter and laser printer fonts, but are totally useless for mag- azine articles and books. Only two companies manufacture equipment which can scan and recognize almost any type style or font, and this equipment costs between $20,000 and $40,000. [Author's note: in just the past 2 years, much has changed; now these capabilities can be had for $2000- $3000!) High Tech to the Rescue - the Palantir CDP I contacted the manufacturer of one of the most sophisticated text scanners available, the Palantir Compound Document Processor, and proposed my project to the president of the corporation. He agreed that it while it would pose little in the way of a challenge to their equipment, it nevertheless would make for some very interesting publicity for his firm. Scanning insurance documents or court transcripts doesn't make for exciting reading. But having Sherlock Holmes enter the computer age via the Palan- tir CDP had a certain air about it and might make for some good publicity. (By the way, the Sunday New York Times of November 15, 1987, ran an article about the CDP and the scanning of the Canon.) During the months of May and June, 1987, I spent four days at the Bos- ton sales office of the Palantir Corporation where I learned how to use the CDP and "fed" it a copy of The Complete Sherlock Holmes. What had taken a fellow Sherlockian in Toronto over three years to accomplish (on a part time basis - he estimated about three months of full time work) the CDP was able to do in a little over three days. Admittedly the scanning was not perfect. The accuracy rate ranged from about 92% to about 97%. While that sounds high, it means that 5 characters out of every 100 were wrong - or about 150,000 out of the 3,000,000 characters in the Canon. Luckily most of the errors could be dealt with automatically by a good spelling checker or word processor. For example, the CDP often had trouble distinguishing a "b" from an "h". A simple command to find and replace all occurrences of "tbe" with "the" cured these problems. However, each occurrence of "he" and "be" needed to be proof-read manually. The Proof is in the Proofing Proof-reading some of your favorite literature sounds like it could be fun. Unfortunately the human mind is very forgiving and easily overlooks typographical errors when concentrating on the call of the hunt and the thrill of the chase. One must turn to such techniques as reading backwards in order to catch the CDP's oversights. I might still be proof-reading even now if it weren't for the help offered by several fellow Sherlockians who learned of the project. While there are thousands of Sherlockians all over the world who belong to the loose network of the Baker Street Ir- regulars and scion societies, not that many have computers or were aware of the project. As I attended some meetings and workshops over the summer, I did meet several who were very interested in the project, including one fellow who had himself started typing the Canon into his computer manually. With the help of these Sherlockians (credited in the Acknowledgements) this electronic version of the Holmes stories now exists. The fellowship that exists among Sherlockians is astounding. Yet upon reflection it ap- pears that all the right ingredients exist in the scion societies: Meet- ings are just two or three times a year. Membership dues are nominal if they exist at all. People who attend do so for the sheer pleasure of being in the company of others who are as enraptured with Holmes as they are. Little of world import or serious consequence is discussed. Serious and not-so-serious research, analyses, and rumors are presented. Pre-dinner toasts are made liberally - to Holmes, Watson, Mrs. Hudson, Mycroft, Queen Victoria, Irene Adler, the "literary agent" and even to Moriarty him- self. (See the Constitution and Buy-Laws of the Baker Street Irregulars in the Appendix.) Where else could one hope to meet people interested in so quixotic a project? If the above description appeals to you, by all means seek out a scion society. If one doesn't exist, start one. Most are somewhat anarchistic in character. John Bennett Shaw, the Sherlockian's Sherlockian, suggests that a scion begin with three members: two can then kick out the third if they so wish. Pick out an appropriate name and start meeting. What could be simpler? The Appendix lists the known, active societies and the name of the last known contact person. Access to Tools Some Holmesians will purchase this volume for the sheer sake of having a computer readable version of the Canon to add to their collection. But I suspect that those who take the time to master at least some of the pro- grams described in this section will gain an expanded perspective of Holmes and Doyle not available to those who do not learn to use these tools. As in other areas of life, some of these tools are simple to use while others will require a good deal of effort. Some are for the technically minded, and some will be used by everyone. Most will be used seriously, but at least one is silly (and sneaky!). The intent of all these tools is to al- low you to manipulate the text in some way, analyze it and make new dis- coveries regarding these wonderful stories and their creator. The easiest to use and handiest tool is LIST. LIST allows you to read a text file, page forward and backward through the text, and perform simple searches of the text being LISTed. It is also handy for extracting por- tions of the Canon into a separate file for other uses (e.g., creating a collection of quotations, segregating quotations by character, etc.). ANYWORD allows you to index and find any word in any story instantly. If you can't remember what the dog did in the night (in SILVER BLAZE), you can search for the word "dog" within 50 characters of the word "night" and see the section of text in question. This is probably one of the biggest bargains in the Companion. It is a commercial program which retails at $49.95 itself. It does have some limitations. It can't index text files containing more than one megabyte (about one-third of the Canon), the in- dexing process is slow (8 hours to index one megabyte), and the index file takes as much space as the text file itself. Within its limitations, how- ever, it will probably be one of the most useful programs provided. The manual references "A Scandal in Bohemia" which is pre-indexed for you. A Movie Database is also referenced which is available from the author at low cost. MTAS is a wonderful introduction to literary and linguistic research. It includes a BROWSE program similar to LIST but with much more powerful search capabilities. MTAS also produces alphabetical word frequency lists and can even graph word distribution and word density throughout a text. Its Glossary of Terms should be consulted to give you access to the spe- cialized vocabulary used in literary analysis. Turbo Pascal source code is provided. KWIC and SRCHUTIL were written by a professor of classics as tools ap- propriate for use in his discipline. SRCHUTIL shares some characteristics with LIST and BROWSE but also includes editing and sorting capabilities as well. KWIC is a traditional key-word-in-context concordance generator - a manual version of AnyWord; its output can go to your printer or to a disk file. Turbo Pascal source code is available from the author. PC-READ grades the "readability" of text. It answers the question, "How difficult is this document to read?" It can give a running analysis of a story, showing how the reading grade level changes from section to section, giving interest and variety to the text. You can also "mark" specific sections of text within a file and compute its readability. Turbo Pascal source code is available from the author. Software Tools were derived from the book Software Tools in Pascal by Kernighan and Plauger. They are single-purpose general tools which can be combined with each other and with the facilities of MS-DOS to perform many useful functions, e.g., another way to create a KWIC concordance, perform word frequency counts, etc. These tools are from several sources, but the book should be purchased for complete documentation. SORTF (by the author of LIST) is included in this section since it too is a "software tool," though it is a more sophisticated implementation of the program presented in Kernighan and Plaugher. It should be used instead of the SORT program provided with MS-DOS. FLEXTEX is also a "software tool" type of program, though it is a bit more like a Swiss Army knife. It is a powerful "find and replace" utility, which can be used to convert text file formats, remove characters or strings, put text in upper case, and add double spacing, for example. It can also be used as an indexing program. When used with other tools such as SORTF, it can produce word counts, co-occurrence patterns, concordances, and indices. It is a powerful program which requires a thorough under- standing of its many functions in order to get its full benefits. Source code is provided, but you will need a macro assembler to compile it. LETRFREQ is a program (source code included) which implements the ideas presented in the supplied BYTE reprint, "The Literary Detective" (how appropriate!). It produces tables showing the frequency of occurrence of single-, double-, and triple-letter combinations in a text. This has been shown to be a useful tool in assessing authorship of text. Finally, we will have one more clue as to whether Doyle wrote the stories, or whether he was just Watson's literary agent (as most Sherlockians suspect). Computer Programs for Literary Analysis by John Abercrombie (U. of Pennsylvania Press, Philadelphia, 1984) is one of the pioneering texts in its field. It is a tutorial both for text analysis and for writing effec- tive programs needed in the humanities. The book is necessary for any practical use of these programs. However most all the functions performed by these Turbo Pascal programs (source included) are duplicated by the pre- viously introduced tools, and they are included for the programmer who likes to "roll his own" and for those who are just curious. Finally, DECEIVE is a program which Moriarty himself would be glad to call his own. Imagine yourself at work, taking just a two-minute break (well deserved, of course) and reading a Holmes story at your computer. Echoing down the hallway are the familiar footsteps of your boss who does not understand your infatuation with the Canon. You deftly touch two keys and the Holmes story is instantly replaced by a screen of numbers from your spreadsheet or from your database manager. As soon as he leaves, touch two more keys and you are back in late Victorian England, and the game is again afoot. This program can save your job! File Format and File Preparation File Format There were a number of technical considerations in making available the Canon in computer readable form. The first had to do with translating from the printed page to disk, and the second had to do with distributing the text in a format most useful to people with differing preferences in several areas. A casual glance at any printed version of the Canon will reveal some structural considerations: how many words on a page, how deeply to indent paragraphs, how to indicate long quotations from letters or newspapers, how to present graphical images (i.e., dancing men, maps, fragments of hand- writing), for example. And thought was given to the fact that most people would probably want to refer to a printed version of the Canon even as they used the electronic version. So, the decision was made to use The Complete Sherlock Holmes (Doubleday) since it is very common, still in print, and relatively inexpensive (about $20). This edition was scanned by the Palan- tir Compound Document Processor described earlier. However, the two volume The Annotated Sherlock Holmes (Clarkson N. Potter, Inc.) is widely regarded by Sherlockians because of the voluminous annotations by William S. Baring- Gould. There are minor differences in the actual text (and further dif- ferences between the British and American versions, between the original manuscripts and the Strand publications, etc.) which are of interest to obsessive-compulsive Sherlockians and literary types (not necessarily mutually exclusive categories). Some time in the future I will probably make available an update which details these differences (but only if one of the obsessive-compulsive literary types clues me in). [NOTE: The following two paragraphs apply only to the commercial version. For this Shareware version, page numbers were removed completely, and carriage-return/line-feed were added after each line.] Luckily, some things are possible with an electronic version which are impossible with a printed edition. Therefore, embedded in the text on disk are page numbers referencing both the Doubleday and the Annotated editions. Page numbers from the Doubleday edition are enclosed in double square brackets (e.g., [[137]] ) while references to the Annotated edition are en- closed in double curly braces (e.g., {{I-137}} means Vol I, p 137). (When a page ended on a hyphenated word, the hyphen was removed and the page num- ber follows the full word.) It should be noted that the order of the stories on disk follows from the Doubleday edition; the Annotated edition presents the stories in chronological order as determined by Baring-Gould. One or both of these sets of page numbers may be optionally removed by the reader with the supplied programs REM_COMP and REM_ANNO. For sheer reading pleasure, one need not remove either set. However, if you own one of the printed editions, you may find it convenient to have the page references included so that you may refer to the printed pages as needed. You may also elect to remove both sets of numbers so that the text analysis pro- grams do not count these numbers as words or include them in an analysis. The choice is yours. In order to accomodate both sets of page numbers, a particular format was chosen for the stories on disk. This stream-oriented format places a carriage- return line-feed combination only at the end of paragraphs, and not at the end of each line. Few word processors accept this type of format (XyWrite and Mi- crosoft Word do, for example), so another utility program is provided (CONVCRLF) which MUST be used before the stories can be used with the analysis programs or read into your own word processing pro- gram. After running the stories through CONVCRLF, the text will be "plain vanilla" ASCII text which can be read by almost all word processors. Note, too, that this approach, as well as the 80 column limitation of most com- puter screens, means that line 5 on a certain page in the printed edition will almost assuredly NOT correspond to line 5 when viewing that same page on your computer. Other conventions were adopted in putting the text on disk. These in- clude avoiding the use of tabs. When a paragraph is indented, the tab character is NOT used; rather, five spaces are used for indenting a new paragraph. When a letter or newpaper article is quoted within a paragraph, it is separated from the main paragraph with a blank line (actually a carriage-return line-feed) and its first line is indented three spaces with following lines returning to the original left margin. The quotation is followed by a blank line and the original paragraph continues from the left margin. If the quotation ended the paragraph, the next paragraph begins with a five space indentation. Hyphenated words which are usually hyphenated are hyphenated in the electronic version. However, if a word was hyphenated at the end of a line in the printed edition, it was NOT hyphenated for this edition (unless it happened to be a usually hyphenated word as well). Pauses, unfinished statements, or side remarks which are shown in the printed text as long dashes are represented in the electronic edition as double dashes with a space in front and one behind if appropriate. The British pound sign (signifying money) was replaced by the capital letter 'L' preceding the numeric amount. Any graphical representations (i.e., dancing men) were omitted and several blank lines (carriage-return line-feeds) note this on the screen. British spellings were left as such, and accented foreign characters in words were treated as the nearest English ASCII equivalents. Italics are not shown. Chapter numbers and chapter headings (if present) follow the Doubleday edition in capitaliza- tion and centering. Undoubtedly there are discrepancies remaining between the Doubleday text which was used for scanning and this electronic edition. Double quotes (") may appear in the place of single quotes (') and vice-versa. Numerous spelling errors doubtlessly remain. Perhaps words have been left out, added, or strangely mutated. If you are one of the obsessive- compulsive types mentioned previously, feel free to rant and rave and send me the locations of such errors so that I may make corrections. You will gain a sense of deep satisfaction knowing that people purchasing the Com- panion after you will receive a better version and that your neurosis does indeed have some redeeming social value. (Seriously, such corrections would be appreciated.) SPECIAL OFFER!! Since you are enough of a Sherlock Holmes lover to have ordered this Shareware version, I would like to make you an offer that I hope you can't refuse. The regular price of An Electronic Holmes Companion is $59.95 plus $3 shipping. It includes 15 5.25" disks (or 7 3.5" disks) in a plastic disk holder, and a 40 page professionally printed 8 1/2 x 11 manual. I do pay royalties to Dame Jean Conan Doyle as well as to several of the authors of the licensed programs. I will offer it to you for $39.95 postpaid if you mention this notice. Such a deal! How can you refuse? P.S. If you have a friend who only has a Mac, I can offer a Mac ver- sion, but for the regular $59.95 plus $3.00 shipping. Write for details.