\documentstyle[11pt]{article} \title{M2P, A Modula-2 Preprocessor} \author{Dennis Brueni} \date{30 July 1991} \topmargin 0in \oddsidemargin 0in \evensidemargin 0in \textheight 9in \textwidth 6.5in \begin{document} \maketitle %==================================================================== \section{Introduction} This archive contains a preprocessor written in Modula-2 and intended for Modula-2 programs\footnote{Actually, M2P actually will work with about any text file}. The purpose is to provide pre-compile time control of source code in a portable manner, that is to allow the same source to be used for several target machines, operating systems, and/or compilers. It is also very good for controlling the inclusion or exclusion of such things as debugging code or nonessential features. It will strip all comments from a file, except those containing directives, which are usually marked with a '\$'. eg.\ (*\$S+*) would not be stripped. Comments containing directives must be at the first level, otherwise they will also be stripped from the file. Nested comments are handled properly. M2P will also strip excess whitespace from a file, only leaving that which is needed by the compiler to parse the input correctly. M2P tries not to interfere with the line spacing of the file, so that error messages flagged by the compiler on the preprocessed file correspond correctly to their location in the original file. However, since comments and whitespace are stripped, the horizontal position may be wrong for those compilers that do indicate the exact line position. All preprocessor commands are prefixed by '@'. This is chosen to aviod conflict with any Oberon or Modula-2 lexicals \footnote{The new Modula-2 standard declares '@' as the 'required dereferencing operator', which causes a conflict for those people choosing to use '@' rather than '\^\,'.} (unlike the C-Preprocessor where '\#' may be confused with not equal). Preprocessor commands may be given in free format (unlike most C-preprocessors where the '\#' must be in the first column.) %==================================================================== \section{Preprocessor Commands} The following statements are recognized by the preprocessor. \subsection{@DEFINE {\em identifier}} This defines an identifier with the value TRUE. An identifier is associated with the boolean value FALSE if it is not @DEFINE'd. For example: \begin{quote}\begin{verbatim} @DEFINE AMIGA @DEFINE THE @DEFINE BEST @DEFINE COMPUTER \end{verbatim}\end{quote} \subsection{@UNDEF {\em identifier} } This cancels the most recent definition of the given identifier. Normally, this associates FALSE with the identifier, unless the identifier had been multiply defined earlier. \begin{quote}\begin{verbatim} @UNDEF MSDOS @UNDEF MAC \end{verbatim}\end{quote} \subsection{@INCLUDE 'filename' } The named file is preprocessed and fed into the output file. Caveat Emptor: This takes care not to ruin line spacing by filtering the linefeeds and such out of the include file, so the resulting preprocessed file may contain some very long lines. If your compiler cannot handle such long lines, then this may create a problem. It is suggested that INCLUDE files only contain preprocessor directives (MACROS and DEFINES) to aviod this problem. \begin{quote}\begin{verbatim} @INCLUDE "MACROS" @INCLUDE "CopyRight" \end{verbatim}\end{quote} The environment variable 'M2PInclude' may be used to set the search path for INCLUDE files. Paths may also be specified from the command line by using the INCLUDE switch. \begin{quote}\begin{verbatim} > setenv M2PInclude "'DH0:M2/Macros/','DH0:M2/OldMacros/'" > M2P INCLUDE "DH0:M2/NewMacros/" Args.mpp \end{verbatim}\end{quote} M2P searchs include paths in the following order: \begin{enumerate} \item Path(s) specified on the command line \item The current directory \item Path(s) specified in the environment varaiable M2PInclude \end{enumerate} Hence, in the above example, {\tt DH0:M2/NewMacros/} will be searched first, then the current directory, then {\tt DH0:M2/Macros/}, and finally {\tt DH0:M2/OldMacros/}. \subsection{@MACRO {\em identifier} [ '(' \{ {\em arglist} \} ')' ] {\em definition} @ENDM} A macro definition. There may be an arbitrary number of arguments separated by commas, which are used as local macros within the macro definition. The opening (left) parenthesis for the arguments must immediately follow {\em identifier}, otherwise it will be treated as part of the macro. \begin{quote}\begin{verbatim} @MACRO VectorSum(Array1,Array2,LO,HI,DestArray,i) @i:=@LO; WHILE @i <=@HI DO @DestArray[@i]:=@Array1[@i]+@Array2[@i]; INC(@i); END; @ENDM \end{verbatim}\end{quote} Macro definitions may be nested arbitrary levels. Realize that nested macro definitions will be evaluated lazily, and are not going to be treated as locals unless you explicitely place an @UNDEF at the end of the outer macro to undefine the local bindings. I caution against the use of nested macro definitions unless you are {\em very} sure about what you are doing. \begin{quote}\begin{verbatim} @MACRO Ugly @MACRO LocalToUgly a surfboard @ENDM @MACRO Sentence well known that life is@ENDM @MACRO Global It @ENDM @Global is @Sentence sure @Global is @Sentence @LocalToUgly, and that to @LocalToUgly @Global @Sentence @Global. @UNDEF LocalToUgly @UNDEF Sentence @ENDM \end{verbatim}\end{quote} \begin{description} \item[Question:] What would {\tt @Ugly} evalute to? \item[Answer:] It is well known that life is sure It is well known that life is a surfboard, and that to a surfboard It is well known that life is It. \end{description} The symbol table of M2P is a hash table which uses a linked lists for each bucket. These linked lists behave somewhat like stacks, so when a symbol is redefined, the most recent binding is the one always used. When that binding is removed, the previous binding will appear on the top of stack and be used from that point on. This allows one to define local macros or flags within macros and then @UNDEF them at then end of the macro. However, whenever a flag is redefined, an error message is written to standard output. Finally, it is not illegal to define a macro recursively, but it will take an infinite amount of time and space to expand. \subsection{@ {\em identifier} [ '(' \{ {\em arglist} \} ')' ] } A macro instance. The arguments are bound as local macros and inserted into the previous macro definition which is expanded and placed in the output file. The number of arguments used must be less than or equal to the number of arguments in the corresponding macro definition. An error will be flagged if more arguments are used. Also, if the argument used in the macro requires expanding, but was not specified, an error will be flagged. Arguments are evaluated lazily and not treated any differently than any other macro. {\em Be careful with this! } If you have a global defined having the same name as one of the arguments, and do not specify that argument, the global will be used in its place. For example: \begin{quote}\begin{verbatim} @MACRO c GOTCHA @ENDM @MACRO a(b,c) (@b,@c) @ENDM @c ==> GOTCHA @a(one,two) ==> (one,two) @a(one) ==> (one,GOTCHA) \end{verbatim}\end{quote} \subsection{@LINE and @SPACE} When a macro is recorded, line feeds and carriage returns are stripped from the macro, so that when it is later inserted in a file, it will not change the line spacing of the file, (ie. it is inserted entirely on one line). However, sometimes this may not be desirable because linefeeds in the file may have meaning and cannot be stipped (eg. assembly language source). To solve this shortcoming, @LINE can be used to insert a line feed into the output file. and @SPACE can be used to insert a single space Of course, the use of @LINE will destroy the line numbers of the output file as it corresponds to the input file. For example: \begin{quote}\begin{verbatim} @MACRO Length(Loop,Areg,Dreg,Scratch) @IF NOT @Scratch THEN @MACRO Scratch D0 @ENDM @END @SPACE move.w @Dreg,@Scratch @LINE @Loop tst.w (@Areg)+ @LINE @SPACE dbeq @Scratch,@Loop @LINE @SPACE sub.w @Scratch,@Dreg @LINE @ENDM \end{verbatim}\end{quote} An instance of this macro will always produce four lines of code, throwing the line count off by four. \subsection{@@} A way to get an @ sign into your output file. \subsection{@(, @), and @,} Built-in macros for '(', ')', and comma, respectively. These above macros are meant to be used only in weird situations using macro arguments. Normally it is unnecessary to use them to get ',' '(' or ')'. The wierd situations pop up when you want to place a comma or an unbalanced number of parenthesis in a macro argument. For example: \begin{quote}\begin{verbatim} @MACRO identity(a) @a @ENDM } @identity (argument) ==> argument @identity ((((5)))) ==> (((5))) @identity ((5@,6@,7)) ==> (5,6,7) @identity (@)@,@() ==> ),( \end{verbatim}\end{quote} \subsection{@IF {\em expression} @THEN {\em statements} \{ @ELSIF {\em expression} @THEN {\em statements} \} [ @ELSE {\em statements} ] @END} The IF-THEN-ELSIF-ELSE-END clause. The form of this statment is identical to the Modula IF-THEN-ELSE-ELSIF-END statements, with the exception that the keywords are preceded by a @. The expression is a boolean expression made up of boolean flags (identifiers defined with @DEFINE or @MACRO), AND(\&), OR(|) and, NOT(\~\,). Either the MODULA or OBERON syntaxes are accepted for AND, OR, and NOT. Precedence is given by first parenthesis, then NOT, then AND, then OR, as usual. The @ before THEN is optional as it is not absolutely required for parsing the statement. \begin{quote}\begin{verbatim} @IF AMIGA @THEN "The Best" @ELSIF MSDOS @THEN What can I say @ELSIF MAC @THEN pity, a real pity. @END @IF Debug1 THEN @IF Debug2 THEN @IF Debug3 THEN This is really fun!!! @ELSE @IF Debug6 THEN @IF Debug7 THEN @IF Debug8 THEN @IF Debug9 THEN @IF Debug10 THEN We could do this all day!!! @END @END @ELSE Couldn't we? @END @END @END @END @END @ELSIF Debug2 | Debug3 | Debug4&Debug5 | Debug6 THEN Whew, there's no end to the possibilities @ELSIF Debug7&~Debug8&~Debug9 THEN When will this nut stop? @ELSE Here, I suppose. @END \end{verbatim}\end{quote} \subsection{@STRIP and @NOSTRIP} The {\tt @STRIP} and {\tt @NOSTRIP} directives are used to control whether M2P should strip comments or not, respectively. Comments not containing directives within {\tt @MACRO} definition statements are always stripped. %==================================================================== \section{Using M2P} If M2P is invoked with no arguments it will give a usage message in the form of a regular expression giving the format of the command line parameters. M2P will accept as many as 128 parameters. \begin{verbatim} > M2P T1.mpp(A B C) T2.dpp T3.lpp(DEBUG) T4.app TO test5.a68 \end{verbatim} This will first invoke M2P on T1.mpp. The output file will be automatically translated to T1.mod, and the symbols A, B, and C will be defined as true. The symbol table will be cleared, and T2.dpp will be processed and output will go to T2.def. T3.lpp will be processed with DEBUG defined and the output will go to T3.lsp. Finally, T4.app will be processed and the output will specifically be sent to test5.a68. Without the TO clause, the output would have went to T4.asm. If an input file is given as -- then the standard input will be used. Likewise if an output file is given as --, the output will go to the standard output and extraneous messages will be suppressed. This makes it possible to use M2P with redirection and concurrent pipes. The following rules are used for automatic translation of filename extensions: \begin{center} \begin{tabular}{|l|l|l|} \hline \hline {\em Input extension} & {\em Output extension} & {\em Example File Type}\\ \hline .mpp & .mod & Modula-2 \\ .dpp & .def & " \\ .cpp & .c & C \\ .hpp & .h & " \\ .app & .asm & Assembly \\ .lpp & .lsp & Lisp \\ .tpp & .txt & text \\ .ppp & .pas & pascal \\ .opp & .mod & Oberon \\ .fpp & .for & Fortran \\ .spp & .scm & Scheme \\ \hline \hline \end{tabular} \end{center} Although I included the .cpp to .c and .hpp to .h translations for C source code, M2P will not always work properly with C source code since certain legal syntaxes in C look like comments in Modula-2. It is recommended to turn off comment stripping (using {\tt @NOSTRIP}) if using M2P on C source code. M2P reads the M2PFlags environment variable and places all flags seen there on the symbol table defined as true. ENV: must be assigned to prevent a requester from popping up asking to (on the Amiga) "Please Insert Disk ENV:" See your 1.3 enhancer manual for more information. If ENV: is assigned, but M2PFlags is not defined, it is ignored. The following options are also accepted by M2P as command line arguments. \begin{description} \item{INCLUDE {\em path}} Adds {\em path} to M2P's searchpath for INCLUDE files. \item{STRIP} Instructs M2P to strip comments. (default) \item{NOSTRIP} Instructs M2P not to strip comments. \end{description} M2P can also be ran from the Amiga Workbench. It will open its own window and assign the standard input and output channels to that window. %==================================================================== \section{Memory Requirements} M2P requires about 45K statically at runtime. However, as computations progress, and large macros are defined, memory is dynamically consumed in 4K chunks. Typically, for a large file M2P may need around 80K total. The M2P executable requires less than 30K. %==================================================================== \section{Error Messages} \begin{itemize} \item Strings may not be broken across line boundaries. A LF,CR,FF, or VT control charactor was found inside a string. This may also happen if there is loose apostrophe inside a comment containing directives. \item Comment not closed with *) End of file was reached before M2P could locate the closing *) of a comment. \item Token is too long, limit is 255 charactors You have a string or an identifier which is ridiculously long. \item Filename (string) expected following @INCLUDE Recall the syntax: @INCLUDE "filename" \item Unrecognized Preprocessor statement Something was found following an @ which did not form a valid preprocessor statement. \item Identifier must be defined as a macro An attempt was made to use @ as a macro, but is not a macro. \item WARNING: Symbol has already been defined An attempt has been made to bind a symbol to a macro definition, or @DEFINE it, but that symbol was previously defined. The new definition will be stacked on top of the old definition. \item WARNING: Symbol has not been defined An attempt was made to use @ as a macro, but is not defined, possibly an argument which was not supplied to a macro. \item Identifier expected following @UNDEF \item Identifier expected following @DEFINE \item Identifier expected following @MACRO See Section 2 (above) \item Unbalanced parenthesis in expression If unbalanced parenthesis are desired, use @) or @( to designate the unbalanced member. \item Illegal factor in expression The expression parser expected to see an ID, NOT, or \~\,. \item Expected @ELSE or @END \item Expected @END \item Expected THEN \item Expected END after IF-THEN See Section 2 (above), under IF-STATEMENT \item Expected , or ) \item Expected Identifier for macro argument name See Section 2 (above) for macro instances. \item Too many arguments in macro You may not supply more arguments to a macro than those given in the definition, but you may supply less. \item Argument to macro not terminated EOF was encountered before finding , or ) in a macro instance. \item Macro definition not terminated with @ENDM End Of File was encountered in a macro definition. \item Internal Failure An internal sanity check failed. Please try to replicate and then report any such problems. \end{itemize} %==================================================================== \section{Amiga CLI Return Codes} \begin{description} \item[5: IOError,] A required file could not be opened. \item[10: NoMemory,] Could not allocate storage for a dynamic list node. \item[15: Errors,] Preprocessing is complete, but there were errors in the file. \item[20: BadParams,] Unrecognizable command line parameters \end{description} %==================================================================== \section{Debugging Features} M2P also supports four hidden command line switches, TOKENS, TRACERDP, SYMTAB, and MACLIST. TOKENS will report every token recogized by the lexical analyzer after it is recoginized. TRACERDP will trace calls in the recursive descent parser. SYMTAB will report all accesses to the symbol table. MACLIST will report uses of the MacLists module functions. They must be given as the first arguments and may be given in any order. \begin{quote}\begin{verbatim} > M2P TRACERDP TOKENS - TO - \end{verbatim}\end{quote} This will read from the standard input, and write to the standard output, and report token information and recursive calls. {\em Note:} for performance, these features are optional at compile time. The compiled version included does not support these options. To include any one if these features in a recompilation, simply include the flag DEBUG with the M2FLAGS in the makefile. %==================================================================== \section{To Do} The support for M2Amiga and BenchMark is not really there with the exception of getting a few compiler directives right. Important things like interfacing to their CommandLine, debugging, I/O, and RunTime libraries still needs to be resolved. I don't have either of these compilers, so really couldn not do much. %==================================================================== \section{Implementation Notes} The M2Sprint version is much better than the TDI version. Due to the way TDI's Runtime library is set up, some features had to be left out. To be specific, the TDI version does not trap CtrlC's, or work from the Workbench. Also, it is larger and slower than the M2Sprint version. %==================================================================== \section{Porting} I have tried to avoid as much OS and compiler dependent stuff as possible, but some things were hard to justify losing. This describes the major areas that may need looking at in order to port M2P. \begin{itemize} \item The Args module needs to be able to get at the command line arguments and parse them. It uses special parsing methods to separate (, --, and )'s from other arguments. This way you can avoid hitting the space bar a few times. If your compiler provides the C-like argv and argc variables, then you could use the following code to feed the arguments to Args. \begin{quote}\begin{verbatim} FOR i:=0 TO .argc-1 DO AddArgs(.argv[i]^); END; \end{verbatim}\end{quote} See the internal documentation in Args for more details. \item IO houses the file handling stuff. There are three areas of implementation dependent code. \begin{itemize} \item The type definition of 'RealFileType'. This is at the top of the Implementaton module. \item After all the interface procedures, there is an implementation dependent section of code that houses all the dirty stuff, such as: ReadInfo, Flush, OSOpen, OSCreate, OSAppend, OSClose, and OSGoodFile. \item The standard I/O channels are picked up in the inialization code. \end{itemize} I plan to substitute the standard Modula-2 I/O library for this library in a future release. \item The Env module uses the Amiga specific ENV: handler. Please insert your own local method for getenv and setenv. \item You will have do some simple bootstrapping to move this code to a totally new platform. This is because all the .mod files here are .mpp files and have to be ran through M2P before compilation. To bootstrap M2P, use the .mod files in the 'mods' subdirectory. This are the results straight from M2P with the flags DEBUG and AMIGA. Make the changes needed to get M2P running on your target system, and be sure to write them down. When you are done, incorporate those changes into the original .mpp files and remake the system. You will probably have to define some unique flag for your compiler and system. Define that flag in the M2FLAGS variable in the Makefile. \item Most of the .mpp files @INCLUDE the file "MACROS" This file, as the name implies contains some macros. All of these macros involve the inclusion/exclusion of compiler directives since this varies from implementation to implementation. Edit the macros as needed to add the proper directives for your system. If in doubt, no harm is done if no directives are included, just some runtime performance loss. Some of the code in "MACROS" also serves to configure M2P dynamically to the system. \end{itemize} %==================================================================== \section{References and Thanks} Much of the ideas behind M2P was inspired from the C-Preprocesser and also the Modula-2 preprocessor supplied with the YAML distribution (by Olsen \& Associates). %==================================================================== \section{Disclaimer} This program is placed in the public domain, and may be used or distributed as you like. If you make useful changes or bug fixes to M2P, I would like to know about them so they may be incorporated into future versions. M2P is provided on an as-is basis; I assume no responsibility for GURUS, system crashes, bedwettings, or failed marriages due to the use of M2P. \section{WhoAmI} \begin{tabbing} Dennis Brueni\\ 900 Glade Road, \#9 \\ Blacksburg, VA 24060 \\ \\ EM: brueni@csgrad.cs.vt.edu\\ \end{tabbing} If you by any chance feel guilty about getting something for nothing, then send a picture of Abe Lincoln to me at the above address, and we'll both feel better. $\smile$ \end{document}