DataIO.Doc How to use the DataIO hooks. New features at doc' end. OVERVIEW This document describes how to access numerical data from MathVISION. Numerical data can be encapsulated in an IFF MTRX type file. By keeping data in an IFF file it becomes easy to use it between programs. Currently only Seven Seas Software is using this format. We will help any developers write readers and writers for this format. There is nothing proprietary about this format. We will allow you to redistribute the program that translates from ASCII files to MTRX files. The MTRX file solves several data file handling problems. The first is saving an arbitrary data structure. Data structures in MTRX files are very much like structures in ` C ' and can contain arbitrary arrangements of structures and arrays. Arrays not only contain sequential values, but may contain real indexes that allows the user to interpolate between points. Charts, for instance contain scales that represent distance and location. MTRX files allow scales to be attached to arrays. MTRX files implement matrixes as arrays of arrays. The basic sequence to convert a file from ASCII to MTRX and use it with the MathVISION system is as follows: 1. Using the program MakeMTRX, create an IFF MTRX file from your text data file. 2. Using the DataIO hook, load the MTRX file into memory. 3. Write an appropriate formula in MathVISION to access the data in memory. There is an intermediate format, a type of IFF file called "MTRX", which we have defined to store matrix data. The program MakeMTRX reads your source file, and creates a MTRX file containing the same data, but in the MTRX structured form. The DataIO hook reads the MTRX file, and stores the data in memory, so that it is quickly accessible to MathVISION. In MathVISION, you must write a formula using one or more of the four functions `get()', `put()', `gets()', or `puts()' to access the data. Here is a diagram of the data flow. +--------+ +------+ ________ get | Source | MakeMTRX | IFF | DataIO hook / \ put |==========| | Text |===========>| MTRX |=============>| Memory |======>|| Screen || +--------+ +------+ \________/ |==========| ======================================================================== USING MakeMTRX ======================================================================== MakeMTRX is located in the Support directory of your MathVISION disk. It is meant to be used from the CLI (Command Line Interface). It expects a command line in the following format. MathVISION:Support/MakeMTRX Template Source Destination where - Template is the name of file which describes the Source file - Source is the name of text file which contains the data to be read - Destination is the name of a file to be created to hold the IFF MTRX data. TEMPLATES A Template is a file which describes the source file. There are a number of sample templates in the MathVISION:DataTemplates drawer. The basic function of a template is to describe the structure of the data. For example, here is a structure definition for a 3 by 5 double-precision array. For more examples, examine the templates in the DataTemplates drawer. BEGINSTRUCTURE DEFARRAY 3 0.0 1.0 ! three elements, scaled 0.0 to 1.0 DEFARRAY 5 0.0 1.0 ! five elements, scaled 0.0 to 1.0 DEFDATATYPE IEEE8 ! store double precision IEEE numbers ENDARRAY ENDARRAY ENDSTRUCTURE This is a nested definition. You may arbitrarily create arrays of arrays, or arrays of structures, or structures of structures, whatever it takes to define your data. Arrays have elements which are all the same type, and they have scaling, to allow more meaningful access using non integer indices. Structures have fields which may be different types, and are always accessed with integer indices. You will probably need to edit the template and/or your data file so they match. First, find the template which matches your data best, and modify it as needed to describe your data. Hopefully you will not have to edit your data file. Frequently the only change you will need to make is to modify the number of data points read. The reader skips over all non numeric characters that it can not read and does a pretty good job at parsing numbers out of character data. The commands which are allowed in the template for the data reader MakeMTRX are as follows: ---------------------------------------------- DEFINE variablename INTEGER | REAL ! define a new variable of given type ASSIGN variablename value ! give variable a value DISPLAY variablename ! print variable, good for debugging READ variablename [ variablename] ! read variable value from source file BEGINSTRUCTURE ! Surround the definition of whole structure ENDSTRUCTURE DEFARRAY elements low_limit high_limit ! define an array define element type here ENDARRAY ! arrays have same type elements DEFSTRUCT fields ! define a struct (fields of different types) ! define field one ! define field two ! define field n ENDSTRUCT DEFDATATYPE FFP | IEEE8 | LONG | WORD | BYTE ! define data in structure ! FFP is a 4 byte real number, using the Motorola Fast Floating Point ! encoding, and corresponds with the .ffp version of MathVISION ! IEEE8 is an 8 byte real number, double-precision IEEE, the native ! type of the .ieee version of MathVISION ! LONG is a 4 byte integer ! WORD is a 2 byte integer ! BYTE is a 1 byte integer RECURSEREAD ! fill the structure from the source file ! How should you separate the data values? MakeMTRX will skip all ! spaces, tabs, and end-of-lines, and will also skip one comma. ! This gives you considerable flexibility. LOOK at files in the MathVISION:DataTemplates drawer with an editor. ======================================================================== USING DataIO HOOK ======================================================================== The DataIO hook is in the Storage drawer of the hooks drawer. When started, it creates a small window on the MathVISION edit screen. You have four options. Load IFF MTRX - load an IFF MTRX file which was created with MakeMTRX. Save IFF MTRX - save a file in IFF MTRX format, in case you put new values in it. Save Text - dump the values to a text file, and create another file with ".hdr" appended which describes the format of the text file. Show Structure - Display the format of the currently loaded data, including the sizes of the structures and arrays, and the scaling applied to arrays. This information can be copied into XMin, XMax, and etc. for scaled access. There are five functions defined for use in your formulas. There are two for access using integer indices (get & put), two for access using real coordinates (gets & puts), and finally an indication of errors encountered when performing the getting and putting (geterr). get( indices ) put( value, indices ) There may be any number of indices, depending on the complexity of your structure. All indices are zero-based. For a one-dimensional array, there will be one index. For a one-dimensional array of structures, there will be two indices, one for the element of the array, and a second for the field of the structure. gets( indices ) puts( value, indices ) These functions work the same as get and put, except the indices which correspond to arrays will be scaled using the array limits. The read in this hook does not interpolate. It recovers the closest point to the index. It would be possible to build considerably more elegant readers. geterr() If an error occured in any of the above routines, a zero is returned, and geterr() may be used to diagnose the problem. The values returned are: 0 - everything OK! 1 - index out of range, either less than 0 or greater than the number of fields/elements. 2 - unknown structure type (internal problem) 3 - not enough indices (structure more complex than you thought!) 4 - conversion overflow - a number could not be converted to the desired type. Either use the IEEE version of MathVISION, or make sure the MTRX file uses a datatype which is big enough. 5 - bad command (internal problem) 6 - too many indices (structure not as complex as you thought! Ilbm2Mtrx: This program is at MathVISION:support/Ilbm2mtrx This converts an Ilbm file to a MTRX file. Simply type: >Ilbm2mtrx Hisogram document: The MakeMTRX program has provisions for reading scatter plot data, especially when particular points occur many times. For example, suppose we have a 4 by 4 grid, and the following data set, where each pair of numbers means that there is one occurence of the data at that point: 1 2 1 2 1 2 2 2 2 2 2 3 2 3 3 1 3 1 4 2 4 2 4 2 4 2 In the template, We first define a 4x4 grid, scaled from 1.0 to 4.0 BeginStructure DefArray 4 1.0 4.0 DefArray 4 1.0 4.0 DefDataType BYTE ! Byte has -128..127, WORD has -32768..32767 EndArray EndArray EndStructure To read this, we put the command 'HistoRead' in the file, followed by a number of indicies. The number of indicies determines how many indicies there are (related to the dimensionality of the MTRX), and the actual numbers given for the indicies determine in what order they are used. In this case, we would use: HistoRead 0 1 because there are two indicies to read, and they occur in x,y order. (to read it in Y,X order, use 'HistoRead 1 0'). Now, suppose the data has been condensed, so that each location in the data is given ONCE, but it contains the number of 'hits' at that location: X Y Count 1 2 3 2 2 2 2 3 2 3 1 2 4 2 4 The command to read this is much the same as before, only we use 'HistoCountRead.' Since there are still only two indicies, only two are specified. HistoCountRead 0 1