xData v1.1 xDataPrefs v1.1a (C) Martin W.Scott, 1993. (NB: Requires OS 2.0+) xData is a program that allows other programs to read xpk-compressed files as if they were normal files. It is similar to PP by Michael Berg, which provided a similar facility for PowerPacked files. However, PP caused problems on my system (it crashed), and anyway, PowerPacker has been superseded by the more powerful and extensible xpk standard. Unfortunately, it takes a long time for data-handling programs to catch-up and recognise the xpk standard, not to mention those programs that can't read any compressed files. The xpk system is distributed with a handler, xfh, which allows a pseudo-partition to be set up to handle xpk'd files in a completely transparent manner. Whilst this method is always going to be more system-friendly than the patch method employed by xData and PP, it isn't as user-friendly -- all your packed files have to be kept in the one place. I didn't want to install the xfh handler, so wrote xData. And remember that the xpk system handles the decompression of PowerPacked files too, giving increased flexibility to xData. I have not included the xpk libraries. If you haven't got them and like what you've read so far, you should obtain the complete xpk distribution (by ftp or fish disks). How does it work ---------------- xData patches a few dos routines to allow named programs to read xpk'd files. These named programs are called clients. This is different to PP, which affected ALL programs, and caused many incompatability problems (on my system at any rate), which is why I decided on the client model. Clients are added using the xDataPrefs program; when a client opens a xpk'd file, the file is decompressed to the t: directory and the client reads the decompressed file. When the client closes the file, the temporary file is deleted (but it's not quite that simple -- see technical details below). If a client examines the length of a compressed file, the xData patches actually tell it the uncompressed length (in case the client allocates memory for the whole file to load into). This client arrangement might seem restrictive, but it guarantees (though I don't :) system compatability. Most users only compress certain files and use certain programs to read them, so these are the only programs that need to use patched routines. It allows you to use your favourite data-handling programs rather than xpk-ified versions, e.g. you can use ViewTex where before you had to use ShowIFF or PPMore (for PowerPacked files). Starting xData -------------- Best done at startup. Can be run from Shell or Workbench, and auto-detaches if run from a shell. You can put it in WBStartup, user-startup or startup-sequence. Since it's sort-of a preferences handler, I put it after the IPrefs command in the startup-sequence. The only restriction is that it must be run after ENV: has been set-up. xData is a commodity, so it may be deactivated and reactivated with the Commodities Exchange. If the patches cannot be removed (because a client has a compressed file open), the user is notified. xData may be terminated from the Exchange too (or by re-running it). NB: I've noticed one or two problems disabling xData while clients are manipulating files, because they open/examine them repeatedly, and turning the patches off in the middle of all this can confuse them. xData checks that all its files are closed, but can't anticipate these sort of problems. However, there is no problem in disabling/enabling xData while clients are not busy opening files. Editing Preferences ------------------- Preferences are edited using the xDataPrefs program, which is based closely on standard Workbench 2.1 preference editors. All the usual menu items are there, and the gadgets work as you'd expect. Only the CLI interface is lacking. From the CLI, xDataPrefs takes either no arguments (in which case the edit window is opened) or one argument, a preferences file to use. On starting xDataPrefs, you will see two lists: the left-hand one is the list of clients xData currently recognises, and the other is the list of processes running on your system. If you wish to add as a client a program that is already running, look for it in the process list, select it and click on the Export gadget. It will then be transfered to the client list. The process list does not update regularly -- you must click on the Refresh gadget to update the list. Other new clients are added with the Add gadget (surprise); pressing this results in 'New client' being added to the list. Press right-amiga-x to clear the string gadget and enter the new client's name (see below for details on names to choose). You would use this method if the client you wanted wasn't running. A sample client list is DPaint ViewTex More MuchMore Display CygnusEd egrep etc. Case is not significant. The way names are given to processes on the Amiga is non-trivial. There are three general classes: 1. Workbench Processes These are the simplest, and the process name is the name of the icon used to start it. e.g. double-clicking on the DPaint icon creates a process called 'DPaint'. 2. Shell Processes Process names are generally non-descriptive, e.g. Shell Process, but the actual command a shell process is running is stored elsewhere, and xData looks there. The names stored there vary depending on how the command was invoked, and what Shell you're using, but all you need to worry about is the command name, i.e. the file part, not the path part. e.g. from the shell, the process started by the command 'run work:gfx/dpaint' will match the client entry 'dpaint', even if the full path and command is stored in the process. 3. Custom Processes These are processes created by programs which are not of the first 2 types. For example, I would type 'ced' to start CygnusEd from a Shell, it actually creates a process called 'CygnusEd', so to enable the editor to load xpk'd or PowerPacked files, a client called 'CygnusEd' must be added. So, the names to use as clients are just the names of the programs themselves, with no paths. Using the process list, you can usually find the correct process name to add to the client list. Problem Examples ---------------- A couple of examples I have come across, that may also be instructive in other situations: PROGRAM: ADPro PROBLEM: Doesn't recognise xpk'd files when 'ADPro' is a client. SOLUTION: ADPro uses sub-processes to operate on files, including loading and saving them. The name of the sub-process used to load a file depends on the file's type. i.e. when loading an IFF file, a process with name 'IFF' is created. Therefore, adding a client called 'IFF' will allow you to load xpk'd IFF files. There is no need to make 'ADPro' a client, just the relevant loaders. PROGRAM: DirWork PROBLEM: Seems to determine what xpk'd files are, but can't load them. SOLUTION: DirWork program spawns separate processes to view text and pictures, so as well as adding the DirWork command to your client list (so that DirWork recognises crunched files as IFF etc.) you must add the names of its spawn processes (which do the loading). A quick glance at the process list shows that these are called 'DW - More' and 'DW - Show', so adding the clients DirWork DW - More DW - Show makes DirWork a fully-working client. PROGRAM: xyzprogram (a dummy example) PROBLEM: Doesn't work as client (name is definitely right) OR: Crashes system when made a client SOLUTION: Don't add it as a client. That last one isn't meant to be a joke. If a program doesn't work when you think it ought to, you'll just have to live with the fact, or contact me with details and I'll see what I can do. Notes and Caveats ----------------- o When a client saves over a compressed file, the new file is not recompressed (I may extend xData's capabilities, but for now, recompression is unimplemented). o If a client copies a file (e.g. DirWork) the destination version is actually uncompressed. Non-clients copy files without this problem, of course. This is a pain in the ar*e, but there's not a lot I can do about it (short of adding mind-reading capabilities to xData). If this really annoys you, don't make the offending program a client. In the case of DirWork, if you have buttons to load text or view pics, selecting these will start the subprocesses without DirWork needing to know the file types, so you can remove it from the client list, just leavine 'DW - More' etc. You have to do without the auto-recognition if you do this though. I've recently been using the new BrowserII, and it's type recognition system is powerful enough to detect not only xpk files, but also what they are, WITHOUT being made a client. o Xpk'd executables will not be auto-decompressed when loaded by clients (for example, binary editors). This is to avoid conflicts with xLoadSeg, but shouldn't be a problem (the only instance I can think of when a program would want to load an executable is the binary editor example). However, xData is completely compatible with xLoadSeg. o Remember that the xpk system handles PowerPacked files, so xData does too. (So if you run PP, you can junk it). Hints ----- To temporarily remove an client from the list, start the preferences editor and prefix the client name with a ';' (or something like that). It will then fail to match program. Credits ------- xData and xDataPrefs are written entirely in C, compiled with SAS/C 5.10. The gadget imagery was created using the excellent GadToolsBox. The system patch method is based on a Commodore-supplied example (but done in an cache-friendly way). The xpk compression system is the creation of Urban Dominik Mueller. PowerPacker.library is copyright Nico François. Technical Details ----------------- Here's a brief explanation of what xData does. When a program attempts to Open() a file for reading, xData checks its client list to see if the caller is a client. If so, the file is checked to see if it is xpk'd (or PowerPacked), and if it is, the file is decompressed to the t: directory, under a unique name. Then xData opens the decompressed file and passes that handle to the caller. Normal calls to AmigaDos functions Read(), Seek() etc. result in actions on the decompressed file; when the file is Close()ed, xData checks whether or not the file was decompressed, and if it was, notes that the file is closed; it will then be really closed and deleted (at least) 5 seconds later, by the main xData task. The reason for postponing the deletion is that many programs check what the file contains (by opening it), close it and reopen it for use. Strange though this is (a Seek() would be better in many cases), it does lead to compressed files being decompressed twice (or more), which slows down performance significantly in the case of large files (and on slow processors). The postponement is for a period of 5 seconds, which seems a reasonable tradeof between speeding up reopens and keeping memory use down. Examples of programs that sometimes open twice are ViewTex and Deluxe Paint. If a client calls Examine(), xData checks if the file is compressed, and if so, returns its decompressed length. It determines the length by making some assumptions about the format of an xpk'd (or PowerPacked) file, rather than decompressing the file. This results in minimum overhead (though the overhead is greater than an ordinary Examine(), because the file has to be opened and partially read. The assumptions about file format shouldn't cause problems with future versions of the xpk system, as the format is very unlikely to change. However, I shall keep in touch with the author of xpk, Urban D. Mueller, to avoid future compatability problems. History ------- v1.1a - xDataPrefs: fixed stupid memory bug (wasn't freeing task-list). v1.1 - xDataPrefs: major enhancement - now has task list so addition of new clients is simplified. - xData: now postpones deletion for 5 seconds (for reasons why, see technical details above). v1.0a - xDataPrefs: fixed bug when entering name on empty list. - xDataPrefs: window now font-sensitive. v1.0ß - initial release. Legalese and Distribution ------------------------- It seemed fitting (and safe) to use the same disclaimer as the xpk system has, so here it is, pertaining to the xData package: This package may be freely distributed, as long as it is kept in its original, complete, and unmodified form. It may not be distributed by itself or in a commercial package of any kind without my permission. These programs are distributed in the hope that they will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. All of that said, I spend a lot of time writing and polishing stuff for release, and would appreciate any monetary contributions if you find xData useful. None of my work is shareware, but do believe people should acknowledge others' work if they rate and use it [end of plea]. Contact ------- I can be reached with comments, suggestions, bug reports, praise, money etc. at the following addresses: BEFORE END OF JULY 1993: Martin W. Scott, 23 Drum Brae North, Edinburgh EH4 8AT United Kingdom AFTER END OF JULY 1993: Martin W. Scott, c/o 557 Great Western Road, Ground flat right, Aberdeen AB1 7PA United Kingdom OR BY EMAIL UNTIL END OF JULY 1993: mws@castle.ed.ac.uk