;---------------------------------------------------------------------------- Jan 7 91 - we need a skeleton device driver; preferably a RAM drive in ML source to consume a file: get: name size *sparrwed flags *comments *datestamp actual position on media create: master checksum incremental checksum kind of file (figure this out) should be able to take apart qbak disks (even if not ordered) files should be consumed relative to their sub-directories. once a file is consumed it appears at the current level of the workspace if an entire media is consumed then that media appears in the workspace from workbench you can move the file, from cli you can rename it if a file has duplicates then the user should somehow be informed about the duplicates, and where they are. ;---------------------------------------------------------------------------- Jan 7 91 so for a minimum implementation: read a disk, all contents into a fascimile image in the mega database present the mega database via a device driver, visible to wb and cli. search the mega database for duplicate instances, and make locations visible to user immediately. ;---------------------------------------------------------------------------- another way: read in a disk, stash contents into mega (assign disk id number) print a list of all duplicates support building of a master image from a master image can do diff for near duplicates ;------------------------------------------------------------------------ read dir (or media) tell-database-i-am-in-dir-x if any sub-dirs then call self recursively with subdir loop: read next file, add to database,goto loop till done tell-database-i-am-out-of-dir-x exit i-am-in-dir-x add x name to database, remember position i-am-out-of-dir set jump of array size in old position as end-of-dir add-to-database array: is file or dir, if dir then marks off certain # of array els ;---------------------------------------------------------------------- - need to be able to flexibly add and remove files from conceptual level of dir. current file systems, such as those on harddrives probably have a hash table off to a subdir, which then has its own hash table off to its portions. - we can emulate this by having each dir in an array, linked together by a flexible scheme - entire database should be checksummable by segment to prevent trashing from some outside source - we could actually "have" dirs on disk, and their contents would be only single files describing that disks normal contents - then at the top level we'd have media names rather than ordinary dirs? - a "media" should also be a dir i guess, in that all the subdirs and files in it are given the same media id number. where the name of the "media" as a dir is the same as the agreed upon id number, and all subdirs and stuff just keep their original names. - the database should be all in ram at a time however i think? hmmm but it can probably be around 1/2 meg, and more error prone that way... ;----------------------------------------------------------------------- ok so the scheme is: - we assign a media number for the disk we are consuming, that becomes its dir, "consume" then call "create-dir" create-dir: *doesn't* ask for original media (of course) creates dir in image move: rename: *doesn't* ask for original media, rather just updates internal image. can move into garbage, may be special command to forget a duplicate. read: write: edit: delete: operates on specified media (may ask user to insert disk #x) - theory is that you should be able to do actual work from the virtual enviroment, all normal i/o access to the file works as normal, its just its conceived position that is indirect/deferred. open old file open new file read write close seek waitchar examine object examine next diskinfo parent delete object create dir locate object copy dir free lock set protect setcomment rename object inhibit rename disk ;------------------------------------------------------------------------ january 16th: - accept and handle all dos device packets: DIE OPENRW OPENOLD OPENOLD READ WRITE CLOSE SEEK - do normal stuff, like checking if file is in this dir - pass request onto the real device (with full file path) EXAMINE NEXT EXAMINE OBJECT FREE LOCK LOCATE OBJECT INFO PARENT DELETE OBJECT CREATE DIR COPY DIR SET PROTECT SET COMMENT RENAME OBJECT - talk to my own information (pass request to my info handler) DISK INFO - talk to my own info dir inhibit rename morecache waitchar flush rawmode - dont support ;------------------------------------------------------------------------ There is an indirect level of requests, some requests get passed onto the appropriate media, other requests get intercepted and passed to my own handler. In a sense there are three main segments 1) the VIR: device which gets all initial requests. 2) the normal device which is an i/o request destination. 3) another not-visible "device" of mine which provides the functionality of VIR: Part (3) must have the following modular upgradable components receive and return the desired info from standard requests, as based on an internal database the internal database being structured in such a way as that it is accessible and extensible possibly being located on disk for sturdyness ;---------------------------------------------------------------------------- possible structures for vir: internal database CHEAP - doubly linked list of names, with branches at dirs supplemented by info structure per file IF RAM - written to disk occasionaly as backup - always loads up as a clean minimal memory whenever possible IF DISK - mockup directory structure GOOD - Linked list of mini-hashed tables using the perfect hash - each hashed reference IF RAM - each hashed reference points to a chunk of data with all file quals IF DISK - each hash table has an associated file, acts same as above General - there are browse commands and create/delete commands mostly - all commands get passed to whatever handler, they get back the information they want... dirs - each name points to a place, if its a dir then that place is either a sub hash or a new linked list... required info: link.l forward name 130 chars size.l protect.b edit date.l master csum.l csums.l (little csums) comments (if any) get: name size sparrwed flags comments datestamp actual position on media create: master checksum incremental checksum kind of file (figure this out) should be able to take apart qbak disks (even if not ordered) - can we use the management implementation scheme already in place in a standard ram drive? - all packet commands act as normal except these ones: OPENNEW - is not permitted, return some fail condition OPENRW - OPENOLD READ WRITE CLOSE SEEK - passes request "verbatim" onto real owner flags the client as reply path ;--------------------------------------------------------------------------- dear matt here is a case where you're actually releasing something to the public domain is going to come back and help you in turn. I wish that I were so lucky! (if I could keep my stuff from being censored!). - lattice requires a -V on this - Note that we can use "handler" which is supposedly better. - Minimally modify the device driver - Submit modified device to Matt Dillion for checking if bug free - Allow all file management to still function - Remove OpenNew - Add "real owner" field to ram file node - Read/Write requests get passed to real owner device - force all files to have a very small size (preferably 0) - add these functions - reload vir: (from disk database) - unload vir: (to disk database) - coalesce: (search and merge duplicate files) - toolbox space ;---------------------------------------------------------------------------- - write a "learn" function acquire media references onto virtual device (have media reference) imbed an "info" function into this learn vir: to vir: should acquire others special junk talks to a special vir: packet - pass requests onto device (named reference?) - either must have internal "positive id" - ;-------------------------------------------------------------------------- - LEARN is just a COPY which writes a special packet type to host - build up info about next file, export to vir: - do directories etc. - open new, as called by learner writes a special header to file, and thats all - open old automatically checks if header is mine, if mine then patches request onto real destination. - each of my format files has its full path name imbedded in itself - how do we refer to the specific device? get source for a copy command (such as arp copy etc). typically - a user issues a dos specific command - whatever utility passes the path along to DOS - dos parses out the device - dos attempts to access that device, if not present it requests it - the file is accessed changes - i need to supply the positive id of a media type, not merely the name but the internal def'n unique identifier data - this may not be possible to issue at a high level? why? how do people discriminate normally? simply by device? so i could just say insert disk in df0? - i guess i really have to ask for not merely a named device but a specific handle for a device... ;--------------------------------------------------------------------------- January 18th 1991 SIMPLE IMPLEMENTATION SET: FEATURES: - ACQUISITION -> learning media (quals/csums) - PASS THRU -> requesting external online/offline media - COALESCING -> locating duplicates (csums) INTERNAL PHYSICAL COMPONENTS: - wild card handling + recursive dir descent - file and directory management - amigados packet handing - handler skeleton VISIBLE PHYSICAL COMPONENTS: - learner - coalescer - device driver - (device driver internal data block) INTERNAL LOGICAL COMPONENTS: - all I/O commands are pass thru (to real device) - all management commands are managed by moi - the stub which passes off to either of above ATTACK: - clean up Matts code into C++ "strands" of functionality (seperate all internal physical components) - First write AQUISITION such that RAM DRIVE can aquire indirect references - Write PASS THRU (possibly at same time) such that RAM DEVICE is now functioning virtually. - Rewrite database such that it is disk resident always - Write COALESCE - Write ARCHIVAL UTILITY - Write TOOLSPACE with CLOCK, POWEROFF, TRASHCAN, SCRIPT-EXE ;--------------------------------------------------------------------------- Positive identification of offline media: - We can't use a straight numbering scheme, unless we retain the disk positive ID as well (which is feasible) - Does each file retain its entire path? Yes, except perhaps its master media. The reason being that the file can be moved around the vir: space, but still retains its linkage to its physical existance. The media positive id could be maintained as a seperate node, always referenced however, by the file. - Unless some obvious, simple and efficient scheme can be arranged for storing media positive ids seperately I may as well store it specifically for each file. This wastes tons of ram, so I'll have to think about it. - Agreed upon ID numbers should grow by multiples of 100 just to leave insertion room if required. - AmigaDos implements a mechanism to request a positively ID'ed media. How do I patch into that mechanism? [ I dunno, emulate it! ] - What constitutes a positive ID under AmigaDOS (what do I stash). - I can emulate the AmigaDOS positive ID scheme if I stash away the media datestamp,checksum,name, then perform my own search for that media. If I fail to find the media I am looking for I guess I should fail, although there should be an option to force acceptance of generic media's. - When my device driver attempts to pass the DOS packet off to the proper device its going to have to search the device list for the positive ID qualities, and then use the general quals (name) to open up that device for packet communications, then pass off the packet to it (i presume). Which brings us to another question: - I presume I can open a DOS device quite normally and pass off a packet to that device. But it would be truly grand if I could drop myself out of the process...simply connect my client to their real request and not require my presence at all. - I could even support new writes to the VIR: media in this manner, where the destination was that of the current directory (and perhaps the top level was just a RAM disk). Issues to code: - (i should get positive ID info from learner, and stash it away) - Attempt to open a DOS device, and drop a packet into it. As a test I can have all the I/O to VIR: redirect itself in a natural manner to DR1: - Assignments do this, how? -> Probably have a device definition with a message structure also, and that receiving code fragment passes it along very much the way I want too. [ SB suggests this ] - Soooo... our emulation figures out the device name, positively attempts to identify it, (or requests it be brought online) i then find the message port of the device and patch the request through to it - in such a way that it doesn't return to me. - PipeHandler imbeds pipe specific commands into a device specific reference, like P:abc/def. What can I use that for? ;------------------------------------------------------------------------- Jan 18th 1991 - It appears that the command "P:abc/def" gets passed in its entirety to the device driver (in Arg2 of the packet). This actually occurs because the shell is trying to execute that command. However at some earlier point the appropriate device has been determined, elsewise P: wouldn't have gotten the message... [ Pickpacket confirms this...DOS determines the device, nicely enough we get the 'VIR:' header with all files, (sometimes we may be on 'VIR' so it doesn't really help, or matter ] - lets see exactly what a "copy" command passes eh? [ not enuf (no source path (ARG!!))...but maybe we can hack it ] ;------------------------------------------------------------------------- Jan 18th 1991 editing new files on vir: VIR: generates a positive device ID and absolute file position using an internal default, be that your main harddrive, or ram:, vir: will always put the file at the root level, giving a unique numeric name appendum if file names collide (in which case you still refer to the vir: instance by its normal name). From that point on the file is treated as an ordinary existing file. reading (existing) files on vir: editing (existing) files on vir: all requests are passed to the appropriate device, using the positive device id and absolute file position info that vir keeps. closing edited files on vir: when you close a modified file vir: rebuilds that files checksums. learning (file aquisition): there is a "learn" command which acts exactly like "copy" as far as the user can tell. Internally passes a little bit of extra-info which specifies at the outset the files owner positive id... if it is possible the 'copy' command will be detected and changed to a 'learn'. how about copy from vir: to vir:? ;------------------------- x to vir: (new) -> either non-permissible or copy to default x to vir: -> patch request to owner vir: to vir: -> just copy reference (implicit or explicit?) vir: to x: -> patch request to owner x: to x: -> who cares ;-------------------------------------------------------------------------- new-file alternatives: 1) write to the media of which i am a sub dir of Not acceptable, because it destroys portability 2) copy to default, a poor but simple solution 3) copy with heirarchy if possible (so-so) 4) copy create heirarchy (so-so) 5) disallow (so-so) Best not to implement some arbitrary scheme lacking of exposure to mainframe virtual storage system solutions; so simpler is better. ;---------------------------------------- anyway new file looks like a hassle; just disable for now, figure out later... ;---------------------------------------------------------- jan 19th 1991 ok so we're left with the issue of positive media identification fib_diskkey is what? NOT DOCUMENTED... SHIT hmmmm.... ;---------------------------------------------------------------- operating on a positive id search device list for that device name if its a physical media type device then also search for positive id get message port of device I wonder how DOS does the search? does it forbid etc? ;------------------------------------------ all i/o gets passed onto named device pull file id tag: "filepath/file-orig-name" device pointer (has device name, positive id if any, handle) open a message port with the device (just LOCK opened one up with us) ;--------------------------------------------- handling old stuff 1) packet request comes in 2) if management request then handle as normal 3) if i/o request then: - get device name, device positive id - get process identifier port struct MsgPort *pro = DeviceProc("df0:");/* example */ - (or fail) - for now reissue the packet and wait, then reply - later pass packet verbatim to port, don't reply ;--------------------------------------------------------------------------- ACTION_OPENRW: ACTION_OPENOLD: ACTION_OPENNEW: - get pointer to devinfo - get msgport of named device, attach to handle - attach subdir to filename - patch request thru to device ACTION_READ: ACTION_WRITE: ACTION_CLOSE: ACTION_SEEK: - patch request thru to device (use handle messageport) ACTION_DIE: ACTION_EXAMINE_NEXT: ACTION_EXAMINE_OBJECT: ACTION_INFO: ACTION_DISK_INFO: ACTION_PARENT: ACTION_DELETE_OBJECT: ACTION_CREATE_DIR: ACTION_LOCATE_OBJECT: ACTION_COPY_DIR: ACTION_FREE_LOCK: ACTION_SET_PROTECT: ACTION_SET_COMMENT: ACTION_RENAME_OBJECT: ACTION_INHIBIT: ACTION_RENAME_DISK: ACTION_MORECACHE: ACTION_WAIT_CHAR: ACTION_FLUSH: ACTION_RAWMODE: - all act as normal - get pointer to devinfo - get msgport of named device, attach to handle must have devinfo available by pointer just call a gumby routine to return this vector - attach subdir to filename (don't bother for now) - patch request thru to device ;-------------------------------------------------------------- The driver will have 2 modes 1) RAMDISK Mode This is used to "learn" files The learn command submits "ordinary" files to save onto vir: thus any destination can be used for learn (ie: RAM:) All reads and writes simply operate on the RAMDRIVE as a RAMDRIVE 2) Virtual Mode All reads and writes are patched through to their real owner A special check is performed to only accept VIR: type files COMMENTS: a) VIRTUAL mode is always on as far as the client is concerned, learn triggers into RAMDRIVE mode, and then back out. b) There is a special user do-able way to goto RAMDRIVE MODE thus user can share files between different VIR:'s, or make a backup of the VIR: media. c) 'learn' is built directly into the DEVICE, issuing a LEARN command "makes" a learn function! or the command can be specified as one batch... 'learn_df0:mymedia_to_vir:myfile' if it accepts spaces then thats great! d) coalesce simply eliminates the second media reference of a pair which are the same... ;------------------------- AMIGADOS TECH REF PAGE 268 !!!!! To distinguish disks with the same name AmigaDOS timestamps the volume on creation, then saves timestamp in list structure. new handler process per instance of df0:, each having own list struct, if not inserted then save-away any active locks + presumably VolDays... on reinsert then probably looks for VolDays, reattaches locks learn - build up dir,path,filename,checksums, type submit file as a packet to specified device ;---------------------------------------------------------------------------- Sunday Jan 20 00:30 ok so the learn is coded and works fine now: device gets incoming "open" request 1) search devinfo list for named device 2) possibly request media insertion (wait around till its inserted or aborted) (perhaps make it so that it won't be let go before step #5) 3) make sure that positive id matches 4) substitute entire path into lock 5) patch request to that device - need mode to switch between being ramdisk and being virtual 1) ram mode is just normal 2) virtual mode patches all read/write requests what are the trade offs of storing FIB info directly into RAM FIB struct versus keeping it as a file? - there is no ram fragmentation benefit either way (ie better ram solutions can be figured out for either) - RAM FIB might be faster (marginally) - file format: external modules like learn are simpler (excellent reason -> allows human handling, truckability) (also makes learn useful for non-internal enviroments) ;------------------------------------------------------------------------ Jan 22 91 ok so now the requests are being patched off to the proper handler but we're encountering a bug which I think goes like this: 1) client finds my process ID and submits a LOCK request 2) I patch the packet off to other process ID which returns LOCK 3) client passes LOCK to me for a read 4) I don't understand the lock because it isn't mine; failure. to prove if this is in fact the problem; I should patch ALL requests through, and see if I get a READ packet at all. If I don't then that suggests that the LOCK carries its own PROCESS ID, and that the bug lies elsewhere. IE the command READ is generic -> but we know thats not true because you must call the desired handler. If this is the problem then the solution is like so: 1) client finds my process and submits LOCK request (as before) 2) I submit same request to other process ID, and wait for reply. 3) On success I tag on my own info to the LOCK 4) client passes LOCK to me for a read 5) I check to see if my OWN info is appended 6) If so then I patch LOCK onto other process So the issue is that the OPEN and READ requests are both going to come to me; and I'll need some conscious bridge to remember if I opened it; and which device I opened it of. I think all I need to stash in the LOCK is the actual device process. It would be nicer if all the low level AMIGADOS functions extracted process ID from the lock itself, rather than making that a user responsibility. This is the root of the problem. Basically I'm just acting as a DEVICE ARBITRATOR - - - I can think of the process as 2 parts: 1) Determining destination DEVICE 2) Patching *ALL* related requests to that device. In my code the process seems a bit cloudy because on the OPEN I do both more or less atomically. I should seperate these out to make the relationship more clear. Possibly having a flag which states if this request is to be specially reserved as a patch request. So in that case do I really have to wait for an ACK from the OPEN? Somehow if I could stash my information directly into the initial LOCK I would be freed of the responsibility of having to intercept and reply to any packets, I could just pass them straight on. Of course it is true that the initial packet must also be modified, and anyway there is a solution, and it looks as if things like getting the initial LOCK back so I can patch it, and modifying the submitted packet in the first place are going to prevent me from having an idealized solution for the OPEN request anyway. All the other requests can be straight patches presumably. Ok so now lets test if my initial premise is correct. I'll do this by patching all requests so my ramdisk justs acts like a debugger peeking into packet transit between two other entities. [ YES - the premise appears to be correct; with some qualifications ] 1) The reason I didn't see any READ's before when I attempted this is because I forgot, I'm supposed to be addressing the VIR device. 2) There seems to be some kind of general LOCK going on, right away 3) The LIST and DIRECTORY commands only require 1 access? WHY? This is very important to figure out. I can't understand why I am not getting an EXNEXT? Lets look at 3 We get a Packet 8 -> LocateObject It returns a LOCK *** Presumably the LOCK has the MESSAGE PORT in it ALSO *** Interestingly enough "list" never releases the LOCK it gets (probably bound up with directory stuff...)...Or something like it -> It does act normally if you're not ON the device, but if you are local to it then it *knows* about the lock already available, and simply uses it? (hmmm...actually maybe 'list', 'ls' and 'dir' act the same) anyway so if you're on VIR and you do a DIR where all requests are being transfered over to DR1: VIR won't get any packet requests at all -> presumably because the device is LOCKED and the LOCK is cached *and* the lock has a copy of the message port. [ yes ] that appears to be correct -> the lock does have the msg port. Ok so lets review everything again: 1) we're going to patch thru all I/O packets by extracting some special info we hide in the FileHandle at OPEN Time (because they call me multiple times). 2) all directory and lock related stuff we normally just handle ourselves, but if we wanted to patch them through then we would just patch them and forget them...thats all. If somehow somebody doesn't use the LOCK port, but the original PORT (me) then that gets patched also anyway so there is no chance for collision. If they request a file handle based on incorrect information...well I just return "file not found"...no problems... So step 1) is the most important; Is the earlier scheme the best way? 1 well the I/O can either use my FileHandle with their Port and handle hidden in it -> and if user modifies my handle then I modify theirs in kind 2 Or I stash my info into their FileHandle and hope that they don't trash it. 3 Or I create a Hybrid handle of my own, which stashes their port and handle, as in the first case, but continually re-updates it? Or is this functionally equivilant to case 1? I suspect the solution is a combination of the above methods... ;---------------------------------------------------------------------------- The way DOS works: Locate.Object parameters: lock on parent director or NULL (root) name of file comments: presumably there is implicit the parameter of the correct device it requires no parameters other than name in a minimum case, this is why I would refer to it as the "bootstrap" entry gate. Open Old File parameters: Lock on its directory (may be zero = root) Name comments: now it fills up your filehandle (passed by reference) *** lets look at my vir again: client submits an "open" request keyed to the VIR subdir LOCK that the file is apparently in. i want to now open of another device in another directory, but i have to submit the parent lock apparently; i don't see why i can't directly open a file in a sub dir. my ACTION_OPEN doesn't support it though (hmmm...I guess the driver doesn't know about the idea of directories). (so what do you get if you submit a subdir???...1) either the whole thing is submitted -OR- there are multiple calls to tradverse the layers; check out now please. [ the lock was submitted cold - with a parent lock of 0 ] ie: 1> list subdir/mydir/myfile [ resultant issued packet = 8 args: LOCK,LOCK,MODE ] 1> list :subdir/mydir/myfile [ resultant issued packet = 8 args: 0,LOCK,MODE ] This suggests that at least for the examine function the DEVICE does understand directory structures, (thus making life much easier for me) now lets check READ: 1> type :subdir/mydir/myfile [ result = packet 1005 args: FileHandle,0,Name ] 2> type subdir/mydir/myfile [ result = packet 1005 args: FileHandle,LOCK,Name ] Ok thats great! So everything works same as before, but i should make sure to clear the lock field of the open file request (although I understand that Matts handler is performing the extra duty, and in fact other handlers might not bother... - To be on the safe side I might want to just steal the ':' LOCK from the DevInfo struct. *** but actually I know that I don't have to do that *because* its not Matts handler issuing the quasi-legal request, but other programs...so its quite obvious that a 0 LOCK is supported *** Are there any ramifications here about locking the device? (so somebody can't remove it from the system on me?) ok so lets cover everything again.... 0) client is browsing around in vir: 1) all directory handling and renaming is supported right in the handler - acting in its RAMDRIVE way. 2) the examine command is mostly normal except it does a tiny bit of extra work and returns the indirect file size rather than the ramdrives' file size. 3) the open_new command we haven't dealt with, or vir to vir copy also. 4) the open_old figures out and accesses the device (perhaps asking that it be brought online) and gets a file handle on the file. It retains the device message port, and the device and file lock. the client gets back what appears to be a handle on the actual file. The open is structured in source in such a way as that it can rely on (as its second) the same routine that the read/write routine relys on. 5) the read/write commands (yes write is supported even if open_new isnt) pass the requests onto the real device; based on message port info and the proper device lock and file lock. in general all modifications to the lock or filehandle are carried through by vir: the ultimate mediator. 6) learn commands, coalesce, load and save database, turn off the virtual mode of the driver momentarily (process relative to their own operation only) and treat vir: as an ordinary ramdisk. ;------------------------------------------------------- Jan 23 91 An alternative to the entire scheme discussed above is the following: Intercept the calls to _LVOOpen, _LVORead, _LVOWrite (one level up) such that when the system searched for a particular file, it would find it - except that the device would be different. I would watch and intercept calls to _LVOOpen accessing "vir:" and remap them over based on my internal database perhaps this was the better way...but oh well I'm to far into it now. ;-------------------------------------------------------------------- Feb 1 91 ok so the vir: device is working now, we can do the following: - learn new files (should csum each key, assign media #'s) - virtually access files (should lock subdir if name >128) now we need to deal with: - coalesce intelligently (or just find duplicates) - writing new files to vir: NEW FILES: just write to dh0: with a datestamp unique name. (or just write to VIR: as a non-virtual file) COALESCE - accept wildcard source, wildcard dest - compare all sources to all dests - retag dests (copy one to the other) or just print dup addrs ;--------------------------------------------------------- march 2 1991 tidied up vir: a bit, more robust determined that the FIB_PADDING or FIB_RESERVED is really and truly RESERVED unfortunately....I'm currently using the comment field, but I'll have to think more about it. wrote a coalesce based on 'du' from Stuart Mitchell now trying to make coalesce be smart about its directory comparison...not every compare the same file twice and do so implicitly - by copying the anchor chain, and searching next file on as a comparator...not all files again.. ...wait! i can't do that...cause src and dest are not neccessarily the same...i guess i could have an optimized mode (anyway i couldn't get it to work so....)