Path: bloom-beacon.mit.edu!spool.mu.edu!howland.reston.ans.net!cs.utexas.edu!koriel!news2me.EBay.Sun.COM!engnews2.Eng.Sun.COM!samulnori!jaemin From: jaemin@samulnori.Eng.Sun.COM (Jae Min Bae) Newsgroups: alt.soft-sys.tooltalk Subject: FAQ (3 of 4) Date: 5 Jul 1994 18:03:14 GMT Organization: Sun Microsystems, Inc. Lines: 499 Distribution: world Message-ID: <2vc792$o0a@engnews2.Eng.Sun.COM> Reply-To: jaemin@samulnori.Eng.Sun.COM NNTP-Posting-Host: samulnori.eng.sun.com --- (part 3 of 4) ######################################################################### # SCOPE ------------------------------------------------------------------------- WHAT KINDS OF SCOPE ARE THERE? Currently there are only two kinds of scope in ToolTalk: session and file. (Sometimes X session is referred to, but this is really just the same as session scope, except it's session id is advertized on the _SUN_TT_SESSION property of the root X window.) NOTE: Queueing of FILE scoped Requests is currently unimplemented. ------------------------------------------------------------------------- HOW DOES FILE SCOPING WORK? (long) [NOTE: the bug mentioned in the following was fixed in release 1.1] This describes the way ToolTalk handles file-based scoping mechanisms and when ToolTalk does and does not use the ToolTalk database server for handling file-based scoping mechanisms. Specifically, because of a bug in the current release, ToolTalk does not use the DB server for any type of scoping (other then TT_OBJECT scoping) unless an explicit tt_file_join call is made. The following is an explanation of when the DB server is and is not used and how the DB server is used for scoping and how the various file-based scoping methods work. Brief Overview of the TT DB Server Duties: In the current release of ToolTalk, the DB server is used to store 3 types of information: 1) Storage of ToolTalk objects specs (see the tt_spec_write and and the other tt_spec_* API calls). 2) Storage of ToolTalk session IDs of sessions with clients that have joined a file via the tt_file_join call. 3) Storage of a file-scoped (NOT file-in-session scoped) message that is queued because the message disposition is TT_QUEUED and a handler has not yet been started that can handle the message. File-based Scoping in Patterns: With patterns, there 3 types of scopes that use files: TT_FILE, TT_BOTH, TT_FILE_IN_SESSION, TT_SESSION+tt_pattern_file_add() TT_FILE scopes to just a file. A session attribute may be set on this type of pattern to provide a file-in-session-like scoping, but a tt_session_join will NOT update the session attribute of a pattern that is TT_FILE scoped. TT_BOTH scopes to the union of interest in the file and the session. A pattern with just this scope will match messages that are scoped to either the file and/or the session. TT_FILE_IN_SESSION scopes to the intersection of interest in the file and the session. A pattern with just this scope will only match messages that are scoped to both the file and session. Using the TT_SESSION scope and calling tt_pattern_file_add will match session scoped messages that also have the file attribute set to one of the files added to the pattern by tt_pattern_file_add. If a tt_file_join call is made, it will add files to TT_FILE, TT_BOTH, and TT_FILE_IN_SESSION scoped patterns, but NOT to TT_SESSION scoped patterns. This is important because patterns that were registered via a tt_ptype_declare call can only have files added to them via a tt_file_join call. In the case of the TT_FILE_IN_SESSION scope, all you need to do is create a pattern, add the TT_FILE_IN_SESSION scope to the pattern, add a session to the pattern (or call tt_session_join after registering the pattern), add a file to the pattern, and then register the pattern. The following 2 examples register an equivalent pattern (sans error checking): Example 1: ---------- tt_open(); Tt_pattern pat = tt_create_pattern(); tt_pattern_scope_add(pat, TT_FILE_IN_SESSION); tt_pattern_file_add(pat, file); tt_pattern_session_add(pat, tt_default_session()); tt_pattern_register(pat); Example 2: ---------- tt_open(); Tt_pattern pat = tt_create_pattern(); tt_pattern_scope_add(pat, TT_FILE_IN_SESSION); tt_pattern_file_add(pat, file); tt_pattern_register(pat); tt_session_join(tt_default_session()); Because TT_FILE_IN_SESSION scope is the intersection of file interest and session interest, all the information required to properly match similarly scoped messages can be stored in the ToolTalk session and the ToolTalk DB server is never contacted. If TT_FILE scoping had been used, the second example would NOT have updated the session attribute in the pattern. If ToolTalk had no bugs, then setting the scope of a pattern (via tt_pattern_scope_add) and setting the session and/or file of a pattern (via tt_pattern_session_add and/or tt_pattern_file_add) would be all you needed to do and the pattern scopes would match the message scopes as described above. However, because of a bug in ToolTalk, if you do not explicitly join a file, the above scopes will only match messages that are scoped to both the file and the session of tt_open the pattern was registered under. This means that if you do not make an explicit call to tt_file_join, TT_FILE and TT_BOTH effectively act like TT_FILE_IN_SESSION. The reason this occurs is because of how the ToolTalk DB server is used. When a message is scoped to TT_FILE or TT_BOTH, the ToolTalk client library that is sending the message (this is where the tt_message_send call was made) contacts the TT DB server and asks for a list of ToolTalk sessions that have registered interest in the file the message is scoped to. The client library can then send the message to each ToolTalk session in the list. When a ToolTalk session receives the message from the sending client, it can then match the message to patterns that have been registered with the ToolTalk session and send the messages to any clients that have patterns that match. The problem is that the ToolTalk session that a pattern is registered on is not stored on the ToolTalk DB server until a tt_file_join call is made, not when an actual pattern is registered. The reason this is a bug is because when a tt_file_join call is made, this implicitly does a tt_pattern_file_add to every pattern that has already been registered and effectively re-registers the patterns. This means that there is no way to create several different patterns in one ToolTalk proc that all scope to different files. For example, if I make the following calls: tt_open(); Tt_pattern pat1 = tt_create_pattern(); tt_pattern_scope_add(pat1, TT_FILE); tt_pattern_register(pat1); tt_file_join(file1); Tt_pattern pat2 = tt_create_pattern(); tt_pattern_scope_add(pat2, TT_FILE); tt_pattern_register(pat2); tt_file_join(file2); Tt_pattern pat3 = tt_create_pattern(); tt_pattern_scope_add(pat3, TT_FILE); tt_pattern_register(pat3); tt_file_join(file3); Pattern pat1 will actually be scoped to 3 different files (file1, file2, and file3). Pattern pat2 will be scoped to 2 files (file2 and file3), and pattern pat3 will be scoped to just file3. I can't use just tt_pattern_file_add to set individual files on individual patterns, because a tt_pattern_file_add doesn't cause the pattern's ToolTalk session to be stored in the database. Only a tt_file_join call will do that. This bug will be fixed in a future release of ToolTalk. Some may ask, if you fix that bug, how will I scope to the union of TT_FILE_IN_SESSION scope and TT_SESSION scope with one pattern? Currently, if I use TT_BOTH on a pattern, register the pattern, and do NOT call tt_file_join, it effectively provides that ability. The answer is, when the bug is fixed you can still scope to the union of TT_FILE_IN_SESSION and TT_SESSION by adding both scopes to the same pattern: tt_open(); Tt_pattern pat = tt_create_pattern(); tt_pattern_scope_add(pat, TT_FILE_IN_SESSION); tt_pattern_scope_add(pat, TT_SESSION); tt_pattern_file_add(pat, file); tt_pattern_session_add(pat, tt_default_session()); tt_pattern_register(pat); This is the way you should implement the union of the scopes. When the file scoping bug is fixed, just adding the scope TT_BOTH will have a different behavior than it does right now. However, adding TT_BOTH and then calling tt_file_join after registering the pattern will have exactly the same behavior as it does now. File-based Scoping in Messages: Messages have the same types of file-based scoping mechanisms as patterns: TT_FILE - The message is scoped to all clients that have registered interest in a file. TT_BOTH - The message is scoped to all clients that have registered interest in the message's session and/or file. TT_FILE_IN_SESSION - The message is scoped to all clients that have registered in interest in both the message's file and session. TT_SESSION + tt_message_file_set() - I've listed this as file-based scoping mechanism because of the call to tt_message_file_set. This will scope the message to every client that has registered interest in the message's session. When the message is received by a client whose pattern matches, the receiving client can get the file name by calling tt_message_file. The bugs that applied to file-scoped patterns do NOT apply to messages. When a message is TT_FILE or TT_BOTH scoped, the ToolTalk client library will check the database server for all sessions that have clients that joined the file and send the message to all of the interested ToolTalk sessions. The ToolTalk sessions will then match the messages to the appropriate clients. NO explicit call to tt_file_join is required. In the case of a TT_FILE_IN_SESSION scoped message or a TT_SESSION scoped message that has a file set in it (via tt_message_file_set), the database server is not contacted and the message is only sent to clients that are scoped to the message's session. ------------------------------------------------------------------------- WHAT'S THE DIFFERENCE BETWEEN THE TYPES DATABASE AND THE TT_DB DIRECTORIES? WHATS THE TT_DB DIRECTORY I SEE AT THE ROOT OF MY PARTITIONS? The types database (databases actually since you can have user, host and network components) store the static ptype and otype definitions. These declare what messages applications and objects will respond to. tt_type_comp modifies this database when you add/delete/etc. static type definitions. Ttsession reads in these type files when it starts. These static type databases are stored in by default: for user ~/.tt, for system /etc/tt, and for network $OPENWINHOME/etc/tt (see TTPATH). TT_DB is a database created by rpc.ttdbserverd. It contains the associations between files in this partition, and sessions with patterns interested in these files, as well as all the object spec information for file in this partition. ------------------------------------------------------------------------ WHAT'S A HEALTHY TT_DB DATABASE LOOK LIKE? For versions 1.1 or later, healthy databases should contain the following ten files: access_table.ind, access_table.rec, file_object_map.ind, file_object_map.rec, file_table.ind, file_table.rec, file_table.var, property_table.ind, property_table.rec, and property_table.var, with permissions set to -rw-r--r-- For versions before 1.1: the following ten files are present: docoid_path.ind, docoid_path.rec, docoid_path.var, oid_access.ind, oid_access.rec, oid_container.ind, oid_container.rec, oid_prop.ind, oid_prop.rec, and oid_prop.var with permissions set to -rw--------. Note that ttdbck performs maintenance on these TT_DBs. ------------------------------------------------------------------------ WHATS RPC.TTDBSERVERD FOR? It performs 3 major functional duties: Storage of ToolTalk session IDs of sessions with clients that have joined a file via the tt_file_join call; Storage of a file-scoped (NOT file-in-session scoped) message that is queued because the message disposition is TT_QUEUED and a handler has not yet been started that can handle the message; and Storage of ToolTalk objects specs (see tt_spec_write and the other tt_spec_* API calls). ------------------------------------------------------------------------ DO TTSESSION AND RPC.TTDBSERVERD EVER COMMUNICATE? Currently no. Only libtt communicates with rpc.ttdbserverd (not ttsession). libtt passes information back and forth when needed. ------------------------------------------------------------------------ WHAT CAN HAPPEN IF I SET /USR/TMP TO A TEMPFS? (O/S version: 4.1) When there's no TT_DB and you send a file scoped message (say on an existing file like /etc/fstab), the db code wants to create a file called "/usr/tmp/NETISAM.LOCKTABLE". It then tries to do a mmap on it, and after that things go downhill & the server exits. The end result is that the database only gets partially started toward being built. The next time the server fires up, he thinks the db's there, but since it's messed up, subsequent uses of the TT_DB end up failing. Check to see what and what is not a tempfs what is read only, etc. before doing file actions. Note that the 4.1 documentation highly recommends against making /usr/tmp a tempfs. ######################################################################### # PERFORMANCE ------------------------------------------------------------------------- WHAT MESSAGE BANDWIDTH CAN BE SUPPORTED ? About 100 small messages per second in the Solaris 2.0 release, on local systems (SparcStation 2.) Performance mainly depends on how many recipients each message has -- i.e., notices that don't match any pattern are cheapest, messages that match many observers are most expensive. [The 100/sec. estimate is when there are no matching patterns.] Say you were to go into a loop that did nothing but send messages. You're going to basically saturate the TT server (ttsession.) Since ttsession is just single-threaded, what ttsession is doing is basically this: do forever { get message from sender find receivers for each receiver { deliver message } } So the more receivers there are, the longer it will take ttsession to get back to the "get message from sender" part. So your sender basically blocks down in tt_message_send() waiting for ttsession to service the RPC call. We normally say that tt_message_send() is non-blocking, but what we mean by that is that ttsession replies to the RPC call immediately, before delivering the message. So if the sending application were doing any real work, it could continue while ttsession delivers the message. But you can't send another TT message until the server finishes dealing with the previous one. ------------------------------------------------------------------------- IS THERE A LIMIT TO MESSAGE SIZE OR NUMBER OF ARGUMENTS? There's no designed-in limitation to the size of a ToolTalk message or the number of arguments. However, ToolTalk does copy the data several times, both from one spot in the client's address space to another, and by sending it across the RPC connection to and from the server. So if you put, say, a megabyte of data in a ToolTalk message, that megabyte of data is going to be copied at least 4 times: - From your storage to the ToolTalk library's storage - From the library to the ToolTalk server - From the server to the receiver's library - From the receiver's library to the final resting place. If there are processes observing the message, even more copying will take place. (If you get really excessive, you'll run out of swap space when ToolTalk copies the argument values in.) That's a lot more copying that would take place compared to more direct methods (like writing the data out to a file and sending the file name in a message, or sending a socket address in a message and opening a connection between the two processes.) Further, since the ttsession process is single-threaded, no other messages for this session are going to be delivered while this copying is going on. So, there's a possibility of bogging down the session if you send a lot of big messages. Since the increased load of big messages is just the time spent to copy the data, it really doesn't have a "cliff"; you can't say that "messages under 1K are fast and messages over 1K are slow." Basically, the time required to send drops off linearly with the size of the message. So, the principle is not "If message size is beyond X bytes, don't use ToolTalk". It's the (unfortunately) more nebulous "if you plan to send really big chunks of data really often, you might want to think about using some non-ToolTalk way to pass the data." ------------------------------------------------------------------------- WHAT'S THE MOST TIME EFFICIENT METHOD TO SEND A MESSAGE? Directly to process (TT_HANDLER) is faster than procedural messages that match only one receiver. Procedural and Otype messages are about the same. Object messages are slowest (there's an extra I/O to look up the object type.) File scope is slower than session scope. File-scoped messages are our slowest kind, but we always get at least 5/sec when we measure them, and we have a demo that runs fine blasting out several file-scoped deliveries per keystroke in an editor at touch-typing speeds. ------------------------------------------------------------------------- WHAT'S THE NETWORK OVERHEAD INVOLVED? The message is sent directly to the ttsession process for the session. (This could be across the network or not). When a pattern is registered, it also is sent directly to the ttsession process. The ttsession process matches the message against all the patterns and sends the message directly to only the processes that registered patterns that match the message. So if a process off on another machine isn't interested in a message, nothing on that machine may have to wake up and look at it. ToolTalk does *not* use hardware broadcast or multicast, so there's no overhead of sending out the message and having everybody look at it and decide if they want it. [On the other hand, if hundreds of processes register interest in the same message, ToolTalk will have to send the message out hundreds of times. ToolTalk is designed on the premise that each message is only interesting to relatively few processes.] ------------------------------------------------------------------------- DOES TOOLTALK DO LOAD BALANCING FOR HANDLING REQUESTS? ToolTalk is not a load-distribution mechanism. If there are two processes with identical patterns registered, TT ends up just picking one of them and deliver all the messages to it. If you want to do load distribution, you have to do more work, like unregistering the pattern while the process is busy, and rejecting any messages that got in before the pattern was unregistered. ------------------------------------------------------------------------- WHAT RESOURCES ARE REQUIRED BY A TOOLTALK APPLICATION? Coarse numbers say it takes several 100K of working set for a sending client, ttsession, and a receiving client to process messages. ToolTalk's memory requirements don't grow over time, so performance concerns should be addressed by faking the traffic you anticipate and seeing if you like the performance. ######################################################################### # RELIABILITY ------------------------------------------------------------------------- WHAT IF TTSESSION GETS KILLED? For versions 1.1 and beyond (also 1.0 patch level 4): When ttsession is killed, the tt_fd will become active, and most TT API calls will return TT_ERR_NOMP. (That's "No Message Passer", Message Passer being another name for the ttsession process.) Most applications just assume that this means something bad has happened to ttsession, and stop trying to send/receive ToolTalk messages. If an application wants to recover from this situation, it may be possible. Some of the things it would do include: - Recognize the TT_ERR_NOMP situation - do a tt_close() to clean up the connection from its end - re do the TT initialization sequence: -- tt_open, tt_default_session_join, tt_fd (Note that what tt_open does varies with the setting of the environment variable _SUN_TT_SESSION, the value of the _SUN_TT_SESSION property of the root X window (if it exists), some clever manipulation of these is likely necessary when restarting a crashed ttsession to take over where the last one left off. Also, other participants of the crashed session must somehow be made aware of the restarting and the new session id so that they can recover.) -- re-register all patterns and re-declare ptypes.` - continue on The following are lost when ttsession crashes: - patterns registered by procids in the crashed session - outstanding requests from procids in the crashed session - messages bequeathed via tt_message_send_on_exit() by procids in the crashed session. - session props - session-queued messages ------------------------------------------------------------------------- WHAT IF RPC.TTDBSERVERD GETS KILLED? If rpc.ttdbserverd exits unexpectedly, there may be no effect at all, since inetd (Suns) will start a new one to replace it. The worst that is supposed to happen is that one or more API calls dealing with specs or scoped-to files on that fileserver may return TT_ERR_DBAVAIL. If the call returns TT_OK, the dbserver will update the TT databases appropriately, either immediately or when a new dbserver reads the crash recovery log. If the dbserver crashes, data may be temporarily unavailable as zero or more API calls will fail with TT_ERR_DBAVAIL, but no data will be lost. ------------------------------------------------------------------------- WHAT IF A HOST IS DOWN OR A LINK IS DOWN? When TCP notices this, the TCP connection breaks. When a process' connection to ttsession breaks, ttsession acts as if the process exited. All the patterns are cleaned up. The process will get TT_ERR_NOMP next time it tries to send or receive. ------------------------------------------------------------------------- WHAT'S TT_CLOSE DO AND WHAT HAPPENS IF I DON'T CALL IT? On tt_close(), ttsession only closes the current procid, and if that was the last procid to close, all other baggage (structures, etc... all the tt structures created since tt_open()) are cleaned up/out/destroyed. If your program doesn't call tt_close? - if tt_message_send_on_exit() was called previously, then the queued message is sent out before cleanup. Note: tt_message_send_on_exit() first appears in ToolTalk 1.1. NOTE: don't do close() on the fd returned by tt_fd - use tt_close(). Failure to do this will cause your file descriptor count to rise upon successive tt_open() and close() calls. ------------------------------------------------------------------------- IS MESSAGE DELIVERY GUARANTEED ON A NETWORK? Messages are sent via RPC on TCP/IP, so delivery is reliable. ------------------------------------------------------------------------- IS THERE A TEMPORAL SEQUENCE OF MESSAGE DELIVERY? Between a given sender and receiver, message sequence is preserved. That is, if process A sends messages M1 and then later M2, and process B receives both M1 and M2, process B will get M1 before M2. There are two exceptions, both fairly specialized: 1. If a hander receives a message and then rejects it (with tt_message_receive) the request gets redispatched to a second process. In the meantime, (while the first handler is mulling over the request and deciding whether to reply or reject) ToolTalk continues to deliver messages. Those messages can appear to "pass" the first request. 2. If message queueing is used, a process will receive queued messages when it declares a ptype containing the pattern that caused the queuing. That might happen after the process has already received other messages from the sending process...