SE USING CGI

Add To MyInformIT

< Back Contents Next >

Author: Michael Erwin
Retail Price: $59.99
Publisher: Que
ISBN: 0789707403
Publication Date: 4/19/96
Pages: 1000

Chapter 15 - Generating HTML Documents in Real Time

HyperText Markup Language (HTML) lets you publish text and graphics in a platform-independent way. Using HTML, you can easily, via embedded links, weave a world full of sites together.

In this chapter, you examine static and dynamic HTML, concentrating on the latter. Dynamic, or real-time, HTML extends the viability of the Web far beyond its original conception.

You learn what makes real-time HTML tick and how to produce it in a variety of ways. Specifically, this chapter provides

Static HTML

Need to review the complete works of Mark Twain? Want to find the address of a manufacturer in Taiwan? Need the phone number for the White House? Ever wondered how to spell floccinaucinihilipilificatrix, or what it means? (Yes, that's a real word. You won't find it in any dictionary except the Old English Dictionary, though, so put away your Webster's Collegiate.)

The answers are only as far away as your favorite search engine. These types of references are perfectly suited to the Web. They seldom, if ever, need revision; after they're written and thrown on a page, other sites can establish links to them, search engines can catalog them, and you can find them-today, tomorrow, next week, or next year. Because the markup language used to create these pages is HTML and the content of the pages is static (relatively unchanging), such pages are called static HTML.

But what if you want to know the stock prices-not 10 hours ago or 10 days ago, but right now? What if you want to know the arrival time of American Airlines Flight 101? What if you need to know the ambient temperature in Brisbane as of 30 seconds ago?

In these cases, static documents just won't do. Not even if a diligent, never-sleeping Webmaster does his level best to keep the documents updated. For these sorts of applications, you need real-time, or dynamic, HTML.

Real-Time HTML

All CGI-generated HTML is technically "real-time" in that it's generated on the fly, right when it's needed. In data processing circles, however, the term refers more to the data itself than the production thereof.

Therefore, a CGI program that talks to a hardware port and retrieves the current temperature and then generates HTML to report it would be considered real-time. A CGI program that looks up your birthday in a database wouldn't.

In this chapter, I don't worry too much about the technical definitions. I call all CGI programs that produce time-sensitive or user-sensitive output "real-time." This includes uses such as the following:

Benefits of Real-Time HTML

The prime, and most immediately apparent, benefit of real-time HTML is that the information is fresh. Getting the stock market report from yesterday's closing is one thing; finding the value of a specific stock right this minute is something else altogether. The information has different value to the consumer. People pay for up-to-the-minute information.

Another, somewhat less obvious, benefit is that real-time HTML can make your pages seem livelier. For example, in the next chapter you examine a page counter and a random-quote generator. You can put them together on a page to produce output like this:

And so on. Granted, this particular example is rather trivial. Many readers may not even notice that the wording changes each time, and those who do won't have their lives, careers, or religion changed by it. But this example should give you an idea of the sorts of pages you can make by using real-time document generation.

Methods of Generating Real-Time HTML

The following are the four main methods of generating dynamic pages:

In the following sections, you tackle them in order.

Scheduled Jobs

A scheduled job is a batch file, shell script, or other program that runs at a regular interval. These jobs usually run in the background-that is, invisibly and independent of the foreground task-and may run once a month, once a day, or once a minute. The interval is up to you. A special case is the program that runs continuously (called a daemon in the UNIX world, and a service in the Windows NT world), spending most of its time asleep, and waking up only periodically to accomplish some task. Usually, though, background jobs are scheduled. They run at the appointed time, do their jobs, and quit, only to repeat at the next scheduled time.

The method of scheduling varies from operating system to operating system. In UNIX, you find the cron utility most appropriate. Under Windows NT, the AT command makes the most sense.

Scheduled jobs are useful for information that changes infrequently but regularly. A quote-of-the-day program is probably the best example. You don't need to invoke a CGI program to retrieve or regenerate a program that changes only once a day. It's far better to write a program that updates your HTML at midnight and then let the page get retrieved normally.

Regular CGI or SSI

For page counters and similar programs, either CGI or SSI (see the next chapter for examples of SSI) make the most sense. The kind of information being generated is what drives your choice. Because a page count changes only when a page is retrieved, updating it then makes sense. A scheduled job is clearly inadequate for up-to-the-moment data, and the remaining methods-Client Pull and Server Push-are inappropriate because you don't want a continuous update.

A trivial, but nonetheless useful, example of using CGI to provide dynamic HTML is a CGI program that redirects the browser to a static page appropriate for that browser. For this example, assume that you want to provide different pages for each of the following browser types: Netscape, Microsoft Internet Explorer, and Lynx. Any browser that can't be identified as one of these three gets redirected to a generic page.

ByAgent is a complete working sample of using CGI to provide a dynamic response. You should be able to compile it for any platform. You can find the source, plus sample HTML files and a compiled executable for the 32-bit Windows NT/Windows 95 environment, on the CD-ROM accompanying this book.

Compile the code (as shown in listing 15.1) and name it byagent.exe. Put the compiled executable in your CGI-BIN directory. If you're using a 32-bit Windows environment, you can skip the compile step and just copy byagent.exe from the CD-ROM to your CGI-BIN directory.

To test this program, you need to create a number of static HTML files. The first will be used to demonstrate the others. Call it default.htm.

As you can see, this code is fairly straightforward. If your cgi-bin directory is called something else, correct the link in the preceding code.

Now you can create four individual pages: one for Netscape, called netscape.html (see listing 15.2); one for Lynx, called lynx.html (see listing 15.3); one for Microsoft Internet Explorer, called msie.html (see listing 15.4); and one for everyone else, called generic.html (see listing 15.5).

Put these files together in a directory, and load default.htm into your browser. Click the test link. You should see the page corresponding to your browser. Listing 15.6 shows the actual code to accomplish the redirection.

As you can see, the preceding code is fairly simple. The comments far outweigh the lines of code. The only tricky bits to this program are remembering to format the redirection header correctly, and remembering that Microsoft Internet Explorer appears to be "Mozilla" (Netscape) if you don't look carefully.

In your own program, you may want to incorporate some mechanism to allow the secondary pages to live in a different directory, or even on a different server, just by changing the Location information. You may also consider generating the correct HTML on the fly rather than redirect the browser to an existing static page. Now that you know how to identify the browser and do redirection, your imagination is the only limit.

Client Pull

Several other browsers now support Client Pull, a Netscape enhancement, but you should be careful when writing your HTML to include options for browsers that can't deal with it.

In typical browsing, a user clicks a link and retrieves a document. With Client Pull, that document comes back with extra instructions-directives to reload the page or to go to another URL altogether.

Client Pull works via the <META HTTP-EQUIV> tag, which must be part of the HTML header (that is, before any text or graphics are displayed). When the browser sees the <META> tag, it interprets the contents as an HTTP header. Because HTTP headers already support automatic refresh and redirection, not much magic is involved at all. Normally, the server or CGI program is responsible for sending the HTTP headers. Netscape's clever idea was to allow additional HTTP headers inside a document.

Say you have a Web page that reports election returns. A background process of some sort reads the precinct numbers from a Reuters connection (why not?) and once every 10 seconds rewrites your Web page with the current data. The client can hit the reload button every 10 seconds to see the new data, but you want to make that process automatic. Listing 15.7 shows how to do it.

Note the <META HTTP-EQUIV> line. This line causes the browser to refresh the page once every 10 seconds. Of course, for this example to be useful, you need to have some other process updating the page in the background, but this example works-it will reload the page once every 10 seconds.

Why once every 10 seconds? Because each time it fetches the document, the browser sees the instruction to load it again 10 seconds later. The instruction is a "one-shot" instruction. It doesn't tell the browser to load the page every 10 seconds from now until doomsday; it just says to load the page again 10 seconds from now.

You also can use Client Pull to redirect the browser to another page. In listing 15.8, the browser goes to http://www.microsoft.com/ after 5 seconds.

This example uses the URL= syntax to tell the browser to go to the specified URL. The delay is set to 5 seconds. Note also that text is included to explain what's going on, and a manual link is included for people who have browsers that don't support Client Pull.

You can set up a chain of redirection, too. In the simplest configuration, this chain would be two files that refer to each other, as listing 15.9 shows.

When the user first loads page1.html, he or she gets to see page1.html for one second. Then the browser fetches page2.html. Page2.html sticks around for one second and then switches back to page1.html. This process continues until the user goes elsewhere or shuts down his or her browser.

Server Push

Server Push works with more browsers than does Client Pull, but it's still limited. If you use this technique, be aware that some users can't see your splendid achievements.

The Server Push method relies on a variant of the MIME type multipart/mixed called multipart/x-mixed-replace. Like the standard multipart/mixed, this MIME type can contain an arbitrary number of segments, each of which can be almost any type of information. You accomplish Server Push by outputting continuous data using this MIME type, thus keeping the connection to the browser open and continuously refreshing the browser's display.

Server Push isn't a browser trick; you need to write a CGI program that outputs the correct HTTP headers, MIME headers, and data. Server Push isn't for the faint-hearted. To pull it off, you need to understand and use just about every CGI trick in the book.

A Server Push continues until the client clicks the Stop button or until the CGI program outputs a termination sequence. Because the connection is left open all the time, a Server Push is more efficient than a Client Pull. On the other hand, your CGI program is running continuously, consuming bandwidth on the network pipe and resources on the server.

In a standard multipart/mixed document, the headers and data would look something like listing 15.10.

The boundary is an arbitrary string of characters used to demarcate the sections of the multipart document. You use whatever you specify on the first header for the remainder of the document. In this example, BoundaryString is the boundary marker.

Each section of the document begins with two dashes and the boundary marker on a line by itself. Immediately thereafter, you must specify the content type for that section. Like a normal header, the content type is followed by one blank line. You then output the content for that section. The last section is terminated by a standard boundary marker with two dashes at the beginning and end of the line.

Server Push uses the same general format but takes advantage of the MIME type multipart/x-mixed-replace. The x means that the MIME type is still experimental; the replace means that each section should replace the previous one rather than be appended to it. Here's how the preceding example looks using multipart/x-mixed-replace:

In a typical Server Push scenario, the CGI program sends the first header and first data block, and then leaves the connection open. Because the browser hasn't seen a terminating sequence yet, it knows to wait around for the next block. When the CGI program is ready, it sends the next block, which the browser dutifully uses to replace the first block. The browser then waits again for more information.

This process can continue indefinitely, which is how the Server Push animations you've seen are accomplished. The individual sections can be any MIME format. Although the example in this chapter uses text/plain for clarity, you may well choose to use image/jpeg instead in your program. The data in the block would then be binary image data. Each block you send would be a frame of the animation.

ServPush is a complete working sample of a Server Push program. You should be able to compile it for any platform. You can find the source, plus a compiled executable for the 32-bit Windows NT/Windows 95 environment, on the CD-ROM accompanying this book.

Compile the code for ServPush (see listing 15.11) and name it servpush.exe. Put the compiled executable in your cgi-bin directory, and test it with <a href="/cgi-bin/servpush.exe?">Test Server Push</a>.

..}

Now that you see how it's done, you should be able to make your own programs. If you want to push graphics instead of text, change the MIME header for the individual sections, and output binary data. (See Chapter 3 "Designing CGI Applications," for details about raw versus cooked mode; you need to tell the operating system to switch the STDOUT output mode to binary if you're going to send binary data.)

Interestingly, you can use Server Push to create animated inline graphics in an otherwise static document. To do so, first create your static document. Include an <img> tag, with the source pointing to a CGI Server Push program instead of a graphics file. For example, say that you've written a Server Push program called photos.exe, which outputs a slide show of your family album. Here's how you can incorporate a dynamic slide show into your HTML:

Near Real-Time HTML

As you saw earlier in this chapter in the section "Methods of Generating Real-Time HTML," not everything needs to be generated on the fly. Documents that are updated regularly and served as static documents are often called near real-time, because the information is fresh but the document itself is static. Often, CGI is used to update the document (rather than create the document in real time). This allows the document to reflect changes immediately, but avoids the overhead of running a CGI program every time a browser fetches the document.

MHonArc (pronounced monarch) is a good example of providing near real-time content. This freeware Perl 5 program (available from http://www.oac.uci.edu/indiv/ehood/mhonarc.doc.html) provides e-mail archival to HTML, with full indexing, thread linking, and support for embedded MIME types. Although the HTML pages themselves are already composed and retrieved normally, they can be updated in the background. You can schedule the MHonArc program to run at regular times, or it can be triggered by the arrival of new mail. Although the code is highly UNIX-centric-and therefore not particularly useful on other platforms-you can examine the source for ideas and techniques.

List maintenance also benefits from near real-time HTML. Lists of favorite links or FTP directory listings don't change very often, but you want them up-to-date at all times. A database with a real-time CGI program to retrieve and format information may be overkill here. A more efficient method is to have a CGI program that updates the list as new information is added, or a scheduled job that updates the list from a central database at regular intervals.

The SFF-NET (http://www.greyware.com/sff/) uses a combination of CGI, SSI, and static documents to provide up-to-the-moment lists without running a CGI program every time. When visitors want to propose a new link for one of the lists on the SFF-NET, they fill out an online form that invokes a standard CGI program. The CGI program validates the information, adds the words not validated yet, and appends it, in proper HTML format, to a text file. Users never see this file directly; instead, when they browse a list of links, they see a static HTML page that uses an SSI include file function. The new links (in the text file) show up in the list right next to the existing links. This provides real-time updating of the overall list without touching the main HTML page. The site administrator then looks at the text file of new links at his leisure, and moves new links from the text file to the HTML file.

Server Performance Considerations

Dynamic HTML can be a lot of fun and can be extraordinarily useful at times. However, it doesn't come without cost.

The first consideration is for caching proxy servers. If your page includes a page count or a random quotation or a Server Push animation, it can't be cached. Defeating caching isn't necessarily an evil-you wouldn't want your up-to-the-second stock market quotes to be cached, for instance-but it can create unnecessary network traffic.

If you visit the Usenet groups regularly, you see a recurring theme of experienced old hackers venting their spleens at newbies who chew up bandwidth for no reasonable purpose. The range of opinion you find goes from calm, rational argumentation, to wild, impassioned screeds. Some go so far as to say that any CGI program is evil and that page-hit counters are the devil's own spawn.

In a book with the sole purpose of teaching you how to write your own CGI scripts, you won't find much support for the extremists. The network is there to be used. Like any limited resource, it should be used wisely rather than wastefully. The problem is in determining what's wise. If you keep your high-traffic pages static, you'll make everyone except the true Internet curmudgeons happy. Of course, if you're a Java developer, all bets are off. The new ways of using the Web are completely incompatible with caching from the start.

The second thing to consider is that CGI programs tax the Web server. For each retrieval that calls a CGI program, the server must prepare environment variables, validate security, launch the script, and pipe the results back to the caller. If a hundred scripts are executing simultaneously, the server may become overburdened. Even if the server has sufficient resources to cope, the overall server throughput will suffer.

Server Push puts more of a strain on the system than almost any other type of dynamic HTML, because the script continues executing (consuming processor cycles and memory) theoretically forever. Just a few of these scripts running at the same time can bring an otherwise capable server to its knees. They have a high level of traffic and resource consumption for relatively little gain.

There are no hard and fast rules. As with any system, you must balance performance against cost, and capacity against throughput.

SE USING CGI

Add To MyInformIT

< Back Contents Next >

Featured Author

Featured Book