HOW TO CHOOSE HARD DRIVES FOR THE FLYER: Nonlinear does not mean offline: NewTek's Video Flyer is a nonlinear editor (NLE), a computer-based video editing system which records video and audio to hard drives. Unlike offline editing systems that work with reduced-quality video, the Flyer offers broadcast-quality recording and output. You can produce a finished product directly with the Flyer without going through the additional step of an on-line tape edit. That means the Flyer is an online nonlinear editing system. Most online nonlinear editing systems in today's market limit the user's freedom to choose hard drives. Such systems, in order to support broadcast quality, may require specific drives in proprietary RAIDs (Redundant Arrays of Independent Disks) or customized hard drives that cost a great deal more. Not so, the Flyer! Throughout the development process NewTek has tested the Flyer with a wide variety of hard drives. These drives are the same off-the-shelf items that our customers can buy locally, however they must have specific firmware revisions and cache settings to work properly. We'll discuss these needs below. Greater freedom of choice: This approach frees the user to select the drives that best fit his or her needs and build the most cost-effective Flyer system. Flyer owners can weigh their needs and resources in selecting a drive. You can take into account the storage capacity you require, the performance characteristics of available drives, the type of production work you plan for the Flyer, and current drive costs. Some users may need high-performance drives which support demanding program sequences and high-detail video imagery for broadcast or entertainment. Others can use less expensive drives for more modest production needs. There are a number of video-capable drives available and drive technology is advancing very rapidly. The drive market sees a constant flow of exciting new products heralding dramatic increases in performance and capacity. FlyerHDTools: To help Flyer owners select drives, NewTek provides a utility program called FlyerHDTools. FlyerHDTools is included with the Flyer system software, and is also available via the NewTek Technical Support BBS (913- 271-9299) or the InterNet at NewTek's anonymous FTP site (ftp.newtek.com). This utility indicates how well a drive will perform with the Flyer. Combining FlyerHDTools with "real-world" evaluation allows our users and dealers to test various drives and assess them for Flyer use. The NewTek BBS also is a resource for timely hard drive information. Users and dealers upload their FlyerHDTools test result files and evaluations of various drives to the "Drive Data" area of the Flyer Zone (in our Uploads and Downloads area). The results of FlyerHDTools tests are available to other users and dealers who may need to find drives suitable for their particular Flyer applications. Once someone has realized the proper configuration settings for a particular drive to perform well with the Flyer (see below), that information can be shared with others. NewTek recommends that most Flyer end-users seek the assistance of a trusted dealer who has experience with Flyer drive selection and knowledge of the Flyer-related drive performance issues. What does FlyerHDTools do? FlyerHDTools evaluates a number of Flyer-related hard drive performance criteria. This data can be used to make an informed decision about the drives you want to utilize with the Flyer. These criteria include the minimum data transfer rate, the drive's cache parameters, thermal calibration, and error mapping. 1) Minimum data transfer rate Not all portions of a drive can transfer data at the same rate. The inner tracks of a drive cannot deliver information at the same speed that the outer tracks can. Drive manufacturers typically list the "best case" transfer speed that the drive can achieve, so manufacturer's specs are not always reliable source of transfer speed data. FlyerHDTools defines the minimum data transfer rate as the slowest rate at which the drive can read or write data. For Flyer operation it is important to know this "worst case scenario" because most users will probably use a drive to its maximum capability. This means that real-time video data will likely be recorded on both faster and slower parts of the drive. The slowest part of the drive is considered its baseline performance ability. For example, a typical drive we tested delivered a healthy 5.6 megabytes per second (MB/sec) at its outer tracks. On the inner tracks, however, the drive slowed to only 3.9 MB/sec. FlyerHDTools reported the minimum transfer speed for that drive as 3.9 MB/sec. Knowing this, you can determine whether this drive will suit your needs. When the Flyer records in Standard Mode, the drive must have a minimum transfer rate of 3.7 MB/sec. In Flyer HQ5 Mode (High Quality 5 Megabyte), the drive needs a minimum data transfer rate of 4.8 MB/sec. This example drive would suffice for Standard Mode video, but would not be acceptable for HQ5 video. If you required the best video quality in your work, you would want to pass this drive up in favor of one that met the Flyer's HQ5 requirements. 2) Drive Cache Parameters Hard drives have onboard RAM used as temporary data storage areas (caches). Data being written to, or read from, the drive is held here. How the drive uses these caches can be changed for different applications. To do so you alter aspects of the drive's "Mode Sense/Mode Select Pages," or just Mode Pages, which retain this data even after power-down. The cache settings for a drive affect the minimum data transfer rate. Most drive caches are optimized for DOS type programs. The caches retain frequently-accessed information so that data can be retrieved quickly if needed again. This avoids having the drive repeatedly look up the same information again and again, which takes longer than simply retrieving the data from a memory cache. For video, however, this is not good. Video data transfers best from the drive when it passes through the caches as quickly as possible. Factory cache settings, therefore, may not be best for video uses. If FlyerHDTools indicates that a drive performs below the performance level you desire (and believe that it can produce), you can employ PC-based drive utility programs to experiment with different cache settings until the drive operates at higher minimum transfer rates. Once you find these settings, we ask that you post the FlyerHDTools log containing them to the NewTek BBS for others who have the same make, model, and revision hard drive. If you are not thoroughly familiar with and experienced in this technical aspect of drive configuration, you would be best to leave such experimentations to knowledgeable technicians. 3) Thermal Calibration As the temperature inside a hard drive changes, its platters (which actually store the data) will slightly shrink or expand. Temperature-induced changes in a platter's size can shift the position of the data tracksÑconcentric rings where information is storedÑenough to cause the drive to exhibit head-position errors. For example, an attempt to locate track 20 might result in the head being positioned at track 23 instead. To compensate for these changes some hard drives perform a periodic (e.g. every 10 minutes) "thermal calibration," or T-CAL. T-CAL recalibrates the head positioning system to true track locations given the current thermal state of the platters. When it occurs, T-CAL interrupts all other drive activity. Read and write operations stop while the heads recalibrate. This interval is displayed by FlyerHDTools as idle time. Naturally, if you're reading or writing video data, the interruption is a potential problem! The Flyer has its own memory buffer to partly overcome short "idle time" interruptions in drive data. This memory is pre-loaded before you see a clip play so that when a T-CAL occurs, data can continue to flow to the Flyer's VTASC compression/decompression chips. If the T-CAL delay lasts longer than 10 frames (about 1/3 second, or 333 milliseconds), the Flyer runs out of video data and playback may appear to stutter while the buffer redisplays the last frame it received from the drive. In practice, other factors can reduce this built-in safety zone to less than 333 milliseconds (ms). There is a statistical gray area between 200ms and 333ms. When idle time or T-CAL delays are less than 200ms, chances of stuttering are minimal. Some hard drives, called "embedded-servo drives," have special servo tracks on the drive surface. They can always position the heads properly with no need for T-CAL. These drives can have an idle time of essentially zero milliseconds and reduce the possiblity of stutter caused by drive delays to nil. 4) Error map settings A rare but possible drive consideration is platter imperfection. When a drive is made there may be microscopic pits or bumps on some of its surfaces. The manufacturer may opt to replace a badly pitted platter or, if there are relatively few problems, may instead program the drive to to "remap" a bad sector to another location on the platter surface. The record of remapped locations is called an error map. Although error maps allow a drive to function perfectly well for data storage, they can introduce cache delays while the heads jump to the remap location and back in order to supply a steady stream of video data. FlyerHDTools will show such delays as part of the drive's idle time. Depending on the severity of the delay, it may or may not cause a video stutter. For this reason it's a good idea to run FlyerHDTools on the entire drive to uncover potential problems caused by imperfect platter surfaces and sector remap delays. As with Mode Page settings, only skilled technical personnel should attempt to change error map settings. Other Factors that Affect Drive Performance: Other factors can affect the performance of the Flyer. These are listed below. 1) Complex Video Video may be complex due to high detail levels, high noise levelsÑor both. A user working with complex video should use a drive which can support Flyer HQ5 mode. A user making extremely fast-cut Flyer sequences using complex video will want a drive with excellent seek time and data throughput performance in addition to HQ5 capability. Seek time is not considered a primary performance factor because modern high speed hard drives generally have more than acceptable seek times. Seek time could be a problem only in exceptional circumstances, such as a combination of very complex video material being used in very fast cuts of short duration (four-field cuts). Note: You may wonder why seek time (the time a drive takes to shift its heads from one track to another) is not mentioned as a primary performance factor. Virtually all modern drives with video capabilities have acceptable seek times. Seek time could be a problem only in exceptional circumstances, such as very complex video material sequenced with a large number of fast cuts of short duration (i.e., four-field cuts). 2) SCSI Cable Quality Always use high quality SCSI cables that are as short as possible. The best quality SCSI cables are double-shielded. No single SCSI cable should be over 6 feet long, and keep the total length of any SCSI chain under 15 feet. Also, it's wise to check the integrity of the connector itself. The connector (on either the cable and the drive enclosure) can become intermittent over time, or from constant cable swapping. Wiggling the connection can make a bad connection begin working again. If you find this to be the case, you may wish to try another cable (or have the connectors checked by a dealer or service center). 3) SCSI Termination Always terminate the last drive in a SCSI chain, and only the last drive. 4) SCSI ID Numbers Every device connected to a SCSI chain must have a unique ID number (between 0 and 6). If two devices share the same number, one of them will not be seen by the host computer (or errors may be reported, or the system may not even power on). Make sure your drives are numbered differently. 5) Termination Power Source Your drives may have termination power jumpers (these select either the SCSI host controller or the drive as the source for termination power). Set the jumpers so that the drive is the source. 6) Grounding and Ground Loops Ground loops can introduce noise into SCSI cabling. This noise can prevent commands or data from passing back and forth between the Flyer and the drives, which can induce stuttering (the Flyer makes several attempts to retrieve data and is forced to redisplay the last received data until the new data arrives). Ground loops can be caused by the arrangement of data and power cables on external drive cases. Try to keep data and power cables short, and try to plug all external drives into the same set of AC outlets powering the computer system. Which drives support HQ5 mode? The significant factor is a minimum of 4.8 megabytes per second data transfer rate over the entire surface of the drive for both reading and writing. This assumes that idle time, T-CAL delays, etc., are within an acceptable range for Flyer use. But what can do you for a drive that can sustain the high data rate that the HQ5 mode requires for only a portion of its total storage capacity? The Flyer always records clips from the outermost tracks (progressing toward the center). The idea that you could use a fully formatted drive with a fast transfer rate on its outer tracks but a slow transfer rate on its inner tracks is a good one, but partially flawed. In preparation for playback, the Flyer must often "cross-copy" portions of clips to different drives on the system. One or more of these A/V temp files might be written to a slower portion of a drive, causing problems on playback (because that portion of the drive does not sustain the minimum transfer rate). Stutter would likely result. The way to make such a drive reliable for A/B roll playback is to short-stroke the drive. This means that you format only the portion of the drive that can handle the data rate you want. FlyerHDTools can not only tell you what percentage of the drive can sustain a given data rate, but also it can format that amount of the drive for you. Short-stroking a drive is a valid means of assuring that its entire capacity is available for high quality recording and playback. Note: That portion of the drive that cannot be used at high speed will be unavailable if you choose to short-stroke a drive. Only the short-stroked portion can be used by the Flyer. RAID systems: Although there are Flyer users who report success with RAID devices, NewTek neither requires nor recommends them. An advantage of the Flyer as compared to other broadcast-quality NLE systems is that the Flyer has been designed for use with off-the-shelf drives. Our perspective is that hard drive technology is rapidly advancing to meet the needs of users working in desktop video. What are audio drive requirements for Flyer? Audio requirements are different from video requirements. SCSI-2 is preferred (although most recently-made SCSI drives should suffice). We recommend a drive with at least 1 megabyte per second data transfer speed, 20 millisecond (or faster) seek time, and 100 milliseconds or less of idle time, for T-CAL, etc. This figure represents the slowest SCSI drive performance that will yield acceptable results when using multiple audio tracks. Note that FlyerHDTools reports on data transfer rates and idle time, but not seek time. The manufacturer's data label on the drive usually provides this information. Removable media drives, if they meet these requirements, should also work. The termination power jumper (if available) should be set to provide termination power from the drive itself. What tape backup drives will work? For a tape drive to work with the Flyer, it must meet the following requirements: a) Tape units MUST be SCSI-2. Many current tape drives have SCSI-1 or SCSI-2 selectable via a jumper or switch. Check your tape drive documentation and set the drive to SCSI-2. b) The tape drive must use 512-byte data blocks or must be able to be set for that block size. c) The tape drive MUST NOT compress the data. Most tape drives which are capable of compressing data can be configured to do so via a jumper or switch; others may require software selection of this feature. Set the drive so it does NOT compress data. Some drives (e.g., Exabyte) use a custom "command tape" which must be inserted and read by the drive to turn this feature on or off. Command tapes should be used to turn the data compression feature OFF. You may need to obtain command tapes from the drive manufacturer, possibly at nominal cost. A QUICK COURSE IN FLYER THEORY: The following information is provided for Flyer owners who wish to gain a greater understanding of how the Flyer operates and achieve a more thorough awareness of the factors that affect drive performance. What data transfer rate is required in order to achieve D2 quality? NewTek defines D2 quality as a video signal made up of 60 fields-per-second, full overscan (approximately 752 pixels wide by 480 lines high) image data within a digital composite video signal that is 8 bits at 14.3 MHz sampling rate. This describes the digital video signal of a D2 video tape deck. It is precisely the digital signal the Flyer uses internally. Since the Flyer uses a variable compression ratio this set of signal characteristics does not dictate a specific rate for data transfer because Flyer data rate varies in proportion to picture complexity. In previous publications such as the Flyer FAQ we described the various Flyer operating ranges which preserve D2 at full broadcast bandwidth or better. The low-end data rate, which could preserve very simple and clean (noise-free) video is 1.4 MB/sec. The upper requirement for highly complex video recorded at full-fidelity and greater-than-broadcast resolution is 8.0 MB/sec. This is well beyond the transfer rate of hard drives made at the time of this writing. The Flyer can operate within the limits of current drive technology by limiting the signal we record to broadcast resolution. By law, the FCC limits the frequency response of broadcast video to 4.2 MHz. The Flyer, in HQ5 and Standard Modes, limits frequency response to this figure through a programmable low-pass filter. This first compression step saves data and bandwidth that would be otherwise wasted by including detail which must be discarded during broadcasting anyway. Standard Mode is based on an assumed drive transfer rate limited to 3.7 MB/sec. HQ5 assumes a drive capable of about 5 MB/sec. Both will deliver lossless D2 video when they can (when complexity, or noise and detail, is low) and both modes will appear identical when complexity permits a data rate of 3.7 MB/sec or lower. If complexity increases, however, Standard Mode will respond with higher compression ratios to keep the data rate down to 3.7 MB/sec or less. The Flyer's VTASC compression system introduces more noise as its compression ratio increases. The VTASC noise resembles random or ordinary tape noise. At the highest compression ratios caused by maximum video noise or detail, VTASC can also reduce the pixel sampling rate to keep the data rate within the 3.7 MB/sec limit. In the worst situations, this can introduce chroma noise. HQ5 mode also employs variable compression, but the threshold (for the shift to higher compression ratios) is 4.8 MB/sec instead of 3.7 MB/sec. VTASC will introduce less noise, if any, in HQ5 mode even with very complex video. In fact, with low noise or smooth rather than detailed video, there may be no difference between the two modes. Extended Mode also uses variable ratio compression, but always employs reduced pixel sampling to reduce data flow. This results in slightly lower resolution Flyer video and increased chroma noise. "I've recorded various tape formats with great results, but with VHS ..." All compression systems have difficulty compressing noisy or snowy video. Such video may result from low quality tape, bad heads, multi-generation dubs, or low light shooting conditions. There is no way for video encoding systems to differentiate between noise and detail, so a noisy image acts like an image that requires high compression or a high data rate. Any compression system, including VTASC, must increase its compression when it encounters noisy video. This can result in a lack of detail and even more noise. Noisy video does not compress well. In such cases, HQ5 mode will probably give you far better results than Standard Mode. Video shot in low light usually has very high noise levels. Consumer camcorders often have built-in gain boosters for low-light shots. They allow recording in low light, but they introduce noise from the camera's light-sensitive CCD chip(s) and the video amplifier circuits. Professional cameras usually have an option whether or not to boost gain in low-light and can provide some control over introducing noise to a low-light shot. Tape noise can be eliminated entirely by recording directly through the Flyer to hard disk. You'll get best possible video recording and lowest possible data rates if you bypass tape. ---------------------------------------------------- All material written by Tim Jenison and Chuck Baker (with contributions from Dan Wolf and James Hebert). ----------------------------------------------------