Next: Previous Examples of Up: Synthetic Movies Previous: List of Figures
Next: Previous Examples of
Up: Synthetic Movies
Previous: List of Figures
The field of communication has always availed itself of the latest technology to enhance the message quality. This can be traced from the invention of the printing press, through the early use of electronics by the telegraph industry, to the development of film and later, video, for the transmission of visual information. In the last few years, the ready availability and low cost of digital electronics has sparked a wide interest in digital transmission of visual information. The flexibility of the digital medium, shown by its ability to integrate text, audio, and video, permits many new methods of obtaining and manipulating information.
The large amount of data required to represent an image sequence restricted research into digital video until recent years.
A typical motion image sequence, such as standard broadcast TV, requires 8.2 MBytes/sec of data per second
.
Conventional image transmission and compression techniques utilize statistical properties of the image sequence and psychophysical properties of the human visual system to reduce the amount of data associated with the image sequence [Netravali89] [Schreiber86] [Pratt78].
Use of these techniques allows image sequence transmission and display using a state of the art personal computer [Watlington87][Watlington88a].
Conventional compression techniques, although making digital video possible, do not utilize the semantic content of the image sequence. This thesis proposes an alternative method of image sequence transmission: a synthetic movie, where the sequence is synthesized at the receiver from a description of the sequence's contents. An analogy may be drawn to digital text transmission, where two methods are commonly used: facsimile and ASCII. Facsimile encodes the text using a statistical model of the text image. ASCII provides a much more efficient coding by transmitting only the semantic content of the text. The text is reproduced at the receiver by using a local description of the characters in the text (the font) and the transmitted ASCII information. Synthetic movies attempt to separate the semantic content of a sequence from the description of the objects in the scene, allowing separate transmittal of the two components.
Synthetic movies are motion image sequences that are constructed as they are being viewed. They are composed at the receiver, usually under the interactive control of the user. The content of the components and the complexity of the reconstruction vary. Examples of synthetic movies range from videodisc training tools, to simple molecular modeling tools, interactive flight simulators and video games.
Synthetic movies may be categorized by the type of synthesis allowed. Intraframe synthesis permits the actual content of the images being displayed to be manipulated, whereas interframe synthesis is restricted to controlling the temporal evolution of the sequence.
Interframe synthesis has seen much attention over the past decade, due to the introduction of a relatively cheap random-access image frame store in the form of the videodisc. In interframe synthesis, the temporal content of a sequence is synthesized by selecting one frame of several possible frames of video for display. The manipulability of the interframe synthetic movie is rather limited, as every frame conceivably required must be present in the object description. In some videodisc applications this restriction is partially circumvented by the addition of a simple computer graphic overlay, such as text or still images.
Intraframe synthesis refers to synthesizing the actual content of each frame in the motion sequence. This synthesis has been addressed by the computer graphics community, although due to the complexity normally associated with realistic intraframe synthesis, rarely have the movies been generated in real-time (ie. as they are viewed ).
Synthetic movies are best suited to transmitting certain forms of visual information. This is due to the fact that the image must be reconstructed at the receiver. When the sequence being transmitted is very detailed, or is not being manipulated, synthetic movies are not the medium of choice. An example of this is artistic works, which are usually meant to be viewed in a single, unidirectional sequence. If the interactive transmittal of visual information is desired, however, synthetic movies are ideal.
Synthetic movies, because they are constructed at the receiver, are inherently manipulable. One component of the sequence may be changed, or an alternative one used. In the case of interframe synthesis, the sequence may branch among a wide possibility of cases. In intraframe synthesis, an object/actor on the screen may mimic the sensed motions of a real person[Ginsberg83][Maxwell83]. Implementation considerations usually restrict the manipulability, but it remains as one of the main reasons for developing synthetic movies.
The manipulability of the synthetic movie is not necessary for some applications. There are, however, many uses of image sequences where interactivity is very desirable, or even required. Here are three example uses for synthetic movies:
Electronic games have become increasingly popular as the cost of the required electronics has plummeted. Games that produce a video display, in particular, have been developed and produced in astounding numbers. The video display generated by one of these electronic games must be viewed as a synthetic movie, because it is an image sequence generated in real time to reflect some internal environment. Object descriptions are usually stored locally in read-only memory, and the movie description is generated by the game's state machine response to user input.
The realism of even the best of these games is rather limited, due to the simplicity of the object descriptions used, but this is changing.
Some commercial video games are already using traditional computer graphic techniques to generate the display.
The computer processors used in video games are becoming more and more powerful.
Synthetic movies may be used for very low bandwidth information transmittal. The ``message'' of the image sequence is contained in the movie description, which is always transmitted. At the receiver, the movie description is reconstructed using locally stored object descriptions. Once object descriptions compact enough to be transmitted can be reconstructed in real-time, the object descriptions may be transmitted along with the movie description.
The displayed image sequenced will be distorted due to several reasons. One reason is the differences existing between the original object description and the one used for reconstruction. Another reason is the omission of parts of the original image sequence not incorporated into the movie description.
One of the possible future uses of synthetic movies is separating the artistic content of a movie from the limitations imposed by physical filming.
A motion picture maker could be freed from the requirement of having the exact lighting required, or the camera
in proper location, during the filming.
In the editing phase, the movie would be synthesized from the database generated from the filming using any combination of lighting, lens characteristics, and camera position desired.
New objects could be added or existing objects removed from the scene.
When the desired effect is produced, the movie description for that scene is stored for inclusion into the movie.
The movie distribution could entail transmitting both the movie description and the object descriptions to the receiver, using a digital transmission medium. Alternatively, the movie could be rendered once and recorded for publication and distribution along more conventional channels.
The earliest computers had no user interface. Batch mode processing limited interaction with the computer to punching programs onto paper cards or tape, then loading them into the computer. Output usually consisted of printing to a line printer. When the teletype appeared, and simple operating systems were designed to use them; it was a great improvement. Teletypes were later replaced by terminals using cathode ray tubes as their display, although the high cost of memory limited the versatility of these displays to that of an electronic teletype ( the DEC VT-100, for example ) Due to the relatively low cost of memory today, the modern personal computer/workstation is usually equipped with a bit-mapped display, usually capable of displaying multiple colors. The user interface, however, usually still consists of little more than a window system which allows the user the convenience of having multiple virtual ``teletypes''.
An exception to this are the object oriented user interfaces descending from the original Smalltalk project at Xerox PARC[Kay77][XEROX Star], such as the Apple Macintosh Desktop. These systems use the graphic capabilities of modern bit-mapped displays to provide the user with a graphically oriented operating system. Files and a hiearchical file system are described using images of applications, documents and filing folders. Common system services (such as the printer, or the routine to destroy a file) are also given icons. If, in addition to being capable of displaying still images, the computer system is capable of supporting moving pictures, how may they be integrated into the user interface? Perhaps the answer may be found by discussing the applications of still images at the user interface.
Still images are commonly used to represent a virtual object which conveys information between the user and the computer. Control bars, buttons, dials, etc., are simple examples of this. Icons are small still images used to represent different elements and functions of the system, such as files and trash cans.
The simplest application of motion sequences in the user interface is an extension of the still image applications. Using interframe synthesis, icons that are themselves image sequences can be made. Although these are interesting, they are limited in the information that can be conveyed.
More interesting applications of motion sequences in the user interface can be found when the content of the image sequence can convey more than one variable of visual information. Intraframe synthesis allows the contents of the frames to reflect the state of many different variables, in a form that is easily interpreted by human users.
