Exploring a Virtual Aquarium with a Multi-modal Interface

Tomoko Koda

MIT Media Laboratory
20 Ames Street, E15-412
Cambridge, MA 02139
e-mail : tomoko@media.mit.edu

How would we communicate with virtual characters if we would converse with them as we do with fellow humans? How could children learn from virtual characters which respond to their speech and gesture? Although more and more computers are used in schools, children use traditional input devices such as keyboards and mice. My research theme is an interface for interactive learning environments where children can play and talk with life-like computer characters using multiple modes simultaneously, such as speech and gesture.

Inspired by Alan Kay's Vivarium project (1985) which sought to develop a multi-media environment for children to learn about animal behavior, the virtual aquarium teaches children about marine life. Children can explore an aquarium which is displayed on a large wall display. Computer-generated dolphins respond to children's gesture and speech: they can talk to or use hand motions to instruct the dolphins what they want to explore or what they want dolphins to do. Speech and gesture are captured with a speech recognizer and body and gesture tracking system.

One of the most important ways to involve children with the learning environment is to ensure emotional contact via life-like computer characters. Nicholas Negroponte says that when we have emotional contact with characters, the human-computer interface "... can also be like some other places, those we know and love, those that are familiar, comfortable, warm, and, most importantly, personal." [1] In a real aquarium, children use natural dialog; they try to look closer, track the movement, identify the fish, call their names and mimic the face of marine life. The goal of the virtual aquarium is to allow such interactions with computer-generated characters, and let children learn using natural dialog, by having fun, by playing with virtual characters, and by being emotionally involved. As Marvin Minsky says, "Enjoyment, which has been banished to the realm of entertainment sciences, may be the most powerful influence of all on how each person learns." [2]


[1] Negroponte, Nicholas. In forward to: Richard A. Bolt, The Human Interface. New York: Van Nostraund Reinhold, 1984, p. xiii.
[2] Minsky, Marvin, "Position Paper about AMT's Future," Unpublished Paper MIT, 1986, p.4.

Presented at Lifelike Computer Characters '95 in September 1995.

To see a mini story with virtual aquarium


Tomoko Koda <tomoko@media.mit.edu>
Last Modified: Sun Feb 18 1996