Paper for the 1997 Socially Intelligent Agents AAAI Fall Symposium


Socially Intelligent Virtual Petz

Adam Frank, Andrew Stern, Ben Resner

501 2nd St, Suite 400
San Francisco, CA 94107
+1 415 495 0400


We have developed a series of lifelike computer characters called Virtual Petz. These socially intelligent agents live on your PC computer desktop. The Petz are autonomous characters with real-time layered 3D animation and sound. Using a mouse the user moves a hand-shaped cursor to directly touch, pet, and pick up the characters, as well as use toys and objects. Virtual Petz grow up over time and strive to be the user's friends and companions. They have evolving social relationships with the user and each other. To implement these agents we have invented hybrid techniques that draw from improvisational drama, cartoons, classical AI and video games. As of June 1997 over a million copies of virtual Petz have been sold around the world.


Traditional digital interactive entertainment has centered around characters for years. Most entertainment products contain computer characters that represent the user, and the user puppeteers the character in its environment. In Petz products, the characters are completely autonomous, the user is represented as a hand cursor, and the pets perceive the user as just another socially intelligent autonomous character. In this way the Petz and the user are on essentially equal footing.

The virtual Petz concept has been a good way for us to explore the complicated issues in creating interactive, real-time autonomous characters. Choosing dogs and cats as our agents helps us in a few ways. People understand what pets are and essentially know how to interact with them. This is critical for producing believable characters. Pets are also a good choice because they are expected to behave in ways that we can successfully implement. They have relatively simple gestures, language and cognitive abilities. Also, animated dogs and cats have well established cartoon archetypes. We base the look and behavior of the Petz on these archetypes.

The virtual Petz experience is non-goal oriented. We allow users to explore the characters and their toys in any order they like. This freedom allows users to socialize with the Petz in their own way and at their own pace. This also encourages users to come up with their own interpretation of their pet's feelings and thoughts. Often the scope of their interpretations exceeds what we originally planned for.

Developing Relationships with Virtual Petz

The goal of the Petz is to build an intimate relationship with user. Therefore the pet's primary motivation is to receive attention and affection. They feed off of this interaction. Without it they become lethargic, depressed, and if ignored long enough, they will run away.

The most direct way the user shows affection is through petting. Using the left mouse button users can pet, scratch and stroke. The pet immediately reacts in a variety of ways depending on what spot on its body is being petted, how fast, and how the pet feels at the time. We call this direct interaction. The fact that you can touch the character is a very effective way of building an intimate relationship. It also goes a long way in creating an illusion of life.

We've tried to make the Petz have equal footing in their relationship with the user. The toys and objects in the pet's environment have direct object-like interaction for both the user and the Petz. Petz have full access to the toy shelf, and if they really want something, they will just go and get it themselves. This helps express the unpredictability and autonomous nature of the Petz. It also requires users to share control of the environment with the Petz. The act of sharing the space is a further strengthens the owner-pet relationship.

By picking up and using a toy, the user can initiate play. For example, throwing a ball may initiate a game of fetch, or holding a tugtoy in front of a pet may initiate a game of tug-of-war. Similarly a pet can get its own toy and bring it to the user to initiate play. This cooperative decision-making helps build the relationship.

We have created a variety of personalities - playful terriers, grumpy bulldogs, hyper Chihuahuas, finicky Siamese cats, lazy Persian cats, aggressive hunter cats, timid scaredy cats and so on. Additionally each pet has its own individual likes and dislikes, spots and body coloration, and personality quirks. Users get to play with a pet and see if they like them before deciding to adopt. Once adopted, the user gives them a name. This individual variation allows the user to develop a unique relationship with a particular pet. Every owner-pet relationship is different.

Communicating with Virtual Petz

Petz communicate with the user in a variety of ways. Emotion can be expressed through different facial expressions (eyebrows, mouth, ears), styles of movement and body language (sad walks, happy trots, various postures, a variety of tail motions), and sounds (excited playful barks and meows, sad whines and whimpers, yelps of pain, etc). They get fat when they eat too much and get skinny when hungry.

Petz communicate their intentions and desires through their actions. We have created a broad base of behavior for the Petz -- they can eat, sleep, play, attack, groom, hiss, explore and so on. When a pet wants attention they may start barking or meowing. If hungry they may go searching for food, hunting mice, or start begging. When they are upset or scared they may run and hide, act aggressively, or cower and shiver.

It is important to note that the only way the user can understand what the Petz are feeling is to interpret their actions and physical cues, in the same way an audience interprets an actor's performance. We do not display bar graphs or text messages describing the pet's internal variables, biorhythms or emotional state. By forcing a natural interpretation of the pet's behavior, we don't break the illusion of a relationship with something alive.

The user can communicate to their pet in a variety of ways. In addition to petting users can gesture to their Petz. By double-clicking in an empty space on the screen the hand cursor animates a "come here" motion and plays a whistling sound. Petz tend to stop what they're doing and come to the user's cursor.

Users can train their Petz to do tricks for treats. When holding a treat, users can make the hand cursor do directional hand motions (up, down, left, right) to which a pet might jump, sit, lie down, rollover, and so on. If you shake the treat over a toy, a pet may attempt a trick using that toy, such as balancing on a ball or fetching.

In general users can reward or discipline any behavior. Methods of rewarding include a giving treats, feeding, petting, or presenting its favorite toy. Discipline is achieved by squirting a pet with a water spray bottle. (This is the SPCA-approved method of discipline.) In this way users can modify a petís overall behavior through positive and negative reinforcement.

The personalities of each pet can change under extreme circumstances. For instance, if any pet is underfed, it will begin hunting and acting aggressively. If a pet is overfed, it will become fat and lazy. Depending on how the user treats their pet, each pet's personality can slide along a broad spectrum of personalities.

Relationships between Virtual Personalities

In our newest product, Petz II, released in October 1997, multiple characters can interact with the user and each other at the same time. When two Petz first meet each other, typically as puppies or kittens, they cautiously investigate each other. As they wrestle and play they form relationships and attitudes towards one another. Some will become best friends and will often groom each other, playfully nip at each other, sleep side by side, bring each other treats, and so on. Other pets may become enemies and will fight, chase and terrorize each other throughout their lives. A dominant dog might steal food and toys from a submissive dog.

Adult Petz tend to nurture and protect younger Petz. They will comfort them if they get scared and carry them to a pillow to sleep. Younger Petz will follow the adults around and attempt to copy their behavior. However once adolescent, a pet will form an adult relationship with the other adults.

Most Dogz and Catz tend to dislike each other. Typically they will growl, bark, hiss, fight and chase each other. However if they are introduced to each another at a young age and grow up together, they may become friends.

The interplay between the variety of personalities gives rise to many dramatic situations. For example, a tired old dog won't be able to go to sleep when a young playful kitten is bouncing around him. The dog may try to take a nap after distracting the kitten with a bowl of food, but the kitten's short attention span defeats this strategy.

Catz are always on the lookout for mice. If the user takes out the cheese and puts it in front of the mouse hole, the mouse may sneak out for a nibble. The cat will immediately begin stalking it for an attack. If the user then brings out a dog, a three-way chase ensues. This usually ends with the dog chasing the cat away and the mouse getting a free meal.

Even a loving relationship between two buddies can be upset when the user pays more attention to one pet than the other. The jealous pet will become angry and upset and may even act aggressively to the user or the other pet. However the two Petz will eventually make up and return to their friendly relationship.

Should Virtual Petz Die?

Petz begin as puppies or kittens and age over time. Users witness their pet develop and can affect what kinds of personalities they turn out to have. However at a certain point the Petz stop aging and do not die.

This has been an issue which our design team has fought over back and forth. Some argued that if the pet eventually dies it makes the lifecycle more important. Immortality may devalue the experience by making the user's time with their pet less precious.

On our website we recently conducted an informal poll asking users if they thought virtual Petz should die. 65% of the responses said they should not die. 30% said they would like to be able choose if their Petz should die. Only 5% said they wanted their Petz to eventually die. After long discussions with our marketing department, it was decided that the disadvantages of death outweighed the advantages.

Instead of death we decided that the only way a user can lose their pet is if they severely mistreat them. After a few days of ignoring a pet, it will actively begin whining for attention. If the user does nothing to remedy this situation, the pet begins acting more and more neglected, even doing "naughty" behaviors to get attention. Eventually the pet will simply run away and not be available to the user anymore. Because the user's actions have consequences, we have added some value to the relationship.

Implementation Principles

To create these socially intelligent agents on a personal computer, we had to develop new display techniques. Our high-degree of interactivity and lifelikeness requires the use of real-time 3D animation. Pre-rendered animation, however beautiful it may be, cannot be as responsive, dynamic, or as varied as a we require. But to deliver this real-time animation on a PC, drastic optimizations on the display level are required. Our proprietary method of achieving this is unique. We chose to err on the side of frame rate versus a photo-realistic image that moves poorly.

The Petz products use a network of basic 3D body animation which has a high resolution of branch points. For example, from any basic pose such as standing or sitting, the pet can transition to any other pose. As the pet's general body animation moves across this network, additional animation is layered on top. This allows the pet to look in all directions no matter what its body doing. Also we layer breathing, blinking, meowing, head shaking and even character specific posture. In fact the look of the each breed and their ages are layers. The subsequent number of permutations is enormous and therefore passes a critical threshold in human perception. This method greatly enhances our ability to create an illusion of life.

Using this display technology we have developed many behavior mechanisms that operate simultaneously. There are small mechanisms to control blinking and breathing and a variety of cognitive mechanisms to keep track of multiple goals, different plans to execute each goal, multiple emotions towards other characters and objects, attitudes towards other characters and objects, and personality-specific quirks.

In general to achieve a real, unpredictable, non-repetitive quality in these mechanisms, we use constrained randomness. For example, there are rules for each individual pet that determine which toys it most likes to play with, but there is some randomness in that decision. This results in the possibility that a pet might play with any toy, not just the ones it likes the most. A lot of development time is spent tweaking the constraints on the randomness to achieve a good balance between consistent personality and unpredictability. We are focusing on the user's perception of the character and trying not to over-engineer the artificial intelligence.

When making consumer software there are major constraints on both design and implementation. Our project has to be used by grandmothers and four-year olds and run on very slow, old computers, on multiple operating systems, and still perform as a seamless, real-time experience. These constraints force us to design with a bottom-up approach - always paying the most attention to what the final user will perceive. We focus our energies on what we can show and not waste valuable time implementing an elaborate system that we cannot express onscreen.


The Petz products are an experiment that turned out to be relatively successful. The product is equally popular among girls and boys, kids and adults. This is unusual in consumer entertainment software. In the program, users can take a bitmap "photo" of their Petz which they can save to disk. We encourage users to e-mail their photos to us, which we post on our website. We've had a tremendous response, receiving thousands of photos, some with personal stories attached and some with elaborately painted scenes of their Petz on Mars, or dressed up in tuxedos, or windsurfing, or flying World War II planes. We receive many letters from these kids describing their relationships with their individual Petz. Some kids we know get up especially early every day to feed them. Some of the stories describe feelings and complex behavior that we never put in the product. It seems that through combining a direct interaction interface, strong and recognizable personalities, and some fundamental social intelligence, an intimate owner-pet relationship can arise.

We've found that developing socially intelligent agents requires people with interdisciplinary skills from both technology and the arts. Until recently, building lifelike artificially intelligent characters has been treated as a computer science problem. Character development, in any medium, is fundamentally an artistic process which combines elements of drama, storytelling and expressive action. Lifelike computer characters are ultimately perceived through the filter of human interpretation. No computer character can be perceived as truly alive and intelligent unless its builders always keep in mind what the final user will perceive.

Of course, there is a tremendous amount to explore in this area and we realize there is much more work to be done.


We thank Rob Fulop, John Scull, Jonathan Shambroom, Alan Harrington, Ted Barnett, Jeremy Cantor, and the rest of the PF. Magic Petz team.