Artificial Artist at Work

A computational model of the artist's eye

Building an artificial artist touches upon problems in many fields. First of all, it needs to see and understand enough about animals to decide what is striking about a subject. Then it must be able to conceive of a design, and have some approximation of the "inner eye" with which we imagine and refine our own artistic ideas. The faculties named in italics are the concerns of computer vision, artificial intelligence, automated design, and computational neuroscience, respectively. None of these fields offer whole solutions to any of these problems, but by using their methods in unorthodox combination, plus a few new ideas, we can map out an account of artistic creation so thoroughly that even a computer can do it.

General strategy

The system starts with a picture of an animal and finishes with a three-dimensional mobile (virtual or milled from metal). The transformation occurs in four steps: analyze, represent, design, and refine.
  1. Analyze the image and identify salient parts.
  2. Build a schematic "mental" image that represents the salient regions.
  3. Make an initial design by projecting this representation into a new medium (e.g., by making a crude mobile)
  4. Refine the result to reinforce resemblances to the original image and formal aesthetic qualities (e.g., curve rhythms)
The first two steps produce an abstracted representation of the animal that omits most detail but maps out the parts of the animal that our eyes might normally dwell on. (This implies a psychological account of what is interesting about animals, which we will sketch below.) The third step sketches out how this map will fit into the constraints of the new medium (e.g., that it make a balanced mobile). The fourth step exploits the remaining degrees of freedom in the medium (e.g., the shaping of the wires) to evoke some of the visual qualities of the original image, and to reinforce aesthetic qualities in the final sculpture. In its barest outlines, this process is not unlike how an artist is taught to draw human figures: study the anatomy, map out principal body masses, make a rough sketch, and fill in details.

Understanding anatomy

First the artist must read the body of the animal, looking for visual qualities that make it dramatic and distinctive. There are many reasons why an animal could look interesting. It might have unusual parts, like an elephant's trunk, or strike an unusual pose, like a bucking bull. An animal might be especially strong and well proportioned (this inspired much horse-portraiture in 18th-century England), or pitifully underdeveloped (e.g., Picasso and Giacommetti's farm animals). These qualities stand out because they contrast with our normative knowledge of how four-legged animals are constructed and how they move.

What it knows

Our visual knowledge of animals is quite sophisticated. For the artificial artist, it turns out that rudiments of this knowledge suffice to identify especially interesting aspects of an animals body. For example, To find these things the system knows that Using this general anatomical knowledge, the system looks tries to explain the animal's body--how it stands, how it moves, where the impetus for motion comes from, where the centers of power and tension are.

What it sees

What kinds of computable visual evidence will support an anatomical analysis? Let us begin with muscles. Tensed muscles ripple the surface of the skin. And ripples are a visual cue that machines can detect:
original
image

The system begins by extracting slow-moving contrast gradients from the image:

contrasts

Then the animal's boundaries are subtracted off and the remains are midpass filtered for illuminant-based surface ripples:

thresholded gradients

As such, this ripple-map is not particularly meaningful. Some ripples indicate tensed muscles, some are from bones, some are from hair, and some are from surface features. To make this map useful, some knowledge is needed: Muscle groups are concentrated around joints that carry a lot of load. Consequently, ripples that are spatially correlated with large joints are probably musclar.

We can't X-ray the animal for joints, but we can make a good guess from its shape. We take the silhouette of the animal, smooth its boundaries, and "skeletonize" the resulting shape by burning away the outermost pixels:

skeleton

This is of course a rather poor guess at the animal's skeleton, but note that the major inflections and branching points are still close to the principal joints of the real body. To find the tensed muscle groups, we cluster the ripples at these inflection points:

ripple clusters

These are grouped and fitted into closed contours, mapping out (in this case) the three main muscle groups of the animal: the hind, the shoulder, and the jaw:

muscle masses

Watching these muscle groups will tell you a lot about a horse. They are the "motors" of the animal. They explain how it moves.

Another part of that explanation must be what the motors move--the limbs that they propel. These include the load-bearing masses--appendages which connect the animal to the ground--and gestural masses--appendages that the animal uses to "interface" with the world. In an encounter with a real animal, these would bear watching, so we add them to the map. The result is a salience map of interesting parts of the horse:

all masses

This, in the its mind's eye, is the horse reduced to its basic regions of interest. We'll call this crude but informative abstraction of the animal the "conceptual sketch."

From the conceptual sketch to art

The next step is to take the conceptual sketch into a new medium, where it will be transformed into an aesthetic artifact. Our target medium, kinetic mobiles, is especially challenging: Not only must the result succeed as a depiction and as an aesthetic object, it must also be compatible with a host of very stringent mechanical constraints.

However, mobiles have the advantage that we can use the salience map almost directly as is. The whole purpose of the conceptual sketch is to direct your attention to the parts of a subject that reveal something deeper about its nature; surely we expect the same from art. Different arts have different ways of being emphatic; in mobiles it is the masses that command one's attention. Thus the system uses the shapes in the conceptual map as the masses of the mobile.

First, they are made a litte more attractive. Adjacent masses are fused and the contours smoothed:

fused masses

In this case (but not usually), some detail is added, producing the final template for the mobile:

mobile template

The template specifies a set of masses and where they should float in space. The system's task is to connect them, using wires, hooks, and loops to make a working mobile. Each wire must obey three kinds of constraints:

The system solves the mechanical problem first: routing wires, finding balance points, preventing collisions. The result is a crude but mechanically valid mobile. Then it causes this mobile to evolve, applying forces that cultivate its representational and aesthetic qualities, concluding with a working piece of art.

Routing the wires

A wire is basically a balance beam: on one end a hook, on the other a weight, and somewhere in the middle a loop from which it hangs. The loop must be exactly at the balance point; the hook must be exactly at the balance point of the subassembly it suspends. The wire itself has mass, so any change to its shape or length moves its balance point, plus those of all wires above it. If the balance point is too low the wire will flip over. If it is too high, there may not be room for other wires. Sometimes the right place for the balance point is occupied by a mass, requiring route-arounds. Finally, the wire must not cut off the connection path of any other wire and must not come too close to any plate (lest the mobile strike itself when pieces are swaying). Many of these problems crop up with irregular and closely-spaced masses:

tricky wiring

A routing policy ensures that the mobile is mechanically valid:

  1. Pick a plate, preferably near the bottom, to be the first balance point.
  2. To hang each plate,
    1. Run a wire from the plate to the last balance point on a path that goes over all hung plates and under all unhung plates.
    2. Calculate the center of mass of the entire hung system.
    3. If the wire lies below the center of mass, pull it up and around any obstructing masses.
    4. Loop the wire at a point at or above the center of mass
A unique property of the artificial artist is that it solves the routing problem, (and indeed all design problems) in imagery--using "active pictures" rather than traditional computer-based problem-solving methods.

Coupled potential fields

Coupled potential fields are shaped force-fields that interact by pushing, pulling, and deforming each other. The are useful for solving spatial planning problems. To design a chair surface that fits your body, make the surface of your skin an attracting field and make blanket-shaped field that floats and deforms to cover your back and thighs. Is the chair too bumpy? Put a field inside the blanket field that resists sharp bends. To find your way out of a maze, turn the maze walls into a repelling field, and put a 1-dimensional field inside of it. Cause the one-dimensional field to grow, so that it stretches down the alleys of the maze. Where ever it goes, make it increase the repelling force from the maze field, especially where it moves slowly. If it gets caught in a dead end, the repulsion eventually forces it out, until it has explored every avenue. Although there is no "central intelligence" following a "strategy" to solve the maze, this double-coupling is like leaving breadcrumbs to remind you where the dead ends are. Singly-coupled potential fields are related to mesh-methods in engineering and have a long history in computer vision as snakes and deformable solids. The artificial artist uses a variety of multiply-coupled fields (on analogy to the linked maps of the visual cortex in the brain) to solve its design problems. For example, to route connecting wires and make an initial mobile design, the system uses a coupling much like the maze-solver to implement the routing policy above.

Routing wires

routing problem

Here is an example of the router in action. In this diagram of a partly-wired mobile, the two arrows show the points that need to be connected by a new wire. The wire will start at the mass on the right, snake under some masses and over others, and arrive at the subassembly balance point on the left. This is accomplished by combining a number of fields, each of which enforces a different constraint on the wire. In the images below, each field is represented as a gradient image, where light areas repel and dark areas attract. To start with, one field establishes a target for the wire:

a = target
To make sure that the wire stays away from masses, a repelling field is constructed by blurring an image of the masses:
b = masses
To implement the over-under rule, all unhung masses are "smeared" up to block anything passing over them, and all hung masses are "smeared" down
c = over-under
Summed, these form the routing field:
a + b + c = routing forces
An embedded 1D field follows the gradient toward the target, snaking around obstacles and finding a path that will guarantee a working mobile:
routing solution

Shaping the wires

The system takes the rough-wired mobile and places it in a set of potential fields--images whose intensity gradients act as forces on the wires. One field pushes the wires away from the plates; another pulls them toward strong contours in the original animal.

horse-template - horse-contours = horse-field

Other fields pull the hooks and loops toward their proper balance points. One-dimensional fields inside the wires keep their contours smooth and piecewise logarithmic. Other one-dimensional fields looped around each mass compute attractive attachment points for the wires. All the fields are coupled and the wires are made to float in the forces they generate. The wires bend and expand like sails in the wind or iron filings in a magnetic field. In a few hundred steps the mobile takes shape:

horse-evolve1
horse-evolve2
horse-evolve3
horse-evolve4
Et voila! Here is a variant rendered in 3D (many different wirings are possible):
horse-mobile

Not only do the wires form a balanced mobile, but they recover many of the contours lost in the visual analysis, in this case the mane and the undulation of the back and tail. The process is quite robust--so much so that if you reach in to push a wire around the system will instantly revise the mobile to gracefully incorporate your curve.

Beauty?

Is the result beautiful? That we'll leave up to you. We can offer a few reasons why it is visual interesting:

More about making mobiles:


Last revised 10jul96

Matthew Brand / MIT Media Lab / brand@media.mit.edu