Composer Morton Subotnick said in 1989 in an interview with Tod Machover "It is important to have new instrumentation which can be more expressive" and "We need to address electronic media and understand that we must create art using all media, especially new interactive media." [SM96]
The thought of creating new musical instruments is not new: it has been around since the beginning of the century. In 1916, composer Edgard Varese said "Our musical alphabet must be enriched... We also need new instruments very badly... In my own works I have always felt the need for new medium of expression." [Cha97:p59] Then again, in 1939, he said "I need an entirely new medium of expression: a sound producing machine." [Cha97:p258]
Many others have tried to develop solutions for the needs mentioned. One of the first was Leon Theremin, who, in 1920, invented the theremin [Cha97:p9][Wax95], an instrument that gave the performer, for the first time, a "free-air" playing environment. It is also not based on any previous instrument. The Russian leader Lenin was introduced to the theremin and started playing it a few hours later. One hand controls the volume and the other hand controls the pitch. One can play a whole range of notes either by sliding or jumping from one to the other using well coordinated movements of both hands. The world's best theremin soloist, Clara Rockmore, who died recently in May 1998 at the age of 87, was able to subtly control the sound by using finger movements. This kind of playing accuracy requires an intimate understanding of the behavior of the instrument and years of playing experience. Everyone can play the theremin but not to an extent comparable to Clara's intricacy. Part of the instrument's difficulty comes from the lack of physical or tactile feedback. Sound is the only feedback. The player must listen very carefully and respond very quickly to what he/she hears.
Since the theremin, there have been many other new instruments. Researchers at STEIM (Stichting Elektro-Instrumentale Muzeik) [Kre90][Rya91][WRC93], an arts research institution in Amsterdam, have made many interfaces for music. These include the Sweatstick [WRC93], the Hands [Bon98][Kre90][Rya91][WRC93], and the Web [Bon98][Rya91][WRC93], which are geared towards bringing out expression and controlling timbre in performances.
In electronic music, this practice of having a live performance is rare even though it is a very traditional way of using musical instruments. The STEIM researchers have put emphasis on creating electronic musical instruments that can be used in live performance. This not only gives the audience the opportunity to see the instruments, it also allows the audience to understand and enjoy the music through the movements and gestures of the performers on stage.
Developed by Ray Edgar, the Sweatstick is an alternative MIDI controller. It is a one-meter aluminum rod, articulated in the middle, with a sliding keypad for each hand. The performance of the instrument resembles that of a Chinese martial art performance with fighting sticks. Like the theremin, the Sweatstick is a "free-air" controller except that there is a physical object to maneuver which gives the player more tactile feedback. I would imagine that Chinese martial art masters could actually transform their acts into musical outputs using the Sweatstick. Since there is tactile feedback, it might be interesting to do a duo for Sweatsticks where additional sensors were added so that "stick contacts" could create another musical or sound output.
The Hands, created by Michel Waisvisz in 1984, is essentially a pair of miniature 16-key keyboards worn on the performer's hands. Together with many different combinations of sensors, the Hands can capture movements of the hands, the fingers and the arm. In 1989, Bert Bongers worked together with Waisvisz and made many improvements to the first version and called it Hands II [Bon98]. Later, it evolved into another controller called the MIDI-Conductor [Bon98]. It is an ultrasound-based, spatial hand-held instrument created in active collaborations among STEIM, the Royal Conservatory of Music in Den Haag, Netherlands and the Institute of Sonology. The ultrasound is used to measure the distance between both hands. There is also a pressure sensor, two tilt sensors, a movement sensor, and a number of switches. The Hands and its successors are all very "free-air" instruments but are very much less tactile than the Sweatstick. This is mainly because the Hands is in a handle-like configuration. It also has a lot more sensors and controls, making it a more difficult instrument to play than the theremin. The greatest drawback of the Hands are the wires that connect them to the digitizer. Very often, handheld or portable interfaces with wires obstruct the performer's mobility. A wireless version of the Hands would definitely provide much better mobility for the performer. A problem with the MIDI-Conductor is that, although ultrasound can be used to measure distance effectively, it is highly directional. With the hands of the performers holding the interface, unless the movement and orientation of the MIDI-Conductor is restricted, the ultrasound measurement will very often be cut off and become useless. If there were two MIDI-Conductors being used at the same time, the deflection of the ultrasound might trigger a false measurement of distance between the two hands of one performer. On the other hand, one may be able to make use of this "crosstalk" to create more interactive MIDI-Conductors for multiple performers.
The Web is a musical instrument designed to control the timbre in multi-instrumental set ups on stage with single finger movements of a solo performer. Also created by Michel Waisvisz, it has been extended to a very large scale installation called Soundnet [Bon98] where one or more performers climb up and move about the various cables or "threads" like "spiders." The idea behind Soundnet was to scale the Web to a size proportional to a human "spider." The weights, tensions and even pulls on the cables are all captured and used for the performance. The instrument can be set up with the web plane either in the vertical or horizontal position. The Web is a highly tactile instrument that requires a lot of energy and concentration as one needs to sustain his own body weight and balance. Like rock climbing, one has to be cautious while playing. The spacing of the cables also makes it suitable for taller players. It would be an interesting experiment to actually push a huge ball on the Web when it is in a horizontal position, with all the cable obstacles. One effect that the ball will have is to create oscillations as it attempts to find an equilibrium. Another interesting and more challenging experiment would be to put the Web, either in its original layout, a modified tree-like layout or other layout, in a zoo with monkeys and other tree-climbing creatures. These creatures have far better climbing and swinging skills than human performers and they might produce very interesting results even with the original mapping of the Web. This could truly become a "Nature Symphony" for the zoo. Not only would this make music, it would certainly make going to the zoo more fun and educational.
In 1991, Neil Gershenfeld, Joe Chung and Tod Machover designed the Hypercello [Mac92][Par97] where, in addition to the audio signals from each string and the finger positions on the fingerboard, the bow position and placement were measured using capacitive sensing methods which are much less sensitive to background noise than most optical or other techniques. Tod Machover's Hyperinstrument [Mac92][Par97] composition Begin Again Again... premiered at Tanglewood, where cellist Yo-Yo Ma played the Hypercello. Since then, there have been more than a dozen performances worldwide. The Hypercello is a very good model for controlling other instruments. Unlike many new musical instruments, it is actually a controller built into a traditional acoustic musical instrument, which makes it more familiar to the cellist. The downsides of the Hypercello are probably the amount of equipment involved, the time it takes to be set up and the need to strap up one of the hands of the cellist for other measurements.
Recently, Tod Machover et al. created the Sensor Chair [BOChair94] [Par97] in October 1994 for Penn & Teller, who are professional magicians. The Sensor Chair allows performers to select different sounds to be played by just pointing their fingers or waving their hands in front of an "invisible map." The player sits on a conductive mesh on the chair and is thereby coupled to an electric field transmitter. Four electric field sensors in a square formation in front of the player sense the strength of the field emitted from the player's hand, allowing software to track the exact position. The only drawback of this instrument is that it can only sense one point in the "invisible map." If the player puts both his hands in, it will average the two to form just one position. Through clever software tracking, the strengths of the four sensor outputs are mapped to electronic instrumentation. There are also two pairs of foot switches used for mode changing and triggering. Like the theremin, the Sensor Chair needs calibration for each and every performer. It is also a "free-air" instrument where there is no physical object to hold on to when playing. Unlike the theremin, the Sensor Chair has a very high-level mapping with pre-composed pieces being triggered at different places and times, so it can be easily played by non-musicians, which was clearly demonstrated by Penn & Teller.
After the Sensor Chair, Tod Machover et al. created the Brain Opera [Par94][Orth97][Wax95][Par97]. The Brain Opera is a set of interactive musical activities designed to allow people, both young and old, to experience electronic music. There are the Rhythm Tree, Melody Easel, Harmonic Driving, Gesture Wall and Singing Tree. With the Rhythm Tree, users can hit many pads to trigger different sounds. These pads, made of hard urethane, are sensitive to impacts from different directions and several dozens of them are mounted on the surface of a huge bag. There are a few of these "trees" around in the activity area where many people can play them together like bands. The Melody Easel essentially lets users draw music on a touchscreen. Different parts of the screen have different musical pieces and voices. There are a few different easels in the area and each one can create different graphics and music. Very much like an arcade racing game, Harmonic Driving gives users control over the flow of a piece using a steering wheel. The wheel can be turned and bent in all directions, making it very flexible. The goal of the player is to steer a graphic representation of a note through a series of tunnels where the choice of tunnels determines what is played. The Gesture Wall is a combination of the Sensor Chair and Melody Easel but it is not tactile. It lets users select sounds and draw on an image projected on a screen. The Singing Tree encourages the user to sing and hold a note as purely as possible. When a perfect note is detected, the screen will slowly unfold beautiful graphics of a rose or a dancer.
The whole Brain Opera is designed for people from all walks of life to play and enjoy. Many of the concepts are good but some are not implemented as well as they were supposed to be. For example, the Melody Easel has a touchscreen that becomes very tiresome to play after just a while because the drawing surface is too hard and both the audio and visual responses are too slow; the pads of the Rhythm Trees are also too hard for prolonged playing and because of the different shapes and surface design, many actually hurt the hands of players; the steering response of the Harmonic Driving is not always as quick as what you would expect; the Gesture Wall animation is slow. There are certain areas where the job is very well done, like the Singing Tree and the Sensor Chair, which is used in a different portion of the Brain Opera that does not involve audience interaction.
The mixed sound generated from the various experiences of the Brain Opera is rather noisy. A challenging idea would be to make all the experiences work with one another in real-time to turn this mixture of sound into harmonized and orderly sound.
The Digital Baton [Par94][MP97][Mar96][Par97], another instrument that came out of the Brain Opera, made use of both infra-red tracking and a 3-axis accelerometer. An infra-red transmitter at the tip of the baton is picked up by a special camera in front of the conductor, allowing the baton tip's position to be tracked. An accelerometer allows directional beats and large gestures to be detected. There are also force sensing resistors in the over-sized handle of the baton for limited finger squeezing control. The Digital Baton is used on stage during the performance of the Brain Opera and is a remarkable instrument that combines both "free-air" movement and a "tactile interface". Aside from the over-sized and hard handle, which would make it difficult for children to play, it can be played by non-musicians too. Like many interfaces, the Digital Baton has a wire and a stationary infra-red tracking device. With these gone, with just the baton, the conductor would be able to move freely in space while conducting and creating an entirely new form of control.
Don Buchla, a famous builder of new instruments, has made an instrument called Lightning [Par97]. It is a pair of wands, transmitting infra-red and tracked by optical triangulation. It gives instantaneous position, velocity and acceleration of the wands in space. Unlike the Hands, the wands are wireless and have no contact sensors. This makes them less complicated and can be played by almost anyone. On the other hand, they are like a pair of batons, where a player expresses his intentions purely through intricate gestures by swinging and moving them in the air.
Buchla also made another instrument called Thunder [Par97]. It is a specialized MIDI controller that senses various aspects of the touch of hands, through the multi-faceted playing surface, organized to complement the shape and reach of the human hand. It transmits the resultant gestural information via MIDI to responsive electronic instruments. It introduces new concepts for defining musically interesting relationships between performance gestures and modern electronic vocabularies. With this, it has a steep learning curve since it takes the user time to understand it and to learn to use it to its fullest potential.
Any new instrument that comes out will always provoke strong reactions, from future predictions of what the next generation of instruments will be to just an announcement of its imminent demise. Even a very powerful controller can fail to catch on due to the lack of knowledge and experience of it by the general public. One such example is the Airdrums [LT93][Par97]. Invented by Pat Downs, they are also nicknamed the "MIDI maracas," and were made by Palmtree Instruments in 1987. This was a pair of sticks, about the size of claves, that send MIDI messages when moved on their axes in various directions with the use of acceleration, tilt, and rotation sensors.
People love music! It gives them sheer pleasure. Playing on the keyboard or pressing buttons to create music is fun but many people cannot play well because they are untrained. In fact, a lot of people do not compose, play an instrument, or even dance to music. For such persons, there is nothing easier than to simply squeeze out the music.
While all of these new musical instruments described explore new possibilities of a single expression or multiple expressions, only the Digital Baton has included "squeeze" as one of its inputs. Even then, the squeezing of the Digital Baton is very limited and proved to be tiresome. There is a need to create an interface that falls into this class to exploit its potential in music or sound manipulation. There is also a need to create more instruments that are not just intended for stage performance or for professional musicians and technologists.
This thesis will focus on the area of squeezing and explore the possibilities of multiple-player musical instruments for non-professionals or amateurs. It will attempt to fill in the gap for "squeeze" interface and also expand to both "squeeze and pull" combination of hand interface in Fig. 1.1.
An advantage of squeezing is that it is intuitively connected to emotions like happiness, fear and anger. For example, when a person is angry or tense, he tends to hold on to things very tightly; if a person is fearful, he tends not to hold things properly because he shivers. Also, many things that we do everyday require some amount of squeezing like washing and picking up a pillow. Squeezing is also a very intimate and delicate action, something that is very close and private like hugging loved ones or the way children treat their favorite teddy bears and dolls. In order to squeeze, there has to be an object that is tactile and physical. This makes squeezing a familiar way for people to explore the context of music and sound.
For the purpose of this thesis, the Squeezable is a category of interface that users squeeze to control the major behavior of a musical system. A Squeezable with a large range of continuous control is an obvious next step as an interface to complement the world of buttons, knobs and sliders in computer music. These conventional interfaces can have hard surfaces and can be uncomfortable to use. Also, they may not be the best way to capture the true feelings of the performer. With the Squeezables, sound designers and computer musicians will be able to enhance their expressions. Instead of conventional inputs like piano keys or cello strings, Squeezables have an interface that requires less practice to master, but which is still capable of careful manipulation, so amateurs will be encouraged to explore sound and music. Children will be able to play with this toy-like object and control the outcome of a piece of music. The Squeezables is an instrument meant for children and amateurs as well as professionals, both young and old. All these experiences can be accomplished by having the Squeezable exert high-level control over the music being created. For example, a Squeezable could control the parameters of rhythm and melody generation algorithms.
The group play feature of the Squeezables also gives players the shared controls of a piece of music. This not only promotes interaction among the players, it also frees the single player of the ever increasing number of control parameters he has to take care of. This seems to contradict the goal of creating more controls for one person, but it does introduce an element of fun when more people are playing in front of one another, especially to the young.
In the chapters that follow, I will describe the evolution of the Squeezables from its inception up to the latest version that senses more than just squeezing. All the important design steps of its development will be described, including experiments with materials, sensors, and packaging.