All of the experiments (described in the following chapters) took place in the same location: a pair of offices in Room E15-001 at the Media Lab. All of the subjects in conditions involving sensors were in office E15-120f, which is pictured below. Situated in this office were the affective sensors used to collect and transmit information related to emotions. Additionally, subjects who were the control analogs of the sensor conditions also used this office.
Participants who received information from these sensors (but who did not use themselves wear or use sensors) sat in office E15-120g. Moreover, participants assigned to control analogs of the conditions receiving information from the sensors also used this office. Both of these offices were equipped with identical Dell computers running Windows XP.
This section will detail the development of the Pressuremouse, which is used in each of the experiments described below. Pressuremouse is a standard computer mouse that has been augmented to capture information about grip force during interaction.
In exploring devices to sense behavior associated with frustration, one method that I explored was equipping a mouse with force sensors. At the outset of this work, I had already constructed a pressure-sensitive mouse that used 8 sensors covered with a conductive elastomer [reynolds2001].
To help better understand how individuals grip-force changes over time I developed a program called Cheesemouse. The program collected data from the surface of the Pressuremouse and rendered it upon a screen-capture video. This work builds off of Cheese [mueller2001].
The data acquisition board for the Pressuremouse was originally developed on a prototyping board. For day-to-day use this was found to be unstable and unreliable. Consequently, I undertook work to first reproduce the design as a printed circuit board (PCB) using the same dual-inline package components.
Working with Keith Battocchi, this design was further reduced in size by switching to surface mount components. This led to a board that was small enough to be housed inside the mouse itself. The data acquisition board was also altered to parasitically draw power from the mouse.
In order to produce a number of Pressuremouse prototypes, Manta Product Development was retained to revise the design. They suggested using off-the-shelf force sensitive resistors in six locations. They also suggested covering the mouse in a shrink-fitted shell that transmitted force to these points.
This was in turn covered with a transparent shell with colored splotches which obscured the sensors. The current design looks and feels very much like a "normal" mouse but provides un-calibrated data relating to how much force is applied to the surface. The dynamic range of the sensor is tuned such that when normal use occurs, there is a low amount of force detected and when the user is over exerting their muscles the analog to digital converter reports a higher reading.
In a a collaboration with Jack Dennerlein of the Harvard School of Public Health the relationship between grip force and user frustration was explored. Participants in the study made use of a web form which I designed to be intentionally frustrating. Specifically, the form's design was mildly unusable and often forced the user to re-enter information. Initially, Jack's experiment tested the hypothesis that all subjects would exhibit more mouse force after frustrating stimulus. After observing no significant difference he then separated participants into high and low response groups depending on those who reported frustration after using this webpage. Using EMG sensors, the activity of several arm muscles was recorded. A force-sensing mouse was used to record grip force using miniature load cells. The study found that "force applied to the mouse was higher (1.25 N)" after frustrating stimulus when compared to interaction during control (0.88N) for the high response group (p=0.02) [dennerlein2003].
This section will detail the development of the HandWave, which is also used in each of the experiments described below. HandWave is a wireless skin conductance sensor.
Galvanic skin response (GSR) is a term which is often used to describe the electrical activity which gives evidence of psychological changes [fuller1977]. Electrodermal Activity (EDA) is a broader term used by the psychophysiology community to describe the changes which take place on the stratum corneum in response to the sympathetic nervous system [malmivuo1995]. It is thought that the autonomic functions of the brain control the output of sweat glands and that electrodermal activity varies with psychological changes like increased arousal and anxiety [fenz1967].
Skin conductance is often measured using a bipolar electrode placement on the medial phalanx. Malmivuo and Plonsey suggest that a voltage of 0.5 V that is kept constant across the skin is present-day practice [malmivuo1995]. This is apparently because the of the conductance of the skin is linear for voltages under 0.7 V.
The skin conductance response consists of two components: the tonic and phasic [boucsein1992]. The tonic is slow moving, oscillating over the course of days. The phasic is fast moving, and spikes sharply when a person is startled, and generally increases when a person is psychologically aroused. Many skin conductance amplifiers include some adjustment so that the tonic portion can be removed and the phasic measured more accurately.
The current design for the galvactivator glove designed by Jocelyn Scheirer and her colleagues makes use of a "Darlington Pair" to amplify 6V dropped across the skin. A super-bright LED is lit in response and the circuit can be varied by adjusting a 500K Ohm potentiometer. This circuit was primarily designed to be inexpensive to reproduce. It however has several design flaws as a more clinical skin conductance design. Foremost, the circuit does not keep voltage constant by buffering it properly. Furthermore the circuit does not regulate voltage so that as the battery drains, the circuit can provide very different responses.
A more sophisticated and expensive design was developed by Blake Brasher. This design made use of a pair of Op Amps: one to buffer and a second to serve as a non-inverting amplifier. This design remedied many of the shortcomings in the original glavactivator circuit. For the problem at hand (use with Bluetooth wireless transmitters) however, this circuit still needs analog-to-digital (ADC) conversion.
A design by Brian McDonald of the Mindgames group provides a much more sophisticated skin conductance amplifier, but also introduces a Butterworth low-pass filter to address aliasing and noise issues. This design was interfaced with a PIC microcontroller which ran assembly ADC code.
McDonald's design was used for Relax-To-Win, a biofeedback game which makes use of skin conductance. The game takes the form of a race in which the players move faster when their skin conductance is lower relative to a baseline [bersak2001].
Over the course of nearly 2 years I experimented with a large number of circuits to amplify the skin conductance response and to pass this information to a host computer via Bluetooth. The later, more sophisticated designs made improvements over these previous designs by providing mechanisms to adjust automatically to skin resistance.
The HandWave device combines analog circuitry to condition the signal collected from electrodes across the skin with an ADC and Bluetooth transceiver. The design pictured above incorporates a 16-bit ADC with enough resolution to ignore the tonic offset. The electrical schematic for this approach is considerably less complex than many of the other approaches tried, due mainly to the removal of the PIC microcontroller. The ADC chosen provides an inter-integrated circuit (I2C) bus which can be interfaced with wireless transmitters like the BlueCore 2.
A large number of different form factors were also experimented with. In collaboration with Marc Strauss, handheld orbs, and wrist-mounted versions of HandWave were tested for use. The most recent design, which is a wrist-mounted variety with large 9V battery, was selected for the experiment.
This design served as a starting point for a collaboration with Marc Strauss to redesign the HandWave device. The result is documented in his thesis, HandWave: Design and Manufacture of a Wearable Wireless Skin Conductance Sensor and Housing [strauss2005]. This more recent design uses an embedded microcontroller to improve the sampling rate achieved with the version discussed above. A larger wrist-mounted form factor was also used to accommodate a 9 volt battery.
For the actual experiments described below, HandWave (to collect electrodermal response (EDR)) and the Pressuremouse (to collect grip force) were used in conjunction with a ProComp+ sensor system (to collect electrocardiogram (EKG) information) and a face-tracking web camera. I wrote Python drivers to log the physiological data as well as keystrokes and mouse coordinates.
I wrote a second python program, called MixedEmotions to display the output of these sensor to other participants in certain experiments. This software connected to the sensor drivers via TCP/IP socket and then displayed the data collected as a video image and strip charts. The charts each showed 500 data points and were updated at various rates: the Force window at 40 Hz, the EKG window at 250 Hz, and the EDR window at 40 Hz.
Participants in the sensor conditions described in the experiments below made use of the program MixedEmotions. This system recorded skin conductance, grip force, and electrocardiogram data, as well as video of the face. In some cases, this data was presented to an adversary.
Video from a face tracking camera had a prominent place in the MixedEmotions interface. Facial expression have long been recognized as important carriers of non-verbal information. Various efforts to systemize facial expressions have reduced them to a two dimensional space [schlosberg1952] and a set of action units describing the movement of the muscles that control facial expressions [ekman1978].