Table of Contents
The purpose of the Quiz Experiment was to create an adversarial quiz-like situation in which the use of affective sensors would be studied. A secondary goal of the experiment was to explore guilt associated with cheating behavior. The adversarial relationship was created between the quiz-taker and the experimenter: the quiz-taker was given the opportunity to exaggerate their score (viz. cheat) for greater financial gain for participating in the experiment while the experimenter sought (ostensibly) to reward genuine work.
In the Quiz Experiment 96 participants were asked to perform a boring and laborious task in an experimental design that allows the possibility of increasing the reward by reporting a score that is larger than what the subject actually achieved. The task asks participants to circle numbers that sum to 10 in a 3 X 4 grid [mazar2004]. The initial design of this experiment was borrowed from Nina Mazar and Dan Ariely. Following a number of changes that were made during a pilot study, the following experimental design was arrived at by myself, Phil Davis, and Rosalind Picard.
Figure 6.1. One of the 30 squares used in the quiz experiment. Participants were asked to find the pair of numbers that sum to 10
There were a total of eight conditions in the 2 x 4 experiment. The first dimension was the use of sensors versus no sensor information being collected. In addition there were four motivators: Control, No Effect on Charity, Charity Gains, and Charity Loses. In a similar experiment conducted as part of Phil Davis' thesis [davis2005], there were also 4 motivators: Control, No Charity, Charity Gains, Charity Loses. In the revised experimental design, the No Charity motivator was changed so that participants encountered a charity but their performance had no effect on the charity's reward. The rationale for this decision was that it is important to balance the presence of a charity. Thus, three of our conditions are identical to those in Davis's thesis, however new participants were recruited for all conditions.
Table 6.1. Quiz conditions
| Control | No Effect on Charity | Charity Gains | Charity Loses | |
|---|---|---|---|---|
| No Sensors | n=12 | n=12 | n=12 | n=12 |
| Sensors | n=12 | n=12 | n=12 | n=12 |
In the Sensors conditions, subjects were wired using ProComp+ 3-lead electrocardiogram and the HandWave skin conductance sensor. Additionally, mouse pressure and coordinates were collected. Participants in this condition also encountered an extra screen that explained how the sensors would be attached.
The No Sensor conditions performed the same task and experienced the same questionnaire instruments as the sensor condition. The difference was the absence of sensors and information related to attaching sensors.
The next few paragraphs will describe the different motivators participants experienced: Control, No Effect on Charity, Charity Gains, and Charity Loses. These motivators provided a variety of different compensation strategies to induce different behaviors. Procedures for each of these different motivators will follow.
Table 6.2. Quiz compensation
| Control (w/ Observer) | No Effect on Charity (w/o Observer) | Charity Gains | Charity Loses |
|---|---|---|---|
| Dollars "equal to half of your score" and $5 to charity regardless of performance. | Dollars "equal to half of your score" and $5 to charity regardless of performance. | Dollars "equal to half of your score" given to both charity and participant as well as $5 for participating. | "Total amount won between you and the charity (not including your $5 participation payment) will be $15." Each correct answer deducts $0.50 from amount charity receives and gives it to participant. |
The Control motivator was designed to discourage cheating through frequent presence of the experimenter in the testing room as well as the handing-in and verification of quiz work. In the Control and No Effect on Charity motivators participants were told, "Your score will determine your compensation for this experiment. The better you score, the more money you will win! Specifically, you will receive an amount of money equal to half of your score (in dollars). For example, if your score on the quiz is 20 then you will receive $10 (half of 20). Also, no matter what, you will receive an additional $5 for your participation. Charity Compensation: In addition, (charity) will win $5 (independent of your score). Please try as hard as you can on this quiz to maximize your earnings!" Here and below "(charity)" was replaced by a charity of the participant's choice.
In the Control there were two points during which the experimenter was in the room and interacted with the subject that did not occur in other conditions. In the practice quiz, once 5 minutes had elapsed an alarm sounded saying "your time is up, stop your work immediately and tell the experimenter you are done through the instant messaging client." This alarm would repeat until the participant had contacted the experimenter, who would then enter the room and disable the alarm. After this, the experimenter would collect the quiz work and look it over and leave the room. During the real quiz, once 5 minutes had elapsed, the same alarm would sound and the experimenter would again enter the room, disable the alarm, and collect and look over the quiz work. These were the only points that the experimenter was present in the room during the Control and not during the other conditions.
The No Effect on Charity motivator was the same as the Control except that the experiment did not enter the room to check the work; instead the subjects reported their scores through a web form.
In the Charity Gains motivator, better performance benefited both the participant and a charity of their choice. More specifically, the charity that was chosen was given as much money as the participant won. Participants were informed "In addition, (charity) will win an amount of money. The better you score, the more money (charity) will win! Specifically, you will win an amount of money also equal to half your score. For example, if your score is 20, you and (charity) will each win $10 (not including your $5 participation payment)! So, by scoring well, not only do you help yourself, but you also help (charity). Please try as hard as you can on this quiz to maximize your earnings!"
In the Charity Loses motivator, participants benefited at the expense of a charity. Participants were told "In addition, (charity) will win an amount of money. Unfortunately, due to budget constraints, the better you score, the less money (charity) will win. Specifically, the total amount won between you and the charity (not including your $5 participation payment) will be $15. So, by scoring well, you help yourself, but you also hurt (charity). Please try as hard as you can on this quiz to maximize your earnings!"
After signing up using the experiment scheduling system described in Appendix A participants arrived at the Media Laboratory. Because of difficulties with participants not appearing for their experiment, we elected to schedule every participant initially for either the Interview or Poker experiment. When one participant arrived for these paired experiments, but the other did not appear after 15 minutes I reassigned the one who had arrived to the Quiz Experiment.
I first escorted the participant to the office used to conduct the Quiz Experiments. This office was the same, regardless of whether the subject wore sensors or was in a condition without sensors. I would then launch GAIM, a freely-available instant messaging client, so that any questions participants had could be answered without them leaving the room where the experiment was conducted. At this point the participant would receive a consent form for the Quiz Experiment (see Appendix A).
The first web page encountered by subjects in Control motivator was used to verify their demographic information. After the consent form was signed and the identity of the participant verified, subjects were asked to move on to the next page.
On the second page of the Control motivator, participants answered initial questions. All participants were asked if they were left or right handed. Next participants were asked to select or name a charity that "may receive compensation in addition to your own compensation." Lastly, participants were asked "How much do you care about the charity you have chosen?" These first two pages were the same across all conditions.
In the case that participants were assigned to a sensor condition, they were shown a page that informed them that sensors would be used and how to contact the experimenter once the experiment started. In the non-sensor conditions, participants received verbal instructions explaining how to contact the experimenter. Participants were shown how to use an instant messaging client to contact the experimenter. This page was accompanied by audio instructions played from a speaker attached to a computer. Note that the page states "no other sensors" will be present. We were thinking at the time about video or audio, which could make subjects feel surveilled. We forgot that the pressure mouse was used, and it is a sensor (although technically so are keyboards and regular mice). Nonetheless, this should be reworded more accurately in future versions.
Following this, instructions for the practice quiz were given on screen and through recorded verbal instructions. Participants were told that "Your task is to find as many of these pairs as possible in 5 minutes. When your 5 minutes is up, you will: a) tell the experimenter that you are done through the instant messaging client so he can take your work. b) enter the number of pairs that you found on the computer. Note: The experimenter will verify that the number of pairs you report is accurate."
Once five minutes had elapsed, in the Control motivator participants were told they must contact the experimenter over the instant messaging client and were not allowed to continue. The experimenter would gather and examine their work to discourage cheating.
The next screen informed participants what actions they will need to perform when they've completed the real quiz. In the Control motivator participants were informed again that the experimenter would verify their score.
The next screen provided instructions concerning the real quiz. In the Control motivator, participants were told that the "experimenter will verify" their work. Outside of these verifications and initial instructions, participants did not interact with and were not directly surveilled by the experimenter.
Following this was a page which described what sort of compensation participants and the charities that they had chosen would receive. In the figure above, the participant has chosen "World Food Programme (UN)" as their charity. Participants are notified that $5 will be given to this charity "independent of their score."
Next participants in all motivators of the Quiz Experiment received a prompt that instructed them to complete the quiz. In an attempt to induce cheating, all participants were told that the average score on the quiz was much higher ("21.7") than the actual observed average.
After 5 minutes had elapsed, a screen appeared along with voice prompts that notified participants that they should either contact the experimenter.
Participants then saw a screen which asked them to enter the number of correct answers they found on the quiz. In the Control motivator, to discourage cheating participants were reminded the "experimenter will verify that the number of pairs you report is accurate."
After finishing the real quiz and reporting their performance, subjects moved on to a questionnaire designed to get at some of the ethical implications with which this thesis concerns itself. The questionnaire asked participants to consider a situation much like the one they encountered.
The survey questions on this page were shown as eight-point Likert-scale questions with a "No Opinion" option. Each of the questions opposed a category related to ethically positive term like "respectful" or "moral" against ethically negative terms like "invasive" or "immoral." Specific to the Quiz experiment was a last section of the questionnaire that asked participants to rate "the intentions with which you did the quiz" between "bad" and "good."
Having completed the questionnaire a page was displayed that explained to participants what their compensation would be as well as what their chosen charity's compensation would be. In an attempt to elicit some emotional states associated with guilt, some pointed questions about dishonesty and whether the money was deserved were displayed to participants. This was used for another piece of research [davis2005].
Finally, participants encountered a page thanking them for participating. The page also prompted participants to contact the experimenter using an instant messaging client so that the experiment could be completed. Informal discussions about the purpose of the experiment were then conducted by myself.
Participants assigned to the non-Control motivators experienced a very similar procedure. This section will detail the differences between the Control motivator and these motivators.
The first difference occurred on the sensor preparation information screen. Participants were told "The experimenter should now show you how to use the paper shredder and then leave the room." This was to show participants how to destroy their work so that no verification could be performed. The ideas was that with a a lack of verification, subjects would be more likely to inflate their score (cheat).
On the "instructions: practice quiz" screen participants were told: "When your 5 minutes is up, you will: a) shred your paper in the paper shredder b) enter the number of pairs that you found on the computer." instead of being told the experimenter would verify their work.
After participants' 5 minutes were up for the practice quiz, subjects in the non-Control motivators saw a screen which asked them to shred the quiz and to remember the number of correct answers they'd found.
On the following page, Practice Quiz Score (figure 6.7), participants were additionally told "REMINDER: Please be sure that your quiz has been shredded before going on to the next page." instead of being told that the experimenter would verify their work.
On the "Instructions: Real Quiz" (figure 6.8) page participants again saw a page that was similar to the Control motivator except they received different instructions: "When your 5 minutes is up, you will: a) shred your paper as you did before b) enter the number of pairs that you found on the computer c) answer some final questions about the experiment before your compensation is determined and given to you."
On the compensation page, participants saw text which reflected their motivator. The specifics of these different motivators were described above when the motivators were introduced. Participants experiencing the Charity Gains and Charity Loses motivators also were informed: "Some subjects have asked if we will really give money to the charity. The answer is: YES! We will give real money to the charity you have chosen depending on your performance!"
The next difference between the Control motivator and the remainder was the existence of a second shred quiz page after their five minutes were up for the real quiz. This page was identical to the figured entitled "Please Shred Your Quiz" above.
On the "your score" page the followed participants in the non-Control motivator were informed: "REMINDER: Please shred the quiz so that later subjects will not see your answers."
The remainder of the quiz was identical in all conditions. The next section turns its attention to the results from this experiment.