PDF Format

Much existing affective computing research focuses on systems whose designer intends to use information related to emotion to benefit users. This thesis discusses several adversarial uses of affective computing: systems where the designer's goal is to hinder some users. The approach taken is twofold: first experimental observation of use of systems that collect affective signals and transmit them to an adversary; second formation of normative ethical judgments regarding adversarial uses of these same systems. I propose to examine three adversarial contexts: the cheat experiment, the mole experiment, and the unfair experiment. In the cheat experiment, participants perform a laborious task that allows increasing monetary reward by deceiving the experimenter. The mole experiment, centers on a job interview where some participants hide information. Interviewers are rewarded for hiring the honest; interviewees are rewarded for being hired. The hypothesis is that participants using systems that sense and transmit affective information will have degraded performance and significantly different ethical evaluations than those using comparable systems that do not sense or transmit affective information. In the unfair experiment subjects are asked to play a simple poker-like game against an adversary who has relevant information. This information takes the form in the control of a card shown to the adversary and in the treatment the form of an interface that displays information related to emotion. The hypothesis is that there will be a significant difference in ethical evaluations of the two different varieties of relevant information. The contribution of these experiments will be design recommendations for affective computing systems used in adversarial situations and observation of situations that are proxies for potentially unethical ones.

When individuals have distinct goals which conflict with one another, the individual's relationship can be described as adversarial. Some of these relationships can be modeled as a zero-sum game in economic or game-theoretic analysis. Mediating technology has already been put to use in some adversarial ways:

Existing work has observed and analyzed adversarial relationships. Cohen et al. performed ethnography investigating the "phenomena of adversarial collaboration" in workflow systems used in a law firm [cohen2000]. Applbaum provides an ethical analysis of adversarial roles [applbaum2000].

Individuals often view their emotions to be especially sensitive and private matters. As such, adversarial uses of systems that sense and communicate affect are especially interesting as a domain of inquiry. The problem that this work is addressing is the lack of information concerning user responses to affective communication systems in a adversarial use context. The approach taken to this problem is to repeatedly induce situations that are adversarial and then collect performance and survey data in an experimental context. The idea is to use this information to inform design of future systems that sense and transmit information related to emotion in ways that are ethical. The next section provides an outline of the content of the thesis itself.

The proposed thesis is broadly split into two main parts. The first is a theoretical development whose goal is to adequately define the concepts the thesis analyzes. The second is the description of two experiments whose goal is to produce design guidelines for use of affective computing systems in adversarial situations. The outline for the final thesis is proposed to be:

This thesis will concern itself with a class of adversarial uses of affective computing: systems that sense and transmit information related to an emotion to an adversary. "Affective computing is computing that relates to, arises from, or deliberately influences emotions" [picard1997]. Affective computing systems can be loosely categorized into systems that reflect affect back to individuals and systems that mediate the communication of affect between individuals. Since adversarial situations involve conflicting goals of more than one agent, the focus of this thesis will be on systems that mediate the communication of affect between at least two people.

What exactly is a system that mediates the communication of affect? Starting from Shannon's view of communication, any system that communicates can be decomposed into an information source, transmitter, channel, receiver and destination [shannon1948].

Figure 1. Shannon's schematic diagram of a general communication system.

A system that mediates the communication of affect is a special case of Shannon's general communication system where the message contains information related to emotion. The information source is most typically a human. The message is a display of emotion by a human. The transmitter often consists of sensors and recognition algorithms that transform this message into a signal that is appropriate for transmission.

It is more enlightening to consider an example of a system that mediates the communication of affect.

Figure 2. Emotemail

Emotemail is an email client that is augmented to convey aspects of the writing context to a receiver [angesleva2004]. The client captures facial expressions and typing speed and encodes them into a signal. The signal is a base64-encoded image that can be read by many email clients. If it is granted that facial expressions and typing speed relate to non-verbal expressions of emotion, then it may be said that the system communicates affective information.

Emotemail's potential to encode and transmit information related to affect gives rise to interesting situations. Consider the case in which the sender and receiver of the affective information are in conflict with one another. For instance, if an affective communication system were used by a government against militant foreign nationals as part of a visa application process.

This thesis wishes to evaluate individual's responses to using such systems from the stand point of ethics. If such an evaluation is to be realistic then different social factors potentially bearing on individual's evaluations ought to be acknowledged. What social dimensions are relevant in analyzing such systems from the standpoint of ethics?

The above table presents a non-exhaustive list of many factors that could influence an ethical evaluations of systems that mediate the communication of affect. Clearly, a thesis cannot throughly consider all of these factors simultaneously. Since the focus of this thesis is on adversarial relationships, the experiments will seek to constrain a number of these variables: most relevantly the goal relationship, power relationship, genre of emotion, and risk encountered by participants.

The contribution of this thesis will be guidelines for the design and use of systems that mediate the communication of affect in adversarial situations. Additionally, this thesis will contribute to the development of computer ethics by observing situations that act as proxies for unethical ones.

One might wonder: "what is the relevance of information and guidelines about adversarial uses of systems that mediate the communication of affect?" Information technology is increasingly being used in adversarial ways that may not benefit individuals but may overtly harm individuals. Consider for example the "RQ-/MQ-1 Predator Unmanned Aerial Vehicle" equipped with Hellfire missiles [buchanan2004]. These semi-autonomous systems have been successfully used to kill individuals in military operations. Also consider the PackBot, a robot capable of full autonomy that will "be equipped with a pump-action shotgun system able to recycle itself and fire remotely" [stamant2004]. The haptic opposition project in the Computing Culture group also examined "human-machine conflict" by building uncontrollable devices [schiessl2003].

When human beings have the capability to observe, report, or harm other individuals they are often aware of ethical and moral considerations. For instance, military officers make use of a Code of Contact that describes their ethical responsibilities [dod1998].

I contend that autonomous systems with the ability to harm ought to have an awareness of ethical and moral constraints as well. Similarly, I think that if affective systems are to be widely deployed and used in ways that could observe and report emotional information or act on this information in a way that harms individuals then designers of affective computing systems should have an awareness of these ethical and moral constraints.

There is existing research exploring adversarial uses of information technology [cohen2000] [veloso1998] but the corresponding human-factors evaluations have not been conducted for affective technology. This thesis and the data collected by the experiments described will provide novel information about human reactions to affective computing technology used in adversarial situations.

Much has also been written that develops the connection between emotion and ethics [spinoza1673EM] [frank1989] [damasio1995] [hatzimoysis2003] [solomon2004]. In these various discussion of the "passions" philosophers have attempted to show how different emotions may interact with and give rise to ethical behavior. Similarly, much has been written that connects power with ethics [spinoza1673PW] [foucault1975] [detel2004]. It is my intention to build upon an existing philosophical analysis of "passions" or emotion, as opposed to performing this analysis from first principles.

It is then important that the scope of this thesis be limited in a manner to topics and material that haven't been well explored. Furthermore, the concrete outcome of the thesis should offer practitioners information that does not already exist.

This thesis will offer recommendations to designers of systems that mediate the communication of affect. These will be framed in a manner that does not require a through understanding of the backing ethical theory. Despite the possibility of generalizing this work to systems that communicate other varieties of information, care will be taken to remain focused on systems that communicate affective information.

In the "cheat experiment," 72 participants will be asked to perform a boring and laborious task in an experimental design that allows the possibility of increasing the reward by deceiving the experimenter. The task asks participants to circle numbers that sum to 10 in a 3 X 4 grid [mazar2004]. One control group 24 participants are given the opportunity to deceive the experimenter by turning in an unverifiable number of correct answers to the experimenter and destroying their proof (the answer sheet with circles). A second control group of 24 performs the same task, but must sign an "honor form." A third group of 24 must give their answers for external verification. In each of these groups, a treatment block of 12 will use an affective communication system to enter their results. The participants will be told that the experimenter's goal is to ensure proper rewards are given.

One half of the summation problems are designed to have no correct answer; so out of 20 problems only 10 have solutions. In each case solutions will be submitted by use of electronic form. Deviating from the original design, participants will be told that their goal is to get as many answers as possible, and the experimenter's goal is to ensure participants are properly rewarded.

In addition to performance measures (how many "correct" answers participants report) there is a deception measure (how much "better" does the population given the opportunity to cheat perform). Each condition will be asked a battery of questions concerning ethical issues:

It is my prediction that participants who are using systems that sense and transmit information related to emotion in adversarial situations will view the setup as more unethical, invasive, uncomfortable, hindering, immoral, and suspicious as well as express a preference for a system without sensors when compared to individuals in a control group.

In the "mole experiment," 144 participants will be placed in a mock job interview. 72 of the participants will be randomly assigned to the role of interviewer for a fictional company. Interviewers will be rewarded for hiring participants who work for "Good Company." Additionally, interviewers will be rewarded for refusing to hire participants who work for "Bad Company."

In contrast, the 72 interviewees will be randomly assigned to have either "Good Company" or "Bad Company" as their existing employer. Interviewees will be provided with a variety of motivations. One group of 24 will be assigned to a control group who used to work for Good Company and are rewarded for getting the job. A second group of 24 with good motivations will be told they used to work for Bad Company and should be deceptive in order to get a job. Furthermore they are told that if they get the job then they will be rewarded and an reward will go to a charity of their choice. A third group of 24 with bad motivations will also be told they used to work for Bad Company and should be deceptive in order to get a job. However, differing from the good motivation block, they will be told that if they do not get the job, a charity of their choice will be rewarded; and if they get the job they will be rewarded instead.

A treatment group of 36 interviewers will be given affective information (facial expressions, EKG, mouse grip force, and skin conductivity). Interviewees, on the other hand, will not be told exactly what information is being collected or how it is going to be used. A control group of 36 interviewers in the mole experiment will perform the job interview using instant messaging software.

Interviewers in both groups will perform the interview using an identical script:

The mole experiment collects performance measures, in terms of how often moles are hired with and without a system that mediates the communication of affect. After the mock interview, both interviewer and interviewee will be asked to fill out the same battery of questions used for the unfair experiment.

In the "unfair experiment," 144 participants play a simplified version of poker in pairs. The games rules are as follows: first each player antes, next each player is dealt two cards, the players attempt bet who has the highest card and are given the opportunity to fold.The players compete for a common pot and their implicit goal is to maximize the reward. The game will be mediated by a simple web-based interface.

Two sets of motivations are set up for the players. Participants will be randomly assigned to have either good or bad intentions. In the good intentions group, rewards are split with a charity of the participant's choosing. In the bad intentions group, rewards come at the expense of the participants favorite charity.

Each participant will experience either a control or unfair treatment. The unfair treatments consist in either being forced to show one card to their opponent or to transmit information related to emotions to their opponent.

In addition to performance measures (how well the players do in the game) each condition will be asked a battery of questions concerning ethical issues (similar to the cheat experiment):

It is my prediction that participants who are using systems that sense and transmit information related to emotion in adversarial situations will view the setup as more unethical, invasive, uncomfortable, hindering, immoral, suspicious, and unfair as well as express a preference for a system without sensors when compared to individuals in a control group.

It is my goal to defend this thesis in 2005 and submit a version with changes in response to feedback from the defense in April 2005. The experimental apparatus has already been procured and or programmed. A pilot has been conducted as well.

A tentative schedule for the thesis would be as follows:

Already in place in support of this thesis is COUHES application (#0407000841), which is in the process of amendment. I recognize that this is an "aggressive timeline" and am prepared for the process to take longer than anticipated.

The main resource requirements will be incentives for the two experimental designs. I request $5040.00 in total

The cheat experiment will offer a minimum of $5 and a maximum of $10 for each participant. With 72 subjects, the cost for the experiment in the worst case would be $720.00.

The mole experiment would offer a $5.00 for participation and $5.00 for being "hired" or "catching" a mole plus an additional $5.00 for charity in some conditions. The 144 participants would then require $720.00 for baseline participation. While it is possible for non-moles to be falsely accused, the most costly situation would be in every case either a mole being caught, or a non-mole hired for which the reward is $15. This brings the total cost of the experiment to $2880.00.

The unfair experiment would offer a maximum reward of $10, starting each subject with $5.00. Since there are 144 subjects, the cost for this experiment would be $1440.00.

The experimental apparatus will be constructed from already-existing systems and freely available software. In terms of computing hardware, I will need access to one computer for the two computers for the "mole" and "unfair" experiments. As for space resources, two small, quiet, and isolated spaces near one another are requested.

Information technologists have not yet experienced something akin to what the physics community experienced with the advent of atomic weaponry. There have been serious incidents like the design flaws of the Therac-25, which lead to death and suffering in cancer patients [leveson1993]. Holocaust survivors also brought suit against a German subsidiary of IBM. The suit claimed that this subsidiary provided computers used to maintain prisoner records [sebok2001]. However, analysis of these incidents doesn't seem to have much to say about real applications that mediate the communication of affect.

The MIT Media Lab is ideal in more than one way for conducting this sort of thesis. Research from the Media Lab has been ridiculed as so technology oriented as to avoid considerations like the ethical consequences of pervasive sensors [anonymous2000]. The Media Lab for some represents the archetype of science without conscious: technology for its own sake.

However, there has already been some work at the lab that addresses privacy [mann1996], social, philosophical, and ethical issues [foner1999] of information technology. What differentiates this proposal from that work is empirical experimental observation of actual systems that mediate the communication of affect. The lab is ideal because of the quality of work on mediation systems and the receptivity to unorthodox approaches, as well as expertise in affective computing.

Picard and Klein broached theoretically unethical uses of affective systems [picard2002]. But these unethical uses were not wholly investigated. Following this work has been some preliminary forays into investigating the privacy consequences of this technology [reynolds2004CHI]. However, privacy is only a single dimension of ethical import.

Value-Sensitive Design [friedman2002] articulates many dimensions that are relevant to systems that mediate the communication of affect. Value-Sensitive Design (VSD) is "an approach to the design of technology that accounts for human values in a principled and comprehensive manner throughout the design process." It considers Human Welfare, Ownership and Property, Privacy, Freedom From Bias, Universal Usability, Trust, Autonomy, Informed Consent, Accountability, Identity, Calmness, and Environmental Sustainability as values that may be of ethical consequence. Friedman and Nissenbaum applied VSD to evaluation of bias in computer systems [friedman1997]. VSD has been applied by others to problems such as online privacy [agre1997] universal usability [thomas1997], urban planning [noth2000], and browser consent [friedman2002HICSS]. The Tangible Media Group has considered various ambient displays that support the something akin to the VSD notion in their research on computer-supported cooperative work and architectural space [wisneski1998].

In "It's the computer's fault: reasoning about computers as moral agents," Friedman also considered how people evaluate the ethical and moral consequences of computer programs [friedman1995]. In interviews with computer-science students, Friedman found that 75% attributed "decision-making" to computers. But only 21% held the computer "morally responsible" for errors. These results indicate that the majority of the interviewees thought a computer could make decisions but a minority blamed the computer for the consequences of bad actions. One participant was quoted as saying "the decisions that the computer makes are decisions that somebody else made before and programmed into the computer . . ." Friedman concludes by noting that "designers should communicate through a system that a (human) who and not a computer (what) - is responsible for the consequences of computer use." This work suggests deeper questions about the possibility of a computer having ethical behavior.

But what does it mean for a computer to be ethical? Does the rule-following of an artificially intelligent chess program count as moral behavior? Perhaps one of the first individuals to explore these questions was Asimov whose fictional work on robot ethics has made the topic interesting and accessible to a wide audience [asimov1956].

Computers, as they are currently designed, do not have the capability to be ethical on their own. Lacking free will, Turing Machines do not make moral choices between "good" and "bad." Instead, they largely carry out their designer's choices. This means that if a designer makes "bad" choices from the user's perspective, the resulting interaction could be viewed as unethical.

Morr, in the classic article "What is Computer Ethics?" [moor1985] conceptualizes computer ethics as dealing with the policy vacuums and conceptual muddles raised by information technology [bynum2001]. This definition leaves open an important area of debate: the foundation of computer ethics. Floridi and Sanders categorize different types of foundations that have been used as a basis for ethical arguments about computers [floridi2004]. Many topics have been analyzed from the standpoint of computer ethics: privacy, crime, justice, and intellectual property [brey2000]. Of these, privacy is a value that is directly linked with communication systems.

Palen and Dourish define privacy to be a "dynamic boundary regulation process" in an extension of Altman's theory [palen2003]. Their view of privacy is as a dialectic process between "our own expectations and experiences" and others with whom we interact. Privacy has also been considered in value-sensitive design methodologies [friedman2002]. Friedman et al. also worked on the impact of informed consent in the domain of web browsers [friedman2002HICSS]. Bellotti and Sellen examined privacy in the context of ubiquitous computing. Their studies took place in the context of pervasive sensors (microphones and video cameras). They found that "feedback and control" were two principles that were important in the design of acceptable environments with sensing [bellotti1993]. Mann found that the notion of symmetry in surveillance can help balance inequities that cause privacy problems [mann1996]. Lederer, Mankoff and Dey studied location-sensing technology and determined that "who" is asking for information is an important factor for those determining preferences or policies for access to private information [lederer2003]. Hong also discussed context fabric as an architecture that provides support for privacy in ubiquitous computing systems [hong2004].

Outside of a narrow focus on privacy, there have also been some unusual approaches to considering computers and ethics. Weld and Etzioni worked to include the notion of "harm" into a planner to create ethical "softbots" [weld1994]. Eichmann proposed an ethic for Internet agents and spiders to limit bandwidth abuse [eichmann1994]. Allen et al. suggest a "moral Turing test" as a method to evaluate the ethical agency of artificial intelligence [allen2000]. Wallach also proposes the research and development of "robot morals" [wallach2002]. Brey proposes "disclosive computer ethics" as a methodology for maintaining human values [brey2000].

After all these different uses of "ethics" designers might ponder exactly "what is ethics?" It is beyond the scope of this proposal to answer this question. But those seeking more information might consult MacIntyre's A Short History of Ethics [macintyre1967]. A gentler introduction for non-specialists is also provided by Introducing Ethics [robinson2001]. Additionally, Sher has collected an anthology of readings related to ethics and moral philosophy [sher1989].

A question that is within the scope of this proposal is: "what do you the author mean by ethical?" Ethics [fieser1999] is often divided into:

What I mean by ethical is the application of ethical theory stemming from commitments to a metaethical position. An example might help clarify what this somewhat cryptic utterance means in practice.

Consider the contractualist metaethical position. Contractualism founds ethical evaluations on a hypothetical or real contract formed between groups or individuals. A enormous amount of metaethical philosophy can be termed contractualist including the work of Hobbes, Rousseau, Rawls, and Gauthier.

Cudd describes the contractual macroethical position in the following manner: "Contractualism, which stems from the Kantian line of social contract thought, holds rationality requires that we respect persons, which in turn requires that moral principles be such that they can be justified to each person." [cudd2000]. Thus, we should offer our moral decisions in public and seek to justify them to each user.

In "Affective Sensors, Privacy, and Ethical Contracts," Reynolds and Picard discuss the application of contractualism to problems involving hypothetical systems that sense and communicate affect [reynolds2004CHI]. Following this, Reynolds and Picard discuss the relationship between contractualism and the value-sensitive design development of informed consent [reynolds2004AD]. Both of these papers make metaethical commitments and then proceed by applying relevant ethical philosophy to problems related to the design of affective computing systems.

Figure 3. Author Picture

Carson Reynolds is a doctoral candidate in the Affective Computing group at the MIT Media Laboratory. He holds a Master of Science in unspecified topics from the Massachusetts Institute of Technology and a Bachelor of Science in Technical Communication with a Minor in Philosophy from the University of Washington at Seattle. Mr. Reynolds is an Eagle Scout and was awarded the Media Lab Europe fellowship in 2003. He has worked as a Systems Analyst for the University of Colorado and Keane, Inc., as an Application Developer for Fine.com Interactive, Soma Corporation, and CVS.com, and as a Researcher for Mitsubishi Electric Research Laboratory and Media Laboratory Europe. His work has been discussed on National Public Radio, Scientific American Frontiers, the Washington Times, Focus, and online in Wired, Engadget, Smart Mobs, and Slashdot. Additionally he has contributed web articles to Lambda the Ultimate, OSnews, Wikipedia, and Slashdot. Carson is also an active contributer to the Gentoo, Del.icio.us, and Mozilla user communities. In his free time he volunteers as an instructor at the MIT Climbing Gym and has performed community service developing charity websites. Currently Carson is researching applied ethics, affective sensors, and physiological interfaces for games.