definitive resource and media library (movies, screenshots, papers,
Common Sense -Informed NLP
Oriented Web Search User Interfaces
Believe: Interactive Computer Story Generation
Sense in ARIA
Ponens / Empathy Buddy
Eye: Visualizing Affective Structure of Stories
Practical Common Sense Reasoning Toolkit
Lexicon: Reinventing the Cognitive Lexicon
H R O N O L O G Y o f P R O J
E C T S *
denotes recent activity on project
Hugo Liu, Glorianna Davenport, and Pattie Maes (2005)
if you could look in the mirror and see not just what you look like,
but also who you are?
Identity Mirror is an augmented evocative object. Looking into it,
the viewer's face is painted over with identity keywords and interest
keywords, sourced from a deep model of the viewer's identity. The
identity model is computed automatically from a viewer's social
network profile or webpage, using the InterestMap.
For instance, the viewer specifies that he listens to "Kings
of Convenience" and enjoys the fiction of Vladmir Nabakov,
and using this, InterestMap situates the viewer within its multiple
neighborhoods of taste. The keywords which paint over the viewer's
face represent his context within taste-space.
gazing into Identity Mirror, a viewer can glean his identity-situation.
Is his hair out of place? Are one of his interests out
of place? How do his facial features combine to compose a gestalt?
How do his various interests come together to compose an identity
or aesthetic gestalt?
mirror reifies its metaphors in the workings of an ordinary mirror.
When the viewer is distant from the object, a question mark is the
only keyword painted over his face. As he approaches to a medium
distance, larger font sized identity keywords such as "fitness
buffs", "fashionistas", and "book lovers"
identify him. Approaching further, his favorite book, film, and
music genres are seen. Closer yet, his favorite authors, musicians,
and auteurs are known, and finally, standing up close, the songs,
movies, and book titles become visible.
ongoing research, we're developing further two particular aspects.
IdentityFixing and the Diderot Effect -- keywords are distributed
between a hearth (keywords aesthetically co-consistent) and a periphery
(outlier keywords seemingly out-of-place about a person); the hearth
covers the face, the periphery covers the hair; the viewer can use
his hands to adjust his hair -- he can dishevel those unwanted periperhal
keywords, or accept them by packing them into his hair. A person
with a strong degree of taste-coherence has ruly hair, whereas a
postmodernist with scattered interests has unruly hair. The second
aspect under development is temporality -- the image of the viewer
will reflect the viewer in relation to the goings on of the world
and of his life. Since the viewer has many facets, various facets
can be teased out by biasing InterestMap with contemporaneous keywords
of current worldly and lively goings-on.
MOV, 2minutes, with sound) NEW! identity mirror
MOV, 1minute, no sound) Older identity mirror, demonstration
and Glorianna Davenport (in works) Self-reflexive performance: Dancing
with the computed audience of culture. International Journal
on Performance Arts and Digital Media.
Glorianna Davenport, Pattie Maes (forthcoming). Taste Fabrics and
the Beauty of Homogeneity. For the Association of Information Systems
SIG SEMIS Bulletin 2(?).
Pattie Maes, Glorianna Davenport (forthcoming, 2006). Unraveling
the Taste Fabric of Social Networks. For the International Journal
on Semantic Web and Information Systems 2(1). Hershey, PA: Idea
Recipes: Articulating Cravings
Hugo Liu and Matthew Hockenberry (2005)
this work, we explore a technological answer to that famous question.
Few of us know with great certitude the exact food we crave, but
instead, we stew on the question and explore the nature of our craving
through imaginative descriptions: "I feel like something light,
fresh, sophisticated, not too mushy -- something influenced by thai
or indian ingredients, something aromatic." Synaesthetic Recipes
is a visual search program which allows such imaginative textual
descriptions, and uses these to drive recipe recommendations. In
the backend, a database of 100,000 recipes are automatically annotated
with common sense about food. An artificial intelligence robotic
reader reads each recipe and based on tastes of the ingredients
and the types of cooking procedures, predicts how a food will look,
taste, and smell. We are translating recipes into the rich descriptive
vernacular of how people naturally conceptualize their cravings
of the Flash SWF Foraging Interface
MOV, 2m34s sound) Synethetic Recipes: foraging for food with the
family, in tate-space
& Matthew Hockenberry, and Ted Selker (2005). Synesthetic Recipes:
foraging for food with the family, in taste-space. Proceedings of
SIGGRAPH'2005, Los Angeles.
A Cultural Fabric of Identities & Interests
Hugo Liu and Pattie Maes (2004)
recent years, social network communities (e.g. inter alia,
friendster, contact lists, weblog communities, newsgroups) have
been steadily building up in the online world. There is now sufficient
critical mass of such infrastructure to postulate things about identity
and the Self, as reflected in the social fabric of the online world.
might we sense identity from the online social fabric? Semiotician
Jacques Lacan has argued that words and concepts carry meaning primarily
by what they signify. Roland Barthes proposed that the particular
mappings between signifier and signified originate in cultural systems
of semiology. Such cultural systems of signs have grown in importance
as mass consumption has replaced subjective culture as the dominant
contemporary cultural paradigm. Because the signifying value of
possessions associated with the Self serves an important social
function in signalling identity, it is possible to view identity
and the Self as a collection of consumption decisions (cf.
Social Constructionist Theory of Identity).
mines online social network communities to create a rich influence
network of interests and subcultures. Some of the interests represented
in the network are, inter alia, television and films, foods,
geographies, music, sports, hobbies, activities, objects, and people.
The strength of connections between interests are learned from the
digestion of on the order of one hundred thousand user profiles
from online social networks. For example, a person who likes X may
also mention Y as an interest on her homepage. Identity and tastes
emerge as patterns of intersection on InterestMap.
InterestMap contains 100,000 interest and subculture nodes, trained
with over 100,000 thousand profiles. It is currently used as an
interest engine, driving serendipitous recommendation and social
introduction facilitation in the Ambient Semantics project, but
it certainly has far broader implications.
also see InterestMap as a novel kind of social recommendation mechanism.
Whereas recommendation systems traditionally works within a single
application domain of interest such as books or music, InterestMap
represents users using a much larger vocabulary of interests (movies,
music, television shows, foods, sports, hobbies, passions) and ways
of describing people (identity labels e.g. "raver", geographical
locations, etc). The result is a multi-dimensional model of a user
that is portable across domains; it can be used by Amazon, or any
other conceivable vendor. More importantly, it has the potential
to instruct computers to understand and describe people as people
Pattie Maes (2005) InterestMap: Harvesting Social Network Profiles
for Recommendations. Proceedings of the Beyond Personalization
2005 Workshop, January 9, 2005, San Diego, CA, USA, to appear. ACM
Glorianna Davenport, Pattie Maes (forthcoming). Taste Fabrics and
the Beauty of Homogeneity. For the Association of Information Systems
SIG SEMIS Bulletin 2(?).
Pattie Maes, Glorianna Davenport (forthcoming, 2006). Unraveling
the Taste Fabric of Social Networks. For the International Journal
on Semantic Web and Information Systems 2(1). Hershey, PA: Idea
of the Who Am I visualisation for InterestMap
MOV, 2m20s no sound) Movie demonstration of "the fabric of
interests". Caption for Video: "As a person specifies
aspects of her interests, she is implicitly situating her identity
when an interest fabric. As a result of this situation, all the
new interests which have become proximal assert their relevance
to her kind of person"
Hugo Liu, Assaf Feldman, Sajid Sadi, Emmanuel Munguia Tapia, and
Pattie Maes (2004)
massive amounts of digital information can be searched through.
Information about friends (e.g. friendster, orkut, im buddy lists),
about the everyday (e.g. weblogs, the news), and about virtually
any book, movie, song, or topic (e.g. amazon, google). Rather than
becoming evermore consumed with being online, we would like to fold
the digital world back into our offline lives.
ambient semantics project uses a simple wearble RFID device to track
things you pick up and people you meet, and looks for opportunities
to present you with useful just-in-time information at the moment
of your curiosity. "What did my friends think of this book?"
"What common interests and friends do i share with this person?"
The system leverages coRelate, a large knowledge base about the
relatedness of interests, to meaningfully connect people with people,
and people with objects. By modeling a person's interests, the system
becomes a personal serendipity assistant that we hope will enrich
Pointer" Technology Review, February 2005
Serendipity" MIT Technology Insider Magazine, November
2004, p. 3
Hugo Liu (2004)
Aesthetiscope is an interactive art installation whose wall of color
reacts to portray the relationship between some idea (a word, a
poem, a song) and a person (a realist, a dreamer, a neurotic) standing
before it. Each idea, for example the word sunset, is rich in association
for a person. Perhaps he remembers in his mind what a sunset looks
like. Or a sunset mades him think of other ideas like warmth, fuzzy,
beautiful, serenity, relaxation. Perhaps it reminds him of some
past event in his life. The contextual sphere of these personal
associations form the Aesthetic about the idea. And the experience
of that aesthetic is called its pathos. I
wanted to choose a medium through which pathos could be convincingly
portrayed, and so I chose colors because they are a complete microconsciousness
of pathos, like taste and smell.
Aesthetic is hard to articulate because it is usually experienced
it as an undeconstructed gestalt. Any analysis of Aesthetic needs
to be sensitive to its complexity -- the multi-dimensional nature
of connotation. The aesthetiscope analyzes each idea through a multi-perspectival
linguistic analysis of connotation. The realms of analysis are "Think,"
"Culturalize," "See," "Intuit," and
"Feel." Each of these realms brings to bear a different
perspectival vocabulary to the pathos description of an
idea. "Think" generates rational connotations and entailments
of the idea. "Culturalize" looks at the cultural entailments
of the idea through the lens of a particular culture. "See"
takes the idea as a source of imagery, bringing to bear our collective
visual memory of objects, places, and events. "Intuit"
is an exercise in automatic free assocations with the idea as a
cue. "Feel" takes a sentimental stance toward the idea,
connecting it to a word of feelings. The results of these analyses
are mapped to a world of colors through psycho-physiocological color
surveys based on the work of Berlin & Kay, and Goethe, and naturalistic
sampling of colors from photos.
these different vocabularies of aesthetic, we can try to make sense
of a "sunset." A sunset may be "Seen," revealing
the dark purple swatches with splashes of warm hues that characterize
the visual rememberance of a sunset. But there is also an inner
sunset. A sunset "Felt" and "Intuited" recalls
warmth, beauty, and serenity, and these will bring about brighter,
warmer, and more intense colors than the outer sunset.
aesthetiscope encourages us to experience and reflect on Aesthetic
in a new way.
Pattie Maes (2005) The Aesthetiscope: Visualizing Aesthetic Readings
of Text in Color Space. Proceedings of 18th International FLAIRS
Conference Special Track on AI in Music and Art, May 15-17, 2005,
Clearwater Beach, FL, USA. AAAI Press.
MOV, 1m05s no sound) AESTHETISCOPE, VISUALIZING THE LYRICS TO "IMAGINE"
BY JOHN LENNON.
MOV, 4m22s no sound) A Movie Demonstrating the Aesthetiscope.
Play Programming with English
Hugo Liu and Henry Lieberman (2004)
who are programmers will tell you that the abstractions of computation
are a tool chest of metaphors for understanding processes of all
kinds. Unfortunately programming is not really taught until high
school and college. The reasons for this are not good ones. The
syntax and semantics of various programming languages tend to difficult
to learn and to gain intuition about. But what if we could program
in plain English? This is the dream of programming in natural language.
Metafor, we are just beginning to scratch the surface of this dream.
While full and complete understanding of language is a concern of
"story understanding" and is often regarded as AI-complete,
we are making the assumption that natural language can be constrained
to simple constructions and inputs, without harming the intuitiveness
of input in natural language. The major insight gained by Metafor
is that natural language IS programmatic in nature. Thus, we simply
need to transliterate this language into a traditional programming
traditional programming languages are beginning just now to be dynamically
typed, natural language was always dynamically typed, by part-of-speech,
syntax, semantics, pragmatics, and common sense. In Metafor, if
you declared that "there is a bin of apples," it is automatically
understood that bins can hold things and is a container object like
a list. English as a programming language is also much more concise
than any traditional programming language. If i said, "Look
in the bin and pick out just the red apples," that's the equivalent
of programming: "map(Pick, filter(lambda apple: apple.color
== red, bin.getApples()))."
doesn't construct whole programs. It builds just the scaffolding
for a program. What visualizing scaffolding does is to help beginner
programmers and non-programmers intuit programming. Perhaps young
children wh play with Metafor can glean programmatic insights that
will fill their tool chests with new metaphors for problem solving.
That's our hope.
the Same Language", Technology Review, July 2005.
Metaphor for Software Engineering", Optimize Magazine,
Programming ", MIT Technology Insider Magazine, April
2005, p. 3.
to Code Converter", Slashdot, 25 March 2005.
turns English into Code Outline" by Kimberly Patch, Technology
Research News, 14 March 2005.
MOV, 1m40s, no sound) A Movie Demonstrating the Metafor Interpreter.
Liu and Henry Lieberman (2005) Programmatic Semantics for Natural
Language Interfaces. Proceedings of the ACM Conference on Human
Factors in Computing Systems, CHI 2005, April 5-7, 2005, Portland,
OR, USA. ACM Press.
Liu and Henry Lieberman (2005) Metafor: Visualizing Stories
as Code. Proceedings of the ACM International Conference on
Intelligent User Interfaces, IUI 2005, January 9-12, 2005,
San Diego, CA, USA, to appear. ACM 2005.
Liu and Henry Lieberman (2004) Toward a Programmatic Semantics
of Natural Language. Proceedings of VL/HCC'04: the 20th IEEE
Symposium on Visual Languages and Human-Centric Computing. pp.
281-282. September 26-29, 2004, Rome. IEEE Computer Society Press.
Lieberman and Hugo Liu (forthcoming).
Feasibility Studies for Programming in Natural Language. H. Lieberman,
F. Paterno, and V. Wulf (Eds.) Perspectives in End-User Development,
to appear. Kluwer. Late, Late 2004.
Would They Think?
Hugo Liu, Pattie Maes (2004)
to improving at any task is frequent feedback from people whose
opinions we care about: our family, friends, mentors, and the experts.
However, such input is not usually available from the right people
at the time it is needed most, and attaining a deep understanding
of someone else’s perspective requires immense effort. This
work introduces a technological solution. We have developed a novel
method for automatically modeling a person’s attitudes and
opinions, and a proactive interface called “What Would They
Think?” which offers the just-in-time perspectives of people
whose opinions we care about, based on whatever the user happens
to be reading or writing. In the application, each person is represented
by a “digital persona,” generated from an automated
analysis of personal texts (e.g. weblogs and papers written by the
person being modeled) using natural language processing and commonsense-based
textual-affect sensing. In user studies, participants using our
application were able to grasp the personalities and opinions of
a panel of strangers more quickly and deeply than with either of
two baseline methods. This research has exciting theoretical and
pragmatic implications to intelligent user interfaces.
you are a researcher and would like to collaborate with us to further
this project, please send me email.
and Pattie Maes (2004). What Would They Think? A Computational Model
of Attitudes. Proceedings
of the ACM International Conference on Intelligent User Interfaces,
IUI 2004, January 13–16, 2004, Madeira, Funchal, Portugal.
ACM 2004, ISBN 1-58113-815-6, pp. 38-45.
Lexicon: Reinventing the Cognitive Lexicon
Hugo Liu (2003)
knowledge representation tradition in computational lexicon design
represents words as static encapsulations of purely lexical knowledge.
We suggest that this view poses certain limitations on the ability
of the lexicon to generate nuance-laden and context-sensitive meanings,
because word boundaries are obstructive, and the impact of non-lexical
knowledge on meaning is unaccounted for. Hoping to address these
problematics, we explore a context-centered approach to lexicon
design called a Bubble Lexicon. Inspired by Ross Quillian’s
Semantic Memory System, we represent word-concepts as nodes on a
symbolic-connectionist network. In a Bubble Lexicon, a word’s
meaning is defined by a dynamically grown context-sensitive bubble;
thus giving a more natural account of systematic polysemy. Linguistic
assembly tasks such as attribute attachment are made context-sensitive,
and the incorporation of general world knowledge improves generative
capability. Indicative trials over an implementation of the Bubble
Lexicon lends support to our hypothesis that unpacking meaning from
predefined word structures is a step toward a more natural handling
of context in language.
Liu. (2003). Unpacking meaning from words: A context-centered
approach to computational lexicon design. In Blackburn et al.
(Eds.): Modeling and Using Context, 4th International and Interdisciplinary
Conference, CONTEXT 2003, Stanford, CA, USA, June 23-25, 2003, Proceedings.
Lecture Notes in Computer Science 2680 Springer 2003, ISBN
3-540-40380-9, pp. 218-232.
A Practical Commonsense Reasoning Toolkit
(formerly known as OMCSNet)
Hugo Liu and Push Singh (2003)
is a freely available semantic network presently consisting of over
250,000 elements of "common sense" knowledge, that is
to say, knowledge that us humans have about the everyday physical
and social world; simple facts like "lemons are sour,"
"a dog is a pet," and that "the effect of eating
food is not being hungry anymore." People can't get along without
"common sense" knowledge, and so naturally we think smart
sociable computers won't be able to get along without it either.
formerly known as OMCSNet, has its roots in the Open
Mind Common Sense Project. The core knowledge of ConceptNet
is continually mined out of the English sentences collected by Open
Mind Common Sense, and put into a simpler form that computers will
understand. For example, the sentence "lemons are sour"
is simplified to the machine-understandable relation, PropertyOf(lemon,sour).
In the semantic network, "lemon" and "sour"
are concept nodes, and "PropertyOf" is a directed edge
connecting the two concepts.
philosophy behind ConceptNet is simple richness. Rich, because ConceptNet
includes a wide range of commonsense concepts and relations, much
like the Cyc Project. Simple, because
we embrace a lightweight easy-to-use graph representation, much
At the end of the day, we want ConceptNet to be simply useful to
AI Researchers who want to experiment with adding commonsense to
make their smart robots and programs smarter. And it's working!
ConceptNet is currently driving tens of new innovative research
projects at MIT and elsewhere!
The ConceptNet Flash Browser! Fun! Try it now!
Liu and Push Singh. (2004). ConceptNet: A Practical Commonsense
Reasoning Toolkit. BT Technology Journal 22(4). pp. 211-226.
Kluwer Academic Publishers.
Liu and Push Singh. (2004).
Commonsense Reasoning in and over Natural Language. Proceedings
of the 8th International Conference on Knowledge-Based Intelligent
Information & Engineering Systems (KES'2004). Wellington,
New Zealand. September 22-24. Lecture Notes in Artificial Intelligence,
Singh, Barbara Barry, and Hugo Liu.
(2004). Teaching Machines about Everyday Life. BT Technology
Journal 22(4). pp. 227-240. Kluwer Academic Publishers.
Liu, Ted Selker, Barbara Wheaton, Abraham Evans-EL (2003)
Intelligence successes have often been in solving specific problems
such as configuring computer systems or playing chess. In the search
for general uses of reasoning and learning systems we took cooking
as a window into culture. The Food Oracle is a set of tools we are
creating that weave learning and reasoning about cooking and food
together into a resource to help people explore creative and intuitive
cooking. Using partially structured knowledge mined from the Internet,
we have constructed a number of prototype explorations for various
kinds of thinking about food. We weave together disparate knowledge
mined from the Web and a culinary expert’s database into a
single rich resource. Our prototype tools reason intelligently about
recipe-sensitive ingredient substitutions, about the “essences”
of recipes, and over a database of food and culture. Our goal is
to create a smart recipe system that will foster users to think
creatively and critically about food as they explore adaptive and
dynamically generated recipes laden with a wealth of practical and
historical information about food.
long term vision of the Food Oracle is an effort to ground ideas
about food in ideas about culture. To this extent, understanding
recipes, food, and the historical and social contexts of food can
help advance our understanding of cultures more generally. In the
commonsense thinking space, food is dually 1) a rich domain in which
a "common sense" is present and felt (e.g. via recipe
"procedures" like poach, braise, e.g. via implicit assumptions
made by recipes), and 2) a domain whose semantics are finite, manageable,
and more precisely defined than in the everyday world domain. To
this end, we believe that recipe understanding can be a good way
to investigate representations and mechanisms for commonsense thinking.
Presentation PDF. Hugo Liu. (2002). A Talk Entitled "The
Web of Finding Food: Browse, Search, Modify & Eat". Given
at the SIG Counter Intelligence Meeting of the MIT Media Lab. October
Liu and Ted Selker (2002). A White paper on the Food Oracle
Eye: Visualizing the Affective Structure of
Selker/Henry Lieberman (2003)
have a thematic structure to them, and the text of stories are often
well-annotated according to thematic structure, such as via chapters,
a table of contents, et cetera. However, at a deeper level
of meaning, stories can also be thought of as having many other
structures. Affective structure may be a significant dimension of
meaning because so much of interpretation is concerned with the
engagement of our emotions. A plot does not climax without an emotional
peak. Unlike thematic structure, affective structure is rarely made
explicit in a story's organization. More likely than not, it emerges
out of imagination and textual interpretation. Even though it is
implicit, we still use it as a socially shared organizer and shared
index in search and navigation, because we can expect that others
will glean similar affective structures (for there is great culturally-driven
commonality of human experience and emotion).
Poseidon's Eye does is to allow the computer to read through stories
for affective structure, and share this structure visually with
people, who may find this preliminary and coarse reading of emotional
subtext useful as an indexing and navigation tool. For example,
perhaps you may want to read a book with the same kind of emotional
rollercoaster as the book you've just finished reading. Or perhaps
you want to skip to the climax of the book. Or perhaps you are a
script writer and you want to compare the emotional developments
of your characters against those in another script. One university
English classroom is even using Poseidon's Eye to cultivate critical
analysis. The professor says: "Poseidon's Eye says this chapter
is angry. What did Tolstoy intend?"
Eye is backed by an emotion and sentiment analysis engine which
uses a novel method of commonsense-backed appraisal of events for
a more comprehensive textual affect sensing machinery. Summarization
of affective structure into increasingly larger chunks is accomplished
with a bayesian classifier trained on conditional mutual information.
the affective visualization I created for CHI'2003, I chose to represent
the emotional slices of a story as a visually sequenced color bar.
Left to right corresponds to story progression from beginning to
end. The colors code for Paul Ekman's six basic emotions of happy,
sad, angry, fearful, disgusted, surprised. Of course, this is just
one of the visualizations for Poseidon's Eye. Look for other incarnations
of Poseidon's Eye in the near future!
Red Riding Hood" Example Story, Screenshot
Liu, Ted Selker, Henry Lieberman (2003). Visualizing the Affective
Structure of a Text Document. Proceedings of the Conference
on Human Factors in Computing Systems, CHI 2003, April 5-10, 2003,
Ft. Lauderdale, FL, USA. ACM 2003, ISBN 1-58113-637-4, pp.
Ponens / EmpathyBuddy
Liu, Henry Lieberman/Ted Selker (2003)
William James noted that the recognition of emotion is intimately
related to traditions and culture. Based on the thesis that much
of our emotional attitudes are dictated by our culture's "common
sense" about everyday situations, this project uses large-scale
affective commonsense from Open Mind (~40,000 facts relating everyday
situations to emotions) to analyze the broad emotional qualities
of sentences. User evaluations from an experimental application
that gives email users automatic affective feedback via Chernov
faces show that our text analysis method is effective. Our approach
addresses many of the limitations of the existing approaches to
textual affect classification (keyword spotting, lexical affinity,
hand-crafted models, statistical NLP) by offering greater robustness,
and extensibility. With our approach, text can be classified on
the individual sentence-level. We believe that this allows for a
higher degree of interactivity in affective applications.
Ponens is being variously at the MIT Media Lab, at MIT Sloan, and
elsewhere. One project at the lab called SETAE
(Bryant, Picard, Cavallo) aims to use Emotus Poens for emotional
pattern matching to pair up teenage girls based on their shared
situations. Emotus Ponens' textual affect sensing won the 2003 ACM
Intelligent User Interfaces Conference Outstanding Paper award.
of the ACM IUI'2003 Outstanding Paper Award. Hugo
Liu, Henry Lieberman, Ted Selker. (2003). A Model of Textual
Affect Sensing using Real-World Knowledge. Proceedings of the
2003 International Conference on Intelligent User Interfaces, IUI
2003, January 12-15, 2003, Miami, FL, USA. ACM 2003, ISBN 1-58113-586-6,
Liu, Henry Lieberman (2002)
(Annotation and Retrieval Integration Agent) is a software agent
that acts as an assistant to a user writing email or Web pages.
As the user types a story, it does continuous retrieval and ranking
on a photo database. It can use descriptions in the story text to
semi-automatically annotate pictures based on how they are used.
Sense ARIA has improved ARIAs automatic annotation capabilities
through world-aware semantic understanding of the text; made photo
retrieval more robust by using a commonsense knowledge base, Open
Mind Commonsense, to make semantic connections between the story
text and annotations (e.g. connect "bride" and "wedding");
and enabled the acquisition of personal commonsense through text
(e.g. My sisters name is Mary.) that can then
be used to improve photo retrieval by enabling personalized semantic
with Common Sense is one of the first interactive applications developed
that uses broad commonsense knowledge about the world to visibly
improve its agent behavior. Commonsense ARIA was the topic of my
Movie Demonstrating ARIA with the "ken and mary's wedding"
Good Summary Paper from an HCI Stance. Henry
Lieberman and Hugo Liu (2002). Adaptive
Linking between Text and Photos Using Common Sense Reasoning. In
De Bra, Brusilovsky, Conejo (Eds.): Adaptive Hypermedia and
Adaptive Web-Based Systems, Second International Conference, AH
2002, Malaga, Spain, May 29-31, 2002, Proceedings. Lecture
Notes in Computer Science 2347 Springer 2002, ISBN 3-540-43737-1,
Meat and Potatoes of Commonsense-driven Conceptual Expansion and
the granddaddy of ConceptNet.
Liu and Henry Lieberman (2002). Robust photo retrieval using
world semantics. In Proceedings of the LREC 2002 Workshop on
Creating and Using Semantics for Information Retrieval and Filtering:
State-of-the-art and Future Research, Las Palmas, Canary Islands,
Masters Thesis Elaborates Everything (160pp).
(2002). Semantic Understanding and Commonsense Reasoning in an
Adaptive Photo Agent, Masters Thesis, Department of Electrical
Engineering and Computer Science, Massachusetts Institute of Technology,
Interactive Computer Story Generation
#3 (length = 9)
John watched TV
John fell asleep
John had a dream
John dreamt about his day
John imagined he met a girl
John went out with the girl
John had fun
John woke up
John tried to go back to sleep.
Story generated by MAKEBELIEVE. (For more stories, see "Media"
is a story generation agent that uses Open Mind knowledge to interactively
compose short fictional texts with a user. While a user must start
a story, MAKEBELIEVE will attempt to continue that story by freely
imagining possible sequences of events that might happen to the
character the user has chosen. The agent uses "commonsense"
about causality and how the world works, mined from the Open Mind
Common Sense corpus, and combines this with very simple lingustic
techniques for story generation to produce pithy but interesting
stories. MAKEBELIEVE also uses commonsense to evaluate and critique
a story it has written to catch logically inconsistent, incoherent
events and actions.
next steps we are working on include expanding stories for multiple
characters, and reworking the story generator as an educational
tool for kids. We hope to incorporate social commosense about interactions
between characters to imagine possible interaction scenarios. By
making MakeBelieve more of a game, we might be able to use this
as an educational opportunity to teach younger kids about consequences
and story writing.
Some Examples of Stories Generated by MAKEBELIEVE
Push Singh. (2002). MAKEBELIEVE: Using Commonsense to Generate Stories.
Proceedings of the Eighteenth National Conference on Artificial
Intelligence, AAAI 2002, July 28 - August 1, 2002, Edmonton, Alberta,
Canada. AAAI Press, 2002, pp. 957-958.
Liu. (2002). Talk on MAKEBELIEVE. Originally given at AAAI2002,
31 July 2002, Edmonton, Alberta, Canada
Web Search User Interfaces
Liu, Henry Lieberman/Ted Selker (2001)
search engine user may find searching the Web for information difficult
and frustrating because she may naturally express search goals rather
than the topic keywords search engines need. GOOSE (goal-oriented
search engine interface) is an adaptive search engine interface
that uses natural language processing to parse a users search
goal, and uses "common sense" reasoning to interpret this
goal, and reason from it an effective query.
example, if a user tells the search engine: "I want to find
other people who like old movies," GOOSE would reason that
old movies is a hobby that someone might have, and that people talk
about their hobbies on their webpage. GOOSE would construct a query
which targets the retrieval of someone's web page, as follows: +"my
homepage" +"interests" +"old movies".
we cannot be assured of the robustness of the commonsense inference,
in a substantial number of cases, GOOSE is more likely to satisfy
the user's original search goals than simple keywords or conventional
Control" Example, Screenshot
of the Best AI Paper Award. Hugo
Liu, Henry Lieberman, Ted Selker. (2002). GOOSE: A Goal-Oriented
Search Engine With Commonsense. In De Bra, Brusilovsky, Conejo
(Eds.): Adaptive Hypermedia and Adaptive Web-Based Systems, Second
International Conference, AH 2002, Malaga, Spain, May 29-31, 2002,
Proceedings. Lecture Notes in Computer Science 2347 Springer
2002, ISBN 3-540-43737-1, pp. 253-263.
Liu. (2002). Talk on GOOSE: Goal-Oriented Search. Originally given
at AH2002, 31 May 2002, Malaga, Spain.
/ MontyTagger: Commonsense-Informed Natural Language Understanding
Tools. (They are good. They are free to you.)
Hugo Liu (2001-2004)
been frustrated with the fact that natural language processing tools
on the web are in such a state of depravity. They are either written
in an outdated language like prolog, or they require "training,"
and of course, they never provide you with a version that just "works!"
out of the box. I was sick and tired of writing glue code to tie
together these various systems. So I decided to write my own natural
language understanding tools, end-to-end, de novo. And
I've made them freely available to the public. MontyTagger is an
implementation of Eric Brill's 1994 TBL part-of-speech tagger, written
natively for Python, and also available for Java. It's been available
since 2001. MontyLingua is a complete end-to-end natural language
processing tool which I released in 2003. It recognizes, tokenizes,
tags, chunks, pp-links, lemmatizes, and semantically interprets
any inputted paragraph of text, all in one step. Pass in some text,
and you are passed out the "meaning."
"Sorry I couldn't meet you on Saturday, but let's have lunch
Output: (action:'meet' agent:'I' patient:'you' when: 'on Saturday')
and (action:'have' agent:'we' patient:'lunch' when: 'on Sunday')
and MontyLingua are the most popular tools of their kind on the
web. They are very simple to use, and they are free. Since 2001,
MontyTagger has enjoyed over 1000 downloads
and 100 registered users. Since its release in late 2003, MontyLingua
has enjoyed over 300 downloads and
50 registered users. Please download and enjoy them. I am working
on a more commonsense-loaded version which should be quite nice!
If you use them in your academic research, I only ask that you cite
them as follows, until I publish a real paper on it. Stay tuned.
(2003). MontyLingua: Commonsense-Informed Natural Language Understanding
Tools. Available at: http://web.media.mit.edu/~hugo/montylingua/
a search for "MontyTagger" and "MontyLingua",
they appear in dozens of research projects and papers. If you are
a fan, please stay tuned for a citeable paper (and use the above
citation in the meantime).
Project Site (download it from here!)
about MontyTagger in Japanese!
Chef Recipe Recommendation Agent (2001)
CHEF is a popular cook-off show in Japan in which chefs have 60
minutes to prepare innovative dishes around the theme of a particular
main ingredient. With some resemblance, the Iron Chef Recipe Recommendation
Agent recommends recipes to the user based around "themes"
about the user's tastes and preferences and time constraints. The
system is an interactive recipe recommendation agent, implemented
in GRASS and Python. Given a user's answers to questions about his/her
mood, lifestyle, tastes, diet, and cooking constraints such as time,
guests, available ingredients, IRON CHEF is able to recommend an
appropriate and relevant set of recipes from the Berkeley SOAR archive
of 700,000 recipes. The GRASS module of our system is a backward-chaining
expert system that uses the user's answers to high-level questions
about the user's preferences and constraints to infer the quantitative
and qualitative properties of desirable and suitable recipes. These
conclusions are then fed into a Frame-Based reasoner, which evaluates
candidate recipe-frames in its database against the desired criteria,
and outputs the most well-suited recipes to the user. Iron Chef
also recommends wine pairings to accompany the food.
Ball Pinball Construction Toolkit (1999)
Ball is a pinball game that was written as part of a small team
project. The game is different from other pinball games because
the user can construct, save, and retrieve his/her own boards. There
are a variety of pieces and flippers which can be placed in any
orientation. The user can map keystrokes to each flipper, the ball
launcher (called the absorber) or active bouncer. In addition, the
pieces can be linked by Rube-Goldberg mechanisms. For example, you
can connect a flipper to a triangle such that when the triangle
is hit, the flipper will be activated. The user has the ability
to change the physics of the board and the elasticities of the pieces,
or even cooler. GizmoBall was an experiment in good GUI design.
The project was first modeled in a UML-clone language and then implemented
in modules. The Gizmo Ball Construction Set is a Java-based application
which runs on any platform equipped with Java 1.2 or Java 1.1.8
with Swing installed. Gizmo Ball takes up 178KB of storage to run
and has a minimum system requirement of 300 MHz for good performance.
Stock Tracker (1999)
Stock Tracker is a stock portfolio organizer that doesn't require
the user to run to websites to look up stocks. It has the ability
to organize the portfolios for everyone in the family, and makes
stocks a breeze! With the ability to organize, save, and retrieve
an arbitrary number of different portfolios, and the ability to
retrieve stock quotes automatically from the Internet, Folio Stock
Tracker is a very handy program for individual investors, or paper
investors who want to play for fun! The program automatically updates
stock quotes every minute or a user can manually update the quotes.
The total value of assets is displayed in the title bar. The program
even handles market indices such as the Hanseng, but these are not
calculated into the total assets. Folio Stock Tracker 1.0 requires
Java 1.1.6 or above and can run as either an application or an applet
over the Internet.
Load Balancing Web Server (1999)
the ACME Web site (fictitious) has been attracting heavier usage.
Like many other overloaded Web sites, ACME is interested in redesigning
their Web server system to be able to handle an increased number
of requests, while at the same time, improving end-user performance.
The further challenge is that the system must be designed for a
global scale, since many of ACME's Web users live abroad. IntelliDistributor
is an intelligent load distribution system that transparently redirects
clients to one of any number of globally positioned mirror (replica)
servers. IntelliDistributor uses an intelligent algorithm to optimize
each redirect for maximum client performance. Overall, the system
also improves global scalability, guarantees content availability,
and provides fault tolerance. The ideas involved in the solution
to the ACME problem may also have other applications.
Digital Repository 2005 (1999)
intelligent digital repository system was designed, with methods
for adding objects, maintaining data integrity and currency, servicing
requests, and deployment of the system. The repository's size is
on the order of 100 terabytes with an average object size of one
megabyte. Objects are stored in a large array of 100-150 disks and
accessed by a unique, 32-bit object identifier. The setup is replicated
at several sites around the world to provide fault tolerance. Sites
are easily synchronized with version numbers, and data integrity
is maintained through the use of MD5 checksums.
Noise Filtering Protocol (1999)
problem of unscrambling an image distorted by noise is presented.
Traditionally, unscrambling images require expert and case-based
manipulation. An Image Restoration Protocol gives a generalized
way to solve all problems of this type, namely, through the systematised
analysis and manipulation of the fft of the image. We apply these
techniques to a sample scenario and successfully descramble the
image. We draw conclusions about the intuitive and technical parts
of this problem.
Digital Piano (1998)
Electric Lego Lab Kits, a specialized datapath was designed and
built that functions as a musical instrument. The design includes
the schematics for the datapath, frequency index counter, and control
finite state machine (CFSM). This project demonstrated the concept
of reconfigurable computers using FPGAs.