hugo's research athenæum
The definitive resource and media library (movies, screenshots, papers, news)

[top]

IdentityMirror
Hugo Liu, Glorianna Davenport, and Pattie Maes (2005)

What if you could look in the mirror and see not just what you look like, but also who you are?

The Identity Mirror is an augmented evocative object. Looking into it, the viewer's face is painted over with identity keywords and interest keywords, sourced from a deep model of the viewer's identity. The identity model is computed automatically from a viewer's social network profile or webpage, using the InterestMap. For instance, the viewer specifies that he listens to "Kings of Convenience" and enjoys the fiction of Vladmir Nabakov, and using this, InterestMap situates the viewer within its multiple neighborhoods of taste. The keywords which paint over the viewer's face represent his context within taste-space.

By gazing into Identity Mirror, a viewer can glean his identity-situation. Is his hair out of place? Are one of his interests out of place? How do his facial features combine to compose a gestalt? How do his various interests come together to compose an identity or aesthetic gestalt?

Identity mirror reifies its metaphors in the workings of an ordinary mirror. When the viewer is distant from the object, a question mark is the only keyword painted over his face. As he approaches to a medium distance, larger font sized identity keywords such as "fitness buffs", "fashionistas", and "book lovers" identify him. Approaching further, his favorite book, film, and music genres are seen. Closer yet, his favorite authors, musicians, and auteurs are known, and finally, standing up close, the songs, movies, and book titles become visible.

In ongoing research, we're developing further two particular aspects. IdentityFixing and the Diderot Effect -- keywords are distributed between a hearth (keywords aesthetically co-consistent) and a periphery (outlier keywords seemingly out-of-place about a person); the hearth covers the face, the periphery covers the hair; the viewer can use his hands to adjust his hair -- he can dishevel those unwanted periperhal keywords, or accept them by packing them into his hair. A person with a strong degree of taste-coherence has ruly hair, whereas a postmodernist with scattered interests has unruly hair. The second aspect under development is temporality -- the image of the viewer will reflect the viewer in relation to the goings on of the world and of his life. Since the viewer has many facets, various facets can be teased out by biasing InterestMap with contemporaneous keywords of current worldly and lively goings-on.

Media

Screenshots galore!

(23MB MOV, 2minutes, with sound) NEW! identity mirror NEW!

(7MB MOV, 1minute, no sound) Older identity mirror, demonstration

Hugo Liu and Glorianna Davenport (in works) Self-reflexive performance: Dancing with the computed audience of culture. International Journal on Performance Arts and Digital Media.

Hugo Liu, Glorianna Davenport, Pattie Maes (forthcoming). Taste Fabrics and the Beauty of Homogeneity. For the Association of Information Systems SIG SEMIS Bulletin 2(?).

Hugo Liu, Pattie Maes, Glorianna Davenport (forthcoming, 2006). Unraveling the Taste Fabric of Social Networks. For the International Journal on Semantic Web and Information Systems 2(1). Hershey, PA: Idea Academic Publishers.

[top]

Synaesthetic Recipes: Articulating Cravings
Hugo Liu and Matthew Hockenberry (2005)

"What's for dinner?"

In this work, we explore a technological answer to that famous question. Few of us know with great certitude the exact food we crave, but instead, we stew on the question and explore the nature of our craving through imaginative descriptions: "I feel like something light, fresh, sophisticated, not too mushy -- something influenced by thai or indian ingredients, something aromatic." Synaesthetic Recipes is a visual search program which allows such imaginative textual descriptions, and uses these to drive recipe recommendations. In the backend, a database of 100,000 recipes are automatically annotated with common sense about food. An artificial intelligence robotic reader reads each recipe and based on tastes of the ingredients and the types of cooking procedures, predicts how a food will look, taste, and smell. We are translating recipes into the rich descriptive vernacular of how people naturally conceptualize their cravings for food.

Media

Screenshot of the Flash SWF Foraging Interface

(15MB MOV, 2m34s sound) Synethetic Recipes: foraging for food with the family, in tate-space

Hugo Liu & Matthew Hockenberry, and Ted Selker (2005). Synesthetic Recipes: foraging for food with the family, in taste-space. Proceedings of SIGGRAPH'2005, Los Angeles.

[top]

InterestMap, A Cultural Fabric of Identities & Interests
Hugo Liu and Pattie Maes (2004)

Over recent years, social network communities (e.g. inter alia, friendster, contact lists, weblog communities, newsgroups) have been steadily building up in the online world. There is now sufficient critical mass of such infrastructure to postulate things about identity and the Self, as reflected in the social fabric of the online world.

How might we sense identity from the online social fabric? Semiotician Jacques Lacan has argued that words and concepts carry meaning primarily by what they signify. Roland Barthes proposed that the particular mappings between signifier and signified originate in cultural systems of semiology. Such cultural systems of signs have grown in importance as mass consumption has replaced subjective culture as the dominant contemporary cultural paradigm. Because the signifying value of possessions associated with the Self serves an important social function in signalling identity, it is possible to view identity and the Self as a collection of consumption decisions (cf. Social Constructionist Theory of Identity).

InterestMap mines online social network communities to create a rich influence network of interests and subcultures. Some of the interests represented in the network are, inter alia, television and films, foods, geographies, music, sports, hobbies, activities, objects, and people. The strength of connections between interests are learned from the digestion of on the order of one hundred thousand user profiles from online social networks. For example, a person who likes X may also mention Y as an interest on her homepage. Identity and tastes emerge as patterns of intersection on InterestMap.

Currently, InterestMap contains 100,000 interest and subculture nodes, trained with over 100,000 thousand profiles. It is currently used as an interest engine, driving serendipitous recommendation and social introduction facilitation in the Ambient Semantics project, but it certainly has far broader implications.

We also see InterestMap as a novel kind of social recommendation mechanism. Whereas recommendation systems traditionally works within a single application domain of interest such as books or music, InterestMap represents users using a much larger vocabulary of interests (movies, music, television shows, foods, sports, hobbies, passions) and ways of describing people (identity labels e.g. "raver", geographical locations, etc). The result is a multi-dimensional model of a user that is portable across domains; it can be used by Amazon, or any other conceivable vendor. More importantly, it has the potential to instruct computers to understand and describe people as people do.

Media

Hugo Liu and Pattie Maes (2005) InterestMap: Harvesting Social Network Profiles for Recommendations. Proceedings of the Beyond Personalization 2005 Workshop, January 9, 2005, San Diego, CA, USA, to appear. ACM Press 2005.

Hugo Liu, Glorianna Davenport, Pattie Maes (forthcoming). Taste Fabrics and the Beauty of Homogeneity. For the Association of Information Systems SIG SEMIS Bulletin 2(?).

Hugo Liu, Pattie Maes, Glorianna Davenport (forthcoming, 2006). Unraveling the Taste Fabric of Social Networks. For the International Journal on Semantic Web and Information Systems 2(1). Hershey, PA: Idea Academic Publishers.

Screenshot of the Who Am I visualisation for InterestMap

(25MB MOV, 2m20s no sound) Movie demonstration of "the fabric of interests". Caption for Video: "As a person specifies aspects of her interests, she is implicitly situating her identity when an interest fabric. As a result of this situation, all the new interests which have become proximal assert their relevance to her kind of person"

[top]

Ambient Semantics
Hugo Liu, Assaf Feldman, Sajid Sadi, Emmanuel Munguia Tapia, and Pattie Maes (2004)

Today, massive amounts of digital information can be searched through. Information about friends (e.g. friendster, orkut, im buddy lists), about the everyday (e.g. weblogs, the news), and about virtually any book, movie, song, or topic (e.g. amazon, google). Rather than becoming evermore consumed with being online, we would like to fold the digital world back into our offline lives.

The ambient semantics project uses a simple wearble RFID device to track things you pick up and people you meet, and looks for opportunities to present you with useful just-in-time information at the moment of your curiosity. "What did my friends think of this book?" "What common interests and friends do i share with this person?" The system leverages coRelate, a large knowledge base about the relatedness of interests, to meaningfully connect people with people, and people with objects. By modeling a person's interests, the system becomes a personal serendipity assistant that we hope will enrich living.

Press Coverage

"Product Pointer" Technology Review, February 2005

"Digital Serendipity" MIT Technology Insider Magazine, November 2004, p. 3

[top]

The Aesthetiscope
Hugo Liu (2004)

The Aesthetiscope is an interactive art installation whose wall of color reacts to portray the relationship between some idea (a word, a poem, a song) and a person (a realist, a dreamer, a neurotic) standing before it. Each idea, for example the word sunset, is rich in association for a person. Perhaps he remembers in his mind what a sunset looks like. Or a sunset mades him think of other ideas like warmth, fuzzy, beautiful, serenity, relaxation. Perhaps it reminds him of some past event in his life. The contextual sphere of these personal associations form the Aesthetic about the idea. And the experience of that aesthetic is called its pathos. I wanted to choose a medium through which pathos could be convincingly portrayed, and so I chose colors because they are a complete microconsciousness of pathos, like taste and smell.

The Aesthetic is hard to articulate because it is usually experienced it as an undeconstructed gestalt. Any analysis of Aesthetic needs to be sensitive to its complexity -- the multi-dimensional nature of connotation. The aesthetiscope analyzes each idea through a multi-perspectival linguistic analysis of connotation. The realms of analysis are "Think," "Culturalize," "See," "Intuit," and "Feel." Each of these realms brings to bear a different perspectival vocabulary to the pathos description of an idea. "Think" generates rational connotations and entailments of the idea. "Culturalize" looks at the cultural entailments of the idea through the lens of a particular culture. "See" takes the idea as a source of imagery, bringing to bear our collective visual memory of objects, places, and events. "Intuit" is an exercise in automatic free assocations with the idea as a cue. "Feel" takes a sentimental stance toward the idea, connecting it to a word of feelings. The results of these analyses are mapped to a world of colors through psycho-physiocological color surveys based on the work of Berlin & Kay, and Goethe, and naturalistic sampling of colors from photos.

With these different vocabularies of aesthetic, we can try to make sense of a "sunset." A sunset may be "Seen," revealing the dark purple swatches with splashes of warm hues that characterize the visual rememberance of a sunset. But there is also an inner sunset. A sunset "Felt" and "Intuited" recalls warmth, beauty, and serenity, and these will bring about brighter, warmer, and more intense colors than the outer sunset.

The aesthetiscope encourages us to experience and reflect on Aesthetic in a new way.

Media

Hugo Liu and Pattie Maes (2005) The Aesthetiscope: Visualizing Aesthetic Readings of Text in Color Space. Proceedings of 18th International FLAIRS Conference Special Track on AI in Music and Art, May 15-17, 2005, Clearwater Beach, FL, USA. AAAI Press.

(5MB MOV, 1m05s no sound) AESTHETISCOPE, VISUALIZING THE LYRICS TO "IMAGINE" BY JOHN LENNON.

(16MB MOV, 4m22s no sound) A Movie Demonstrating the Aesthetiscope.

[top]

Metafor: Play Programming with English
Hugo Liu and Henry Lieberman (2004)

Those who are programmers will tell you that the abstractions of computation are a tool chest of metaphors for understanding processes of all kinds. Unfortunately programming is not really taught until high school and college. The reasons for this are not good ones. The syntax and semantics of various programming languages tend to difficult to learn and to gain intuition about. But what if we could program in plain English? This is the dream of programming in natural language.

With Metafor, we are just beginning to scratch the surface of this dream. While full and complete understanding of language is a concern of "story understanding" and is often regarded as AI-complete, we are making the assumption that natural language can be constrained to simple constructions and inputs, without harming the intuitiveness of input in natural language. The major insight gained by Metafor is that natural language IS programmatic in nature. Thus, we simply need to transliterate this language into a traditional programming language.

While traditional programming languages are beginning just now to be dynamically typed, natural language was always dynamically typed, by part-of-speech, syntax, semantics, pragmatics, and common sense. In Metafor, if you declared that "there is a bin of apples," it is automatically understood that bins can hold things and is a container object like a list. English as a programming language is also much more concise than any traditional programming language. If i said, "Look in the bin and pick out just the red apples," that's the equivalent of programming: "map(Pick, filter(lambda apple: apple.color == red, bin.getApples()))."

Metafor doesn't construct whole programs. It builds just the scaffolding for a program. What visualizing scaffolding does is to help beginner programmers and non-programmers intuit programming. Perhaps young children wh play with Metafor can glean programmatic insights that will fill their tool chests with new metaphors for problem solving. That's our hope.

Press Coverage

"Speaking the Same Language", Technology Review, July 2005.

"MIT's Metaphor for Software Engineering", Optimize Magazine, May 2005

"Natural Programming ", MIT Technology Insider Magazine, April 2005, p. 3.

"English to Code Converter", Slashdot, 25 March 2005.

"Tool turns English into Code Outline" by Kimberly Patch, Technology Research News, 14 March 2005.

Media

(15MB MOV, 1m40s, no sound) A Movie Demonstrating the Metafor Interpreter.

Hugo Liu and Henry Lieberman (2005) Programmatic Semantics for Natural Language Interfaces. Proceedings of the ACM Conference on Human Factors in Computing Systems, CHI 2005, April 5-7, 2005, Portland, OR, USA. ACM Press.

Hugo Liu and Henry Lieberman (2005) Metafor: Visualizing Stories as Code. Proceedings of the ACM International Conference on Intelligent User Interfaces, IUI 2005, January 9-12, 2005, San Diego, CA, USA, to appear. ACM 2005.

Hugo Liu and Henry Lieberman (2004) Toward a Programmatic Semantics of Natural Language. Proceedings of VL/HCC'04: the 20th IEEE Symposium on Visual Languages and Human-Centric Computing. pp. 281-282. September 26-29, 2004, Rome. IEEE Computer Society Press.

Henry Lieberman and Hugo Liu (forthcoming). Feasibility Studies for Programming in Natural Language. H. Lieberman, F. Paterno, and V. Wulf (Eds.) Perspectives in End-User Development, to appear. Kluwer. Late, Late 2004.

[top]

What Would They Think?
Hugo Liu, Pattie Maes (2004)

A key to improving at any task is frequent feedback from people whose opinions we care about: our family, friends, mentors, and the experts. However, such input is not usually available from the right people at the time it is needed most, and attaining a deep understanding of someone else’s perspective requires immense effort. This work introduces a technological solution. We have developed a novel method for automatically modeling a person’s attitudes and opinions, and a proactive interface called “What Would They Think?” which offers the just-in-time perspectives of people whose opinions we care about, based on whatever the user happens to be reading or writing. In the application, each person is represented by a “digital persona,” generated from an automated analysis of personal texts (e.g. weblogs and papers written by the person being modeled) using natural language processing and commonsense-based textual-affect sensing. In user studies, participants using our application were able to grasp the personalities and opinions of a panel of strangers more quickly and deeply than with either of two baseline methods. This research has exciting theoretical and pragmatic implications to intelligent user interfaces.

If you are a researcher and would like to collaborate with us to further this project, please send me email.

Media

Hugo Liu and Pattie Maes (2004). What Would They Think? A Computational Model of Attitudes. Proceedings of the ACM International Conference on Intelligent User Interfaces, IUI 2004, January 13–16, 2004, Madeira, Funchal, Portugal. ACM 2004, ISBN 1-58113-815-6, pp. 38-45.

[top]

Bubble Lexicon: Reinventing the Cognitive Lexicon
Hugo Liu (2003)

The knowledge representation tradition in computational lexicon design represents words as static encapsulations of purely lexical knowledge. We suggest that this view poses certain limitations on the ability of the lexicon to generate nuance-laden and context-sensitive meanings, because word boundaries are obstructive, and the impact of non-lexical knowledge on meaning is unaccounted for. Hoping to address these problematics, we explore a context-centered approach to lexicon design called a Bubble Lexicon. Inspired by Ross Quillian’s Semantic Memory System, we represent word-concepts as nodes on a symbolic-connectionist network. In a Bubble Lexicon, a word’s meaning is defined by a dynamically grown context-sensitive bubble; thus giving a more natural account of systematic polysemy. Linguistic assembly tasks such as attribute attachment are made context-sensitive, and the incorporation of general world knowledge improves generative capability. Indicative trials over an implementation of the Bubble Lexicon lends support to our hypothesis that unpacking meaning from predefined word structures is a step toward a more natural handling of context in language.

Media

Hugo Liu. (2003). Unpacking meaning from words: A context-centered approach to computational lexicon design. In Blackburn et al. (Eds.): Modeling and Using Context, 4th International and Interdisciplinary Conference, CONTEXT 2003, Stanford, CA, USA, June 23-25, 2003, Proceedings. Lecture Notes in Computer Science 2680 Springer 2003, ISBN 3-540-40380-9, pp. 218-232.

[top]

ConceptNet: A Practical Commonsense Reasoning Toolkit
(formerly known as OMCSNet)
Hugo Liu and Push Singh (2003)

ConceptNet is a freely available semantic network presently consisting of over 250,000 elements of "common sense" knowledge, that is to say, knowledge that us humans have about the everyday physical and social world; simple facts like "lemons are sour," "a dog is a pet," and that "the effect of eating food is not being hungry anymore." People can't get along without "common sense" knowledge, and so naturally we think smart sociable computers won't be able to get along without it either.

ConceptNet, formerly known as OMCSNet, has its roots in the Open Mind Common Sense Project. The core knowledge of ConceptNet is continually mined out of the English sentences collected by Open Mind Common Sense, and put into a simpler form that computers will understand. For example, the sentence "lemons are sour" is simplified to the machine-understandable relation, PropertyOf(lemon,sour). In the semantic network, "lemon" and "sour" are concept nodes, and "PropertyOf" is a directed edge connecting the two concepts.

Our philosophy behind ConceptNet is simple richness. Rich, because ConceptNet includes a wide range of commonsense concepts and relations, much like the Cyc Project. Simple, because we embrace a lightweight easy-to-use graph representation, much like WordNet. At the end of the day, we want ConceptNet to be simply useful to AI Researchers who want to experiment with adding commonsense to make their smart robots and programs smarter. And it's working! ConceptNet is currently driving tens of new innovative research projects at MIT and elsewhere!

Media

The ConceptNet Flash Browser! Fun! Try it now!

ConceptNet Website.

Hugo Liu and Push Singh. (2004). ConceptNet: A Practical Commonsense Reasoning Toolkit. BT Technology Journal 22(4). pp. 211-226. Kluwer Academic Publishers.

Hugo Liu and Push Singh. (2004). Commonsense Reasoning in and over Natural Language. Proceedings of the 8th International Conference on Knowledge-Based Intelligent Information & Engineering Systems (KES'2004). Wellington, New Zealand. September 22-24. Lecture Notes in Artificial Intelligence, Springer 2004.

Push Singh, Barbara Barry, and Hugo Liu. (2004). Teaching Machines about Everyday Life. BT Technology Journal 22(4). pp. 227-240. Kluwer Academic Publishers.

[top]

Hyper Recipes
Hugo Liu, Ted Selker, Barbara Wheaton, Abraham Evans-EL (2003)

Artificial Intelligence successes have often been in solving specific problems such as configuring computer systems or playing chess. In the search for general uses of reasoning and learning systems we took cooking as a window into culture. The Food Oracle is a set of tools we are creating that weave learning and reasoning about cooking and food together into a resource to help people explore creative and intuitive cooking. Using partially structured knowledge mined from the Internet, we have constructed a number of prototype explorations for various kinds of thinking about food. We weave together disparate knowledge mined from the Web and a culinary expert’s database into a single rich resource. Our prototype tools reason intelligently about recipe-sensitive ingredient substitutions, about the “essences” of recipes, and over a database of food and culture. Our goal is to create a smart recipe system that will foster users to think creatively and critically about food as they explore adaptive and dynamically generated recipes laden with a wealth of practical and historical information about food.

The long term vision of the Food Oracle is an effort to ground ideas about food in ideas about culture. To this extent, understanding recipes, food, and the historical and social contexts of food can help advance our understanding of cultures more generally. In the commonsense thinking space, food is dually 1) a rich domain in which a "common sense" is present and felt (e.g. via recipe "procedures" like poach, braise, e.g. via implicit assumptions made by recipes), and 2) a domain whose semantics are finite, manageable, and more precisely defined than in the everyday world domain. To this end, we believe that recipe understanding can be a good way to investigate representations and mechanisms for commonsense thinking.

Media

Powerpoint Presentation PDF. Hugo Liu. (2002). A Talk Entitled "The Web of Finding Food: Browse, Search, Modify & Eat". Given at the SIG Counter Intelligence Meeting of the MIT Media Lab. October 2002.

Hugo Liu and Ted Selker (2002). A White paper on the Food Oracle projects. Unpublished.

[top]

Poseidon's Eye: Visualizing the Affective Structure of Stories
Hugo Liu, Ted Selker/Henry Lieberman (2003)

Stories have a thematic structure to them, and the text of stories are often well-annotated according to thematic structure, such as via chapters, a table of contents, et cetera. However, at a deeper level of meaning, stories can also be thought of as having many other structures. Affective structure may be a significant dimension of meaning because so much of interpretation is concerned with the engagement of our emotions. A plot does not climax without an emotional peak. Unlike thematic structure, affective structure is rarely made explicit in a story's organization. More likely than not, it emerges out of imagination and textual interpretation. Even though it is implicit, we still use it as a socially shared organizer and shared index in search and navigation, because we can expect that others will glean similar affective structures (for there is great culturally-driven commonality of human experience and emotion).

What Poseidon's Eye does is to allow the computer to read through stories for affective structure, and share this structure visually with people, who may find this preliminary and coarse reading of emotional subtext useful as an indexing and navigation tool. For example, perhaps you may want to read a book with the same kind of emotional rollercoaster as the book you've just finished reading. Or perhaps you want to skip to the climax of the book. Or perhaps you are a script writer and you want to compare the emotional developments of your characters against those in another script. One university English classroom is even using Poseidon's Eye to cultivate critical analysis. The professor says: "Poseidon's Eye says this chapter is angry. What did Tolstoy intend?"

Poseidon's Eye is backed by an emotion and sentiment analysis engine which uses a novel method of commonsense-backed appraisal of events for a more comprehensive textual affect sensing machinery. Summarization of affective structure into increasingly larger chunks is accomplished with a bayesian classifier trained on conditional mutual information.

In the affective visualization I created for CHI'2003, I chose to represent the emotional slices of a story as a visually sequenced color bar. Left to right corresponds to story progression from beginning to end. The colors code for Paul Ekman's six basic emotions of happy, sad, angry, fearful, disgusted, surprised. Of course, this is just one of the visualizations for Poseidon's Eye. Look for other incarnations of Poseidon's Eye in the near future!

Media

"Little Red Riding Hood" Example Story, Screenshot

Hugo Liu, Ted Selker, Henry Lieberman (2003). Visualizing the Affective Structure of a Text Document. Proceedings of the Conference on Human Factors in Computing Systems, CHI 2003, April 5-10, 2003, Ft. Lauderdale, FL, USA. ACM 2003, ISBN 1-58113-637-4, pp. 740-741.

[top]

Emotus Ponens / EmpathyBuddy
Hugo Liu, Henry Lieberman/Ted Selker (2003)

Psychologist William James noted that the recognition of emotion is intimately related to traditions and culture. Based on the thesis that much of our emotional attitudes are dictated by our culture's "common sense" about everyday situations, this project uses large-scale affective commonsense from Open Mind (~40,000 facts relating everyday situations to emotions) to analyze the broad emotional qualities of sentences. User evaluations from an experimental application that gives email users automatic affective feedback via Chernov faces show that our text analysis method is effective. Our approach addresses many of the limitations of the existing approaches to textual affect classification (keyword spotting, lexical affinity, hand-crafted models, statistical NLP) by offering greater robustness, and extensibility. With our approach, text can be classified on the individual sentence-level. We believe that this allows for a higher degree of interactivity in affective applications.

Emotus Ponens is being variously at the MIT Media Lab, at MIT Sloan, and elsewhere. One project at the lab called SETAE (Bryant, Picard, Cavallo) aims to use Emotus Poens for emotional pattern matching to pair up teenage girls based on their shared situations. Emotus Ponens' textual affect sensing won the 2003 ACM Intelligent User Interfaces Conference Outstanding Paper award.

Media

A screenshot of EmpathyBuddy.

Recipient of the ACM IUI'2003 Outstanding Paper Award. Hugo Liu, Henry Lieberman, Ted Selker. (2003). A Model of Textual Affect Sensing using Real-World Knowledge. Proceedings of the 2003 International Conference on Intelligent User Interfaces, IUI 2003, January 12-15, 2003, Miami, FL, USA. ACM 2003, ISBN 1-58113-586-6, pp. 125-132

[top]

Common Sense ARIA
Hugo Liu, Henry Lieberman (2002)

ARIA (Annotation and Retrieval Integration Agent) is a software agent that acts as an assistant to a user writing email or Web pages. As the user types a story, it does continuous retrieval and ranking on a photo database. It can use descriptions in the story text to semi-automatically annotate pictures based on how they are used.

Common Sense ARIA has improved ARIA’s automatic annotation capabilities through world-aware semantic understanding of the text; made photo retrieval more robust by using a commonsense knowledge base, Open Mind Commonsense, to make semantic connections between the story text and annotations (e.g. connect "bride" and "wedding"); and enabled the acquisition of personal commonsense through text (e.g. “My sister’s name is Mary.”) that can then be used to improve photo retrieval by enabling personalized semantic connections.

ARIA with Common Sense is one of the first interactive applications developed that uses broad commonsense knowledge about the world to visibly improve its agent behavior. Commonsense ARIA was the topic of my master's thesis.

Media

A Movie Demonstrating ARIA with the "ken and mary's wedding" story.

A Good Summary Paper from an HCI Stance. Henry Lieberman and Hugo Liu (2002). Adaptive Linking between Text and Photos Using Common Sense Reasoning. In De Bra, Brusilovsky, Conejo (Eds.): Adaptive Hypermedia and Adaptive Web-Based Systems, Second International Conference, AH 2002, Malaga, Spain, May 29-31, 2002, Proceedings. Lecture Notes in Computer Science 2347 Springer 2002, ISBN 3-540-43737-1, pp. 2-11.

The Meat and Potatoes of Commonsense-driven Conceptual Expansion and the granddaddy of ConceptNet. Hugo Liu and Henry Lieberman (2002). Robust photo retrieval using world semantics. In Proceedings of the LREC 2002 Workshop on Creating and Using Semantics for Information Retrieval and Filtering: State-of-the-art and Future Research, Las Palmas, Canary Islands, pp. 15-20.

My Masters Thesis Elaborates Everything (160pp). Hugo Liu, (2002). Semantic Understanding and Commonsense Reasoning in an Adaptive Photo Agent, Masters Thesis, Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA.

[top]

MakeBelieve: Interactive Computer Story Generation
Hugo Liu (2001)

Story #3 (length = 9)
John watched TV
John fell asleep
John had a dream
John dreamt about his day
John imagined he met a girl
John went out with the girl
John had fun
John woke up
John tried to go back to sleep.

- Story generated by MAKEBELIEVE. (For more stories, see "Media" below)

MAKEBELIEVE is a story generation agent that uses Open Mind knowledge to interactively compose short fictional texts with a user. While a user must start a story, MAKEBELIEVE will attempt to continue that story by freely imagining possible sequences of events that might happen to the character the user has chosen. The agent uses "commonsense" about causality and how the world works, mined from the Open Mind Common Sense corpus, and combines this with very simple lingustic techniques for story generation to produce pithy but interesting stories. MAKEBELIEVE also uses commonsense to evaluate and critique a story it has written to catch logically inconsistent, incoherent events and actions.

Some next steps we are working on include expanding stories for multiple characters, and reworking the story generator as an educational tool for kids. We hope to incorporate social commosense about interactions between characters to imagine possible interaction scenarios. By making MakeBelieve more of a game, we might be able to use this as an educational opportunity to teach younger kids about consequences and story writing.

Media

Some Examples of Stories Generated by MAKEBELIEVE

Hugo Liu, Push Singh. (2002). MAKEBELIEVE: Using Commonsense to Generate Stories. Proceedings of the Eighteenth National Conference on Artificial Intelligence, AAAI 2002, July 28 - August 1, 2002, Edmonton, Alberta, Canada. AAAI Press, 2002, pp. 957-958.

Powerpoint Presentation PDF. Hugo Liu. (2002). Talk on MAKEBELIEVE. Originally given at AAAI2002, 31 July 2002, Edmonton, Alberta, Canada

[top]

Goal-Oriented Web Search User Interfaces
Hugo Liu, Henry Lieberman/Ted Selker (2001)

A novice search engine user may find searching the Web for information difficult and frustrating because she may naturally express search goals rather than the topic keywords search engines need. GOOSE (goal-oriented search engine interface) is an adaptive search engine interface that uses natural language processing to parse a user’s search goal, and uses "common sense" reasoning to interpret this goal, and reason from it an effective query.

For example, if a user tells the search engine: "I want to find other people who like old movies," GOOSE would reason that old movies is a hobby that someone might have, and that people talk about their hobbies on their webpage. GOOSE would construct a query which targets the retrieval of someone's web page, as follows: +"my homepage" +"interests" +"old movies".

While we cannot be assured of the robustness of the commonsense inference, in a substantial number of cases, GOOSE is more likely to satisfy the user's original search goals than simple keywords or conventional query.

Media

"Pest Control" Example, Screenshot

"Movies" Example, Screenshot

Recipient of the Best AI Paper Award. Hugo Liu, Henry Lieberman, Ted Selker. (2002). GOOSE: A Goal-Oriented Search Engine With Commonsense. In De Bra, Brusilovsky, Conejo (Eds.): Adaptive Hypermedia and Adaptive Web-Based Systems, Second International Conference, AH 2002, Malaga, Spain, May 29-31, 2002, Proceedings. Lecture Notes in Computer Science 2347 Springer 2002, ISBN 3-540-43737-1, pp. 253-263.

Powerpoint Presentation PDF. Hugo Liu. (2002). Talk on GOOSE: Goal-Oriented Search. Originally given at AH2002, 31 May 2002, Malaga, Spain.

[top]

MontyLingua / MontyTagger: Commonsense-Informed Natural Language Understanding Tools. (They are good. They are free to you.)
Hugo Liu (2001-2004)

I have been frustrated with the fact that natural language processing tools on the web are in such a state of depravity. They are either written in an outdated language like prolog, or they require "training," and of course, they never provide you with a version that just "works!" out of the box. I was sick and tired of writing glue code to tie together these various systems. So I decided to write my own natural language understanding tools, end-to-end, de novo. And I've made them freely available to the public. MontyTagger is an implementation of Eric Brill's 1994 TBL part-of-speech tagger, written natively for Python, and also available for Java. It's been available since 2001. MontyLingua is a complete end-to-end natural language processing tool which I released in 2003. It recognizes, tokenizes, tags, chunks, pp-links, lemmatizes, and semantically interprets any inputted paragraph of text, all in one step. Pass in some text, and you are passed out the "meaning."

For example:

Input: "Sorry I couldn't meet you on Saturday, but let's have lunch on Sunday"
Output: (action:'meet' agent:'I' patient:'you' when: 'on Saturday') and (action:'have' agent:'we' patient:'lunch' when: 'on Sunday')

MontyTagger and MontyLingua are the most popular tools of their kind on the web. They are very simple to use, and they are free. Since 2001, MontyTagger has enjoyed over 1000 downloads and 100 registered users. Since its release in late 2003, MontyLingua has enjoyed over 300 downloads and 50 registered users. Please download and enjoy them. I am working on a more commonsense-loaded version which should be quite nice! If you use them in your academic research, I only ask that you cite them as follows, until I publish a real paper on it. Stay tuned.

Hugo Liu . (2003). MontyLingua: Commonsense-Informed Natural Language Understanding Tools. Available at: http://web.media.mit.edu/~hugo/montylingua/

Doing a search for "MontyTagger" and "MontyLingua", they appear in dozens of research projects and papers. If you are a fan, please stay tuned for a citeable paper (and use the above citation in the meantime).

Downloads and Fun

MontyTagger Download Site

MontyLingua Project Site (download it from here!)

Wiki about MontyTagger in Japanese!

Undergraduate Research Projects

[top]

Iron Chef Recipe Recommendation Agent (2001)

IRON CHEF is a popular cook-off show in Japan in which chefs have 60 minutes to prepare innovative dishes around the theme of a particular main ingredient. With some resemblance, the Iron Chef Recipe Recommendation Agent recommends recipes to the user based around "themes" about the user's tastes and preferences and time constraints. The system is an interactive recipe recommendation agent, implemented in GRASS and Python. Given a user's answers to questions about his/her mood, lifestyle, tastes, diet, and cooking constraints such as time, guests, available ingredients, IRON CHEF is able to recommend an appropriate and relevant set of recipes from the Berkeley SOAR archive of 700,000 recipes. The GRASS module of our system is a backward-chaining expert system that uses the user's answers to high-level questions about the user's preferences and constraints to infer the quantitative and qualitative properties of desirable and suitable recipes. These conclusions are then fed into a Frame-Based reasoner, which evaluates candidate recipe-frames in its database against the desired criteria, and outputs the most well-suited recipes to the user. Iron Chef also recommends wine pairings to accompany the food.

[top]

Gizmo Ball Pinball Construction Toolkit (1999)

Gizmo Ball is a pinball game that was written as part of a small team project. The game is different from other pinball games because the user can construct, save, and retrieve his/her own boards. There are a variety of pieces and flippers which can be placed in any orientation. The user can map keystrokes to each flipper, the ball launcher (called the absorber) or active bouncer. In addition, the pieces can be linked by Rube-Goldberg mechanisms. For example, you can connect a flipper to a triangle such that when the triangle is hit, the flipper will be activated. The user has the ability to change the physics of the board and the elasticities of the pieces, or even cooler. GizmoBall was an experiment in good GUI design. The project was first modeled in a UML-clone language and then implemented in modules. The Gizmo Ball Construction Set is a Java-based application which runs on any platform equipped with Java 1.2 or Java 1.1.8 with Swing installed. Gizmo Ball takes up 178KB of storage to run and has a minimum system requirement of 300 MHz for good performance.

[top]

Folio Stock Tracker (1999)

Folio Stock Tracker is a stock portfolio organizer that doesn't require the user to run to websites to look up stocks. It has the ability to organize the portfolios for everyone in the family, and makes stocks a breeze! With the ability to organize, save, and retrieve an arbitrary number of different portfolios, and the ability to retrieve stock quotes automatically from the Internet, Folio Stock Tracker is a very handy program for individual investors, or paper investors who want to play for fun! The program automatically updates stock quotes every minute or a user can manually update the quotes. The total value of assets is displayed in the title bar. The program even handles market indices such as the Hanseng, but these are not calculated into the total assets. Folio Stock Tracker 1.0 requires Java 1.1.6 or above and can run as either an application or an applet over the Internet.

[top]

IntelliDistributor Load Balancing Web Server (1999)

Recently, the ACME Web site (fictitious) has been attracting heavier usage. Like many other overloaded Web sites, ACME is interested in redesigning their Web server system to be able to handle an increased number of requests, while at the same time, improving end-user performance. The further challenge is that the system must be designed for a global scale, since many of ACME's Web users live abroad. IntelliDistributor is an intelligent load distribution system that transparently redirects clients to one of any number of globally positioned mirror (replica) servers. IntelliDistributor uses an intelligent algorithm to optimize each redirect for maximum client performance. Overall, the system also improves global scalability, guarantees content availability, and provides fault tolerance. The ideas involved in the solution to the ACME problem may also have other applications.

[top]

Intelligent Digital Repository 2005 (1999)

An intelligent digital repository system was designed, with methods for adding objects, maintaining data integrity and currency, servicing requests, and deployment of the system. The repository's size is on the order of 100 terabytes with an average object size of one megabyte. Objects are stored in a large array of 100-150 disks and accessed by a unique, 32-bit object identifier. The setup is replicated at several sites around the world to provide fault tolerance. Sites are easily synchronized with version numbers, and data integrity is maintained through the use of MD5 checksums.

[top]

Image Noise Filtering Protocol (1999)

The problem of unscrambling an image distorted by noise is presented. Traditionally, unscrambling images require expert and case-based manipulation. An Image Restoration Protocol gives a generalized way to solve all problems of this type, namely, through the systematised analysis and manipulation of the fft of the image. We apply these techniques to a sample scenario and successfully descramble the image. We draw conclusions about the intuitive and technical parts of this problem.

[top]

FPGA Digital Piano (1998)

Using Electric Lego Lab Kits, a specialized datapath was designed and built that functions as a musical instrument. The design includes the schematics for the datapath, frequency index counter, and control finite state machine (CFSM). This project demonstrated the concept of reconfigurable computers using FPGAs.