Impressionist visualization

of online communication

To see the animations, please download vsoc_notext.exe (4MB), vsoc_text.exe (4MB), as well as AVIfiles.exe (22MB) to the same directory. Run AVIfiles.exe, which will extract the digital video files. Finally, you can run vsoc_notext.exe, which is just the animation of the floating video streams, or vsoc_text.exe, which is the same plus the actual textual messages on top of the video streams.
Note that a fast PC is necessary to display the animation properly!


Intuitive, non-textual representation of discussion content

    All textual information which each participant has written, is transformed to a continuous stream of video. The semantic content of the text messages is mapped onto a sequence of videos and pictures. The mapping is realized on the side of the receiver, because a simple text line like "I love cats" means different things to different people. Some would associate this with an ad for cat food, some other would be more negative because they dislike the mentality of cats and would therefore see pictures like a dog chasing a cat. For this purpose, each participant has a personal database of semantic descriptions of pictures and videos. If the participant scans the messages of a group, this textual information is transformed automatically to a user specific multiple stream of video. These video snippets do have mainly connotative meanings, not denotative! And because it is a triadic relation, the video stream generated from a certain collection of email messages is not visualized in the same manner for different users. It might even change over time for a single user, because her/his own personal database of semantic descriptions changes over time too. An example: If I move from Switzerland to Boston, this change of environment will certainly influence my visual connotations to words like "MIT," "Boston," etc. (How to update the database is a not yet solved problem.)

    The background to this idea is the theory behind the semantic differential (mainly by Osgood and Ertel) as well as the semiotic ecological approach of my Psychology Professor at the University of Bern, Alfred Lang.

So if we first look at an ongoing discussion, it might look like that:


Each picture (actually, video stream) represents the transformed collection of already written email messages in the form of a video clip. Of course not only the actual video pictures contribute to the connotative impact of the stream, but also the length of each scene as well as the combination of clips and pictures, means, post production and editing rules apply as well.

This is a single discussion or thread. But as mentioned above, we can participate in more than one group, as long as real time presence is not required. For this purpose, we have a remote control which allows us to zap between discussions like channels on TV. There is an up and down button, and we zap until we see something interesting, until a combination of video streams catches our interest. We can also pre-program our remote control for certain channels, which means, bookmarks to certain threads, and jump directly to a specific group.

On each "channel" = discussion, we can rewind and replay the discussion, which means, previous email messages are transformed to video streams. We have a complete non-linear access to the history of the thread. For this purpose, we have the equivalent buttons of a VCR: play (real time), fast rewind, fast forward (only if we are not already in real time). We also have a counter which shows us where we are, like the potential controls of a time machine:


One of these buttons allows us to jump to the beginning of the discussion, another one let us jump to real time. With the fast forward and fast rewind buttons, we can skim all parts of the discussion backwards and forwards.

The positions of the video windows on the screen are dynamic: they float on the desktop according to which person has replied to which other person recently. The closer two persons are, the more they have replied to or cited each other. This leads to an additional grouping of person representations.


There is also the possibility of layering. The video stream of the most active person is on top, overlaying those of people which have posted messages less recently. Additionally, the video of the person with the most recent contribution at all has a green border.

By clicking on a video, the actual email message pops up, depending on where in the time continuum we are located. Underneath the text boxes, the connotational video interpretations of the email content goes on.



  • One of my initial ideas, the 3x2 meter big wall, is already realized by IBM.
  • Difference between connotative and denotative: A nuclear explosion may be described connotationally as a bright thing, but denotationally it is a dark event, because it means a negative act, a relapse into a dark "barbarian" age.
  • Lang proposes a psychological interpretation of semiotic processes. Each semiosis consists of  a reference, interpretation, and representation; more precise in German: Referenz, Interpretanz, and Repräsentanz. Important is that every relation is triadic, not dyadic! This means in our context: There is not only a sender and a receiver, but also an instance between them which acts as an interpreter. If we look at the connotation of an email message, each one means something completely different to different persons, depending on the third thing, the "Interpretanz". It is important to mention that the components of this sort of semiosis are not predefined object classes, but roles in a relational process!

  • Unfortunately, as far as I know, there are no English translations available for Lang's semiotic ecological approach.
    • Lang, Alfred [1992b] Kultur als 'externe Seele' - eine semiotisch-ökologische Perspektive, in: Christian Allesch et al. (eds) Psychologische Aspekte des kulturellen Wandels, Verlag des Verbandes der wissenschaftlichen Gesellschaften Österreichs, pp. 11-32.
    • Lang, Alfred [1993] Eine Semiotik für die Psychologie - Eine Psychologie für die Semiotik, in: Leo Montada (ed), Bericht über den 38. Kongreß der Deutschen Gesellschaft für Psychologie in Trier 1992, Bd.2, Hogrefe Verlag, Göttingen et al., pp. 664 - 673.
    • Lang, Alfred [1993b] Zeichen nach innen, Zeichen nach außen - eine semiotisch-ökologische Psychologie als Kulturwissenschaft, in: Peter Rusterholz/ Maja Svilar (eds) Welt der zeichen - Welt der Wirklichkeit, Paul Haupt Verlag, Bern, pp. 355-85.
    • Lang, Alfred [1993c] Non-Cartesian Artefacts in Dwelling Activities: Steps towards a Semiotic Ecology, in: Schweizerische Zeitschrift für Psychologie, Vol.2, Verlag Hans Huber, Bern - Stuttgart - Toronto, pp. 148-151.
  • Osgood, C. E., May, W. H., & Miron, M. S. (1975). Cross-cultural universals of affective meaning.  Urbana, IL: University of Illinois Press.

Send me some comments! Stefan Marti Last updated May 17 1998.