Multilingual Computing, assignment 4:
Create a visual language for interaction with customs and immigration officials.

Since I was not familiar with the domain of visual languages, I did an extensive web search on work that could be relevant to this problem (Picture languages / Vedo-Vedi, Bliss, Writing Systems). The amount of work done in this sector is impressive, and since I don't want to invent something that already exists, I will first "dissect" the actual assignment a little.

The assignment is not completely clear. There are two kinds of possible solutions, depending on the following distinction: Is the interaction between the foreigner and the customs officials computer mediated or not?

  • If a computer can be involved in this interaction, the main workload would be on creating some kind of automatic translation system.
  • If not, the involved people will have to learn some kind of language, be it visual or natural. (That's how it is currently.)

I personally think that the final solution should be computer mediated automatic speech translation. However, we are far from having this technology. Some people might doubt that it is possible to do that at all, and even if it could be realized, then not for all languages in our world.

But let's focus on a novel universal visual language. If the computer is not mediating the interaction, the involved people would have to learn some kind of visual language. This sounds to me like we are forced to learn a kind of visual Esperanto, just in order to interact with the customs. I don't think this will happen, not only because it would be an artificial language, specifically created for this rare kind of interaction. I also think that the customs officials are not willing to learn such a novel universal language for the following reason: Although part of the assignment was that the language shouldn't be culturally biased, I don't think that this restriction is relevant for customs officials. In the contrary: For the customs, a person that enters the country has to adapt to the local customs (sic!) and laws, or at least be aware of them. My impression is that the customs do not intend to make that task easier for foreigners. Foreigners that speak the local language are preferred, since by learning a language, one learns probably also about the local customs and laws.

In general, I think the customs situation is not a typical example. E.g., the European Community has basically eliminated it. There are no customs or immigration officers within Europe anymore. However, not all Europeans speak the same language, so a more realistic scenario would probably be buying groceries; communicate in a bar; communicate with police.

Back to the visual language: After an extensive web search, I come to the conclusion that the current systems that I have seen (Bliss, MartinGo, Elephant, etc.) are too complex. Even the fascinating Elephant's Memory is very hard to decipher. Nobody would learn such a system just for the rare occasion of customs, unless it is intuitive, or it can be learned on the fly. One option would be a language based on self-explaining icons. However, the author says that this is very hard, and I am not convinced if it is possible at all.

Part of the problem is that iconic and pictorial languages are on a higher abstraction level than, e.g., illustrations and pictures. In my understanding, there is a continuum from picture, to pictogram, to icon, to symbol, to alphanumeric symbol, to Morse code, etc.: from real to abstract. Here's a more detailed description of this continuum:

The closer the symbol is to a picture (real life situation),
    the bigger the alphabet will be to express all meaning (sum of all symbols),
        the more intuitive/less artificial the symbol will be,
            the faster it can be understood (learned?)
                the more culturally biased it will be.

The more abstract a symbol is (e.g., Morse code),
    the smaller the alphabet will be,
        the less intuitive the symbols will be,
            the longer it takes to learn to understand,
                the more universal/culturally unbiased the content can be.

The following is an example of the item Banana on this continuum, from the photographic picture to Morse code:


What it is

Size of alphabet

Abstraction / Cost of learning

Photographic picture

Indefinitely big: virtually unlimited amount of pictures are possible

Very low


Big: a lot of drawings can be made


Label (icon like)

Probably 1000 different kinds of labels for bananas



Roman letter word

26 letters


-... .- -. .- -. .-

Morse code

3 (long, short, pause)



Having this continuum in mind, the most promising option I see right now is based on two elements: a set of pictures (a real picture book, or pictures on computer), combined with hand and facial gestures (or perhaps touch screen).

The interaction between foreigner and customs would be based on a set of pictures, which are relevant to the domain of customs, and good-old-fashioned gestures.

Examples for set of pictures:

Important is that the dictionary has to be ordered thematically, not alphabetically!


Current situation at customs and immigration

People intending to visit the USA have to do two things: fill out forms, and undergo a personal interview (at least one). They have to fill out the forms beforehand, in the airplane (and are getting yelled at by customs officials if they don't!!). The forms are partially multiple-choice forms about where they come from, how long they intend to stay, where they will stay, as well as all items that they will import to the States. These forms are issued in many different languages. The lower portion of at least one form is a request for help: they ask for suggestions of how to reduce the paperwork. (That's what this assignment is actually about, right?) Once arrived at the airport, the foreigners are interviewed personally, one by one, and basically asked the same questions again. An important part of this interview is probably psychologically motivated: I have gotten the impression that the officials are trying to check if the foreigners have filled out their forms correctly. They do not try to talk to the foreigners in their native language (even if they could...)—in the contrary, they mumble and use slang expressions to make the conversation difficult for people with limited knowledge of English.


Scenario: computer assisted picture book with touch screen (draft)

First half of the immigration procedure would consist of running through a multiple-choice questionnaire program on screen (like the current forms). Second half, a customs person would verify some form entries, getting additional information through this face-to-face interaction. If we assume that the customs people indeed want to communicate with people that do not understand English, they could use the computer based picture book to ask questions like: "Did you bring meat or fruit products?" S/he could do that by simply pointing at photos of meat and fruit. Advantages of retrieving pictures from computer, online, would be: the gallery could be updated easily and centrally, e.g., if new items are added that are prohibited to be imported.

Although this kind of interaction seems to be not very advanced compared to other possible scenarios, it is still an improvement over the current situation. (However, I am not sure if the customs and immigration officials intend to make communication in the first place.) Nevertheless, as I have mentioned before, I think that a realtime speech translation technology would be the best option, but also not very likely to be commercially available in the near future. A new universal visual language is probably not practical for the given sitution of customs and immigration.


Other issues and comments:

  • What about the grammar of picture book interactions?
  • Design new language/font: dimensions:
    2D: left-right is time, up-down is good-bad
    3D: what are the dimensions here?


Interesting links: