Many mobile users listen to music in order to free their minds in the constrained space and time. However, the isolated auditory bubbles make them become further disconnected from the world. How can we use sound as the medium to connect the mobile user to the environment? How can the linking experience be smooth, easy, yet personal? Can the system still be effective on an extremely dense audio map? How can we enhance the simultaneous listening experience based on context?
Loco-Radio is an AR auditory environment for car drivers, bikers, and pedestrians. The mobile user is able to tune in and listen to the surroundings. The system augments nearby places with localized audio streams and immerses the user within a soundscape as he/she moves. It works indoor and outdoor, and is designed at street and building scale. The research explores context-aware strategies of using spatial audio in high-density audio environments.
The outdoor project demonstrated how AR audio could connect mobile users to a large, open urban environment. Now we attempt to transfer the AR auditory experience to an indoor environment and design the experience at building scale. The goal of Loco-Radio Indoor is to realize an AR auditory tour at the MIT Media Lab.
Please see the figure below. The baseball cap is capable of tracking the head direction. The user can press rewind/forward buttons on the line control to zoom in/out. The middle button allows the user to lock in the closest audio source. Holding the button will mute everything except the closest sound source. As users walk around the lab, they hear demos and talks by students and faculty.
Indoor Positioning System
Loco-Radio Indoor retrieves indoor location data from Compass Badge, a geomagnetic based location sensing module developed by Chung (2012). It utilizes ambient magnetic field as a reference to track location and head direction. The major components of the system include (a) the location badge, which contains a 2x2 array of magnetic sensors, (b) the magnetic fingerprint database, and (c) the localization processor based on particle filter algorithm.
A database of magnetic fingerprint was created, which covers a large area on the 3rd floor of MIT Media Lab. The fingerprint is collected for every 0.5 meter of floor, as shown below.
Loco-Radio Outdoor creates an AR auditory experience that enhances the mobile users' awareness of the nearby restaurants. The environment immerses them within a soundscape. As users move on the street, they encounter a series of songs and the perception enhances their awareness of the numbers, styles, and locations of restaurants. The experience helps them construct a mental food map.
The Audio Map
A compact audio map is created by associating genre-matching songs to restaurants. The place information is collected by using Google Place and Yelp API. The database covers 392 restaurants in Cambridge/Somerville MA, and each is attached with a genre-matching song. The below figure shows the restaurants around MIT. Most of them locate near Central Square. In order to overcome the uneven distribution of sound streams, auditory scaling is necessary.
Auditory Spatial Scaling
Auditory scale defines the relations between sound and space; it describes how sounds are heard by mobile users in augmented space. Scaling alters the relations and can be used to transform the auditory experience. Various techniques were introduced: automatic and manual zooming, asymmetric scaling, and stereoized crossfading. They allow designers to create effective UI for AR audio environments with a large amount of simultaneous streams.
The above figure visualizes the automatic zooming process. Each circle represents the effective audible range at the location. The scale is dynamically adjusted so that the user is not overwhelmed by a large number of simultaneous streams.
The AR audio is more fragmented when moving at a faster speed. Therefore, the hearing range needs to be increased with a fast moving speed. In addition, the motion of car and bicycle are more predictable. Therefore, we can play farther along the direction of the user's motion.
Loco-Radio Outdoor was evaluated by 10 subjects (drivers, bikers, pedestrians) in a think aloud study. The study showed that since the user tended to confirm the location of sound visually, it created double impression and allowed the user to remember the place well. Biking was rated the best experience among three modes of mobility. A bicycle ride happened at a moderate speed, so it was less affected by the latency and inaccuracy of GPS. When the user spent less effort in perceiving the audio, he could better blend himself into the environment, and that led to a smooth and more connected user experience.