Home | News | Join Us | People | Projects | Publications | Talks | Courses
Home | News | Join Us | People | Projects | Publications | Talks | Courses
ACM SIGGRAPH Asia 2010. Transactions on Graphics 29(6)-163.
SIGGRAPH Asia 2010 Paper Frequently Asked Questions
 |
|
Figure 1: The HR3D display is brighter and higher resolution than traditional glasses-free 3D displays that use parallax barriers. (Left) The input to our 3D display is a series of views of 3D content, shown here with horizontal head movement. (Middle) A conventional parallax barrier display depicts 3D content by placing a pinhole array mask over an LCD panel (see below). However, such displays are dimmer (due to the pinhole array most light is blocked) and have a lower resolution, since the number of pixels is reduced to the number of pinholes. (Right) The HR3D display achieves the full resolution of the underlying LCD and increases the image brightness, relative to parallax barrier displays. Such improvements are possible due to: (1) dual-stacked LCD hardware and (2) a new method to compute the best pair of patterns to display on such dual-stacked LCDs. The best pair of patterns are given by the solution of a mathematical optimization problem that exploits the fact that the set of views (or the "lightfield") has a low algebraic rank for natural 3D content. |
animation credit: MIT Media Lab, Camera Culture Group |
We have developed a 3D display called High-Rank 3D (HR3D). Our display does not require any special glasses. Unlike traditional stereoscopic displays (e.g., those in movie theaters) that use only two views, HR3D generates multi-view, walk-around imagery.
Existing glasses-free 3D displays use a simple concept known as a parallax barrier that uses a front mask (containing an array of pinholes) and a rear LCD panel. Our work uses two LCD panels. We present a mathematical insight that allows the HR3D display to have significantly higher resolution, to be about three to five times brighter, and to have a higher frame rate than existing parallax barrier displays. We present the details below in the Brief Technical Description section. Multiple views of a 3D scene can be represented by a "light field"; our mathematical insight regards a concept known as algebraic "rank" and specifically the rank of a light field: (1) two-layer displays always produce a rank-1 light field and (2) the light field of a 3D scene can be accurately represented by a low-rank representation.
Next-generation 3D displays must present high-fidelity imagery, but must do so what being power conscious; one cannot simply increase the brightness of LCD backlights, otherwise battery life and power efficiency will be severely reduced. To this end, HR3D enhances the 3D viewing experience via improved brightness and resolution without increasing power consumption.
Please read the FAQ for more information.
|
||
|
(Left) Photo of our Dual LCD Display (Center) 3D image in Pinhole mode (Right) 3D image in HR3D mode. |
||
Douglas Lanman, Matthew Hirsch, Yunhee Kim, Ramesh Raskar. Content-adaptive parallax barriers: optimizing dual-layer 3D displays using low-rank light field factorization. Proc. of SIGGRAPH Asia 2010 (ACM Transactions on Graphics 29, 6), 2010.
|
|
|
Figure 2: The HR3D display uses a dual-LCD display and determines the optimal set of front and rear LCD images by using Non-negative Matrix Factorization (NMF), thereby increasing resolution and brightness in the perceived images of the 3D content compared to parallax barriers. |
|
| Illustration Credit: MIT Media Lab, Camera Culture Group |
 |
|
Figure 3: Achieving 3D display using parallax barriers. (Left) A parallax barrier 3D display ensures that a viewer perceives different image, depending on his viewpoint, by placing a front barrier (i.e., a mask) in front of an LCD display. As shown in the lower magnified image, this front barrier consists of a uniform array of pinholes. (Right) The rear LCD panel displays a set of interlaced images, corresponding to neighboring views of the 3D scene. Depending on where a viewer is standing, only a subset of pixels of the rear LCD image will be visible through the pinhole array; thus, a parallax barrier functions by masking rear LCD pixels using a pinhole array and the set of visible pixels depends on viewer position. |
| Illustration Credit: MIT Media Lab, Camera Culture Group |
 |
|
Figure 4: Achieving 3D display using High-Rank 3D (HR3D). Unlike with parallax barriers, the HR3D display requires a dual-stacked LCD, consisting of a conventional LCD panel covered by a second, transparent LCD. Also unlike with parallax barriers, the front panel is not assigned a specialized role (e.g., a pinhole array). Instead, a mathematical optimization problem is solved to determine the best pair of images to display on each LCD panel to recreate the desired 3D scene. The front and rear HR3D LCD images are shown on the left and right, respectively. Note that the front image passes significantly more light than a pinhole array, thereby increasing the brightness compared to conventional parallax barriers. |
| Illustration Credit: MIT Media Lab, Camera Culture Group |
@article{
Lanman_et_al_2010,
author = {Lanman, Douglas and Hirsch,Matthew and Kim, Yunhee and Raskar, Ramesh},
title = {Content-adaptive parallax barriers: optimizing dual-layer 3D displays using low-rank
light field factorization},
journal = {ACM Trans. Graph.},
volume ={29},
number = {6},
year = {2010},
issn = {0730-0301},
pages = {163:1--163:10},
doi = {http://dx.doi.org/10.1145/1882261.1866164},
publisher = {ACM},
address = {New York, NY, USA}
}
Today's 3D display are not only light deficient, but also "rank deficient". This basic fact results in displays which are too dim, too low resolution, or both. We have developed a 3D display that eliminates the need for special glasses, while pointing a way towards a solution for both light and rank deficiency. Until now, the commercial potential of glasses-free 3D displays, particularly those based on liquid crystal displays (LCDs) and parallax barriers, has been primarily limited by decreased image resolution and brightness compared to systems employing special eyewear.
In the Camera Culture group at the MIT Media Lab, we have found a way to increase the brightness and resolution of LCD-based, glasses-free 3D displays using a method we call Content-Adaptive Parallax Barriers. We call our new display technology High-Rank 3D or HR3D, since our display is capable of displaying a full-resolution light field.
 |
|
Figure 5: Increasing display brightness and refresh rate. The HR3D display, using content-adaptive barriers, is compared to time-shifted barriers with the exposure normalized so the relative image brightness is consistent with observation. The input 3x3 light field is compressed by our display for T<9, where T is the number of time-multiplexed sub-frames to be shown (i.e. the rank of the displayed light field). Experimental photographs (fourth column) are compared to predicted images. All images correspond to the central view of the light field (looking straight at the screen) for the light field shown in Figure 6. |
| Photo Credit: MIT Media Lab, Camera Culture Group |
Current commercially-available, LCD-based 3D displays use a simple concept, first proposed by Frederic Ives in 1903; Ives achieved the illusion of depth by introducing the notion of a parallax barrier. In his design an array of slits is placed slightly in front of a normal 2D display. The slits ensure that each eye sees different regions of the underlying display and, therefore, different images. This method is still widely-employed today, despite its limitations. In particular, the slits required by Ives's parallax barriers function by blocking rays of light. As a result, traditional barriers significantly reduce the resolution and brightness of the underlying display. Our analysis shows that this reduction in brightness and resolution is due to the low rank of the light field produced by this technique.
Content-adaptive parallax barriers improve upon Ives's method by allowing the spacing and orientation of slits to be optimized to transmit as much light as possible, while retaining or improving the fidelity of the projected 3D images. The key insight is to realize that the pattern of slits must be changed depending on the 3D scene being projected. By eliminating the earlier fixed, heuristic pinhole or slit design, content-adaptive parallax barriers can significantly increase the brightness of glasses-free 3D displays based on LCD technology. Furthermore, by using recent high-speed LCDs, we have shown that full-resolution 3D images can be achieved. Content-adaptive parallax barriers are well-suited to mobile devices, optimizing the brightness of displays without reducing battery life.
Thin displays that present an illusion of depth have become a driving force in the electronics and entertainment industries. Binocular depth cues are achieved by presenting different images to each eye. Current-generation 3D displays require special eyewear, such as LCD shutters or polarizing filters. glasses can be eliminated by modifying the optical design of existing displays. Historically, glasses-free 3D displays require adding optically-attenuating masks, known as parallax barriers, or refracting lens arrays to a 2D display. We optimize the performance of 3D displays built by stacking a pair of modified LCD panels. We introduce adaptive masks, optimized for each multi-view video frame that increase brightness and frame rate compared to conventional parallax barriers. These patterns can extend the battery life of next-generation mobile 3D displays and enable full motion parallax—allowing a viewer to tilt his head and still perceive the illusion of depth.
Read our PDF Overview handout.
|
|
Figure 6: Dual-stacked LCDs are considered as general spatial light modulators that act in concert to recreate a target light field by attenuating rays emitted by the backlight. (Left) A conventional parallax barrier configuration for glasses-free 3D display and (Right) a content-adaptive parallax barrier display that allows increased image resolution and brightness.
|
| Illustration Credit: MIT Media Lab, Camera Culture Group |
|
|
Figure 7: Rank constraints for parallax barriers. (Left) Conventional parallax barriers approximate the light field matrix (center) as the outer product of mask vectors (above and to the left). The resulting rank-1 approximation accurately reproduces the circled elements (corresponding to the central views in Figure 4). Note that most columns are not reconstructed, reducing display resolution and brightness. Periodic replicas of the central views are created outside the circled regions. (Middle Left) Time-shifted parallax barriers achieve higher-rank reconstructions by integrating a series of rank-1 approximations, each created by a single translated mask pair. (Middle Right) Content-adaptive parallax barriers increase display brightness by allowing both masks to exhibit non-binary opacities. Here a rank-1 approximation is demonstrated using a single mask pair. (Right) Rank-T approximations are achieved using temporal multiplexing of T content-adaptive parallax barriers. In practice, the light field will be full rank without enforcing periodic replication (as created by conventional parallax barriers). As a result, we do not constrain rays (shown in red) outside the central view in our optimization to find the masks. |
| Illustration Credit: MIT Media Lab, Camera Culture Group |
|
|
Figure 8: HR3D display with content-adaptive parallax barriers. We show that 3D displays can adapt to the content they are displaying, making them brighter, higher resolution, and higher frame-rate. Glasses-free 3D displays, which are actually 4D light field displays, can be analyzed as a matrix approximation problem. This insight leads to new, content-adaptive parallax barriers, applicable to one common type of glasses free 3D display known as a barrier display. (Left, Top) A 4D light field, represented as a 2D array of 2D images. (Left, Bottom) A dual-stacked LCD displays the light field using our content-adaptive parallax barriers, with both vertical and horizontal parallax. (Middle and Right) A pair of our content-adaptive parallax barriers. In a rank-9 decomposition of the 4D light field matrix, nine such patterns would be shown for a single video frame. Compared to conventional parallax barriers (slits or pinholes), content adaptation allows increased display brightness and refresh rate while preserving the fidelity of projected images. |
| Photo Credit: MIT Media Lab, Camera Culture Group |
Our prototype HR3D display relies on an optimization technique to arrive at the optimal set of images and masks to display for a desired output light field. The content creator decides what image quality, brightness, and framerate to target, and the optimization will try to generate a set of mask pairs to achieve this result. Because there is no way to produce negative light, the masks generated by our optimization must be non-negative. Therefore, we use a Non-negative Matrix Factorization (or NMF) in our optimization.
|
|
Figure 9: Approximation error as a function of NMF iteration. The average PSNR of the reconstruction is plotted for a rank-9 decomposition of the light field shown in Figure 6. |
| Illustration Credit: MIT Media Lab, Camera Culture Group |
One of the primary benefits of our work is the ability to make an automultiscopic barrier display with full horizontal and vertical parallax brighter. The HR3D display can trade brightness for image fidelity better than pinholes. Here we characterize that trade.
|
|
Figure 10: Approximation error as a function of gain in brightness. The average PSNR of the reconstruction is plotted for a rank-9 decomposition of the light fields shown in Figure 6. For time-shifted parallax barriers, transmission can be increased either by enlarging slits/pinholes or by brightening the rear LCD. The latter is considered here, however simulations of the former also confirm time-shifted parallax barriers cannot achieve a PSNR greater than 15 dB when increasing brightness by a factor greater than two. |
| Illustration Credit: MIT Media Lab, Camera Culture Group |
Another benefit of the HR3D display is that content-adaptive barriers allow the rank of the displayed light field to vary. This could be achieved with a scanned pinhole by simply removing some of the view positions, but the effect of this has more serious consequences for reconstruction PSNR than our reduced rank reconstruction. Here, we characterize the how reconstruction PSNR changes with decomposition rank.
|
|
Figure 11: Approximation error as a function of decomposition rank. The average PSNR of the reconstruction is plotted for a rank-T decomposition of the light fields in Figure 6. For 3×3 views, a theoretical PSNR of infinity is achieved with 9 time-shifted conventional parallax barriers. In comparison, content-adaptive barriers achieve higher PSNR than conventional barriers when fewer frames are used. Experimental and predicted images with varying degrees of compression are shown in Figure 2. |
| Illustration Credit: MIT Media Lab, Camera Culture Group |
Existing 3D displays come in many varieties. Displays are broadly classified as either glasses-bound, those which rely on the viewer wearing a pair of special glasses, or unencumbered, those which allow the viewer to walk up and perceive a 3D image without any extra equipment.
Glasses-bound displays tend to also be stereoscopic, meaning that they provide only two distinct views – one for each eye. Unencumbered displays, on the other hand, tend to be multiscopic, meaning that many images are displayed at once. (Think of stereo- vs. multi-channel audio). A stereo 3D display will produce strange, non-physical distortion when the viewer moves his head from side to side. In contrast, a multiview, or multiscopic display, allows the viewer to look around objects when moving his head, in closer agreement to the way we see the real world. The figure below, taken from our course, Build Your Own 3D Display, shows how many popular technologies fit into this taxonomy.
|
|
Figure 12: A taxonomy of 3D display devices, taken from our course, Build Your Own 3D Display. |
| Illustration Credit: Matthew Hirsch, Douglas Lanman. BYO3D Course, SIGGRAPH 2010 |
Please see our course, Build Your Own 3D Display for further details about the current state-of-the-art in 3D display. In particular, check out the first half of Section 2, Representation and Display, in which we expand upon the above taxonomy.
The HR3D system is an auto-multiscopic display. It provides many views of a scene, without requiring the viewer to wear special glasses or head tracking equipment. The table below compares the HR3D design to the other popular auto-multiscopic display technologies, along axes that are apparent when viewing the display, as well as cost.
| Technology | Approach | Brightness | Refresh Rate | Spatial Resolution | True Horizontal Parallax | True Vertical Parallax | Cost |
| Stereoscopic (Image Pairs) | Glasses with polarizers, LCDs, or color filters | Medium (glasses usually attenuate a portion of the light) | High | High | No | No | Low |
| Parallax Barriers | Vertical Slits (e.g., Nintendo 3DS) |
Medium | High | Low | Yes | No | Low |
| Pinhole Array | Low | High | Lowest | Yes | Yes | Low | |
| Scanned Pinhole Array | Low | Low | High | Yes | Yes | High | |
| Integral Imaging | Lenticular Sheets | High | High | Low | Yes | No | Medium |
| Fly's Eye Lens Arrays | High | High | Lowest | Yes | Yes | Medium | |
| HR3D | Content-Adaptive Parallax Barriers | Medium (Can be traded) | Medium (Can be traded) | High (Can be traded) | Yes | Yes | High |
The HR3D display is unique in that it allows the content creator to trade between brightness, refresh rate, and image resolution to suit the content being displayed. This is because, unlike any other 3D display, the HR3D display uses parallax barriers that adapt to the content on the screen. The adaptation can be tuned to favor brightness, image quality, or refresh rate depending on viewing conditions, spatial locations of viewers, and the interest of the content creator.
To build our prototype HR3D display, we disassembled two Viewsonic VX2265wm 120Hz LCD panels. The front panel was completely disassembled, removing both the front diffusing polarizing layer and rear transparent polarizing layer. The images below show the sequence of steps to disassemble the display. The most difficult step is the final step of removing the polarizing films from the LCD glass. After carefully pulling up the films, we used a pencil eraser and acetone to remove the adhesive.
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
We presented our technical paper talk on this project at SIGGRAPH Asia 2010 in Seoul, South Korea, on Friday, 17 December 2010, 4:15PM. The talk was in the Imaging Hardware section. HR3D: Content Adaptive Parallax Barriers
For more technical details, see our SIGGRAPH Asia 2010 submission video.
We thank the SIGGRAPH Asia 2010 technical paper reviewers for insightful feedback, the Camera Culture and Information Ecology groups at the MIT Media Lab for their support, and Samsung Electronics for its sponsorship. Szymon Jakubczak contributed to an earlier version of this work presented at SIGGRAPH 2010. Thomas Baran and Gabriel Taubin contributed useful discussions. Douglas Lanman is supported by NSF Grant CCF-0729126, Yunhee Kim by NRF of Korea Grant 2009-352-D00232, and Ramesh Raskar by an Alfred P. Sloan Research Fellowship.
Douglas Lanman, Postdoctoral Associate, MIT Media Lab
dlanman (at) media.mit.edu
Alexandra Kahn, Senior Press Liaison, MIT Media Lab
akahn (at) media.mit.edu or 617/253.0365
|
|
|
|
|
| Cameras |
6D Display Lighting and Viewpoint aware displays |
Bokode Long Distance Barcodes |
BiDi Screen Touch+3D Hover on Thin LCD |
NETRA Cellphone based Eye Test |
Home | News | Join Us | People | Projects | Publications | Talks | Courses