Prototype angle sensitive pixel camera (left). The data recorded by the camera prototype can be processed to recover a high-resolution 4D light field (center). As seen in the close-ups on the right, parallax is recovered from a single camera image.

Abstract

We propose a flexible light field camera architecture that is at the convergence of optics, sensor electronics, and applied mathematics. Through the co-design of a sensor that comprises tailored, Angle Sensitive Pixels and advanced reconstruction algorithms, we show that—contrary to light field cameras today—our system can use the same measurements captured in a single sensor image to recover either a high-resolution 2D image, a low-resolution 4D light field using fast, linear processing, or a high-resolution light field using sparsity-constrained optimization.

Video

Slides

Note: Slideshare does not convert animations properly. I recommend downloading the slides from the slideshare site for the full experience!

Files

Citation

M. Hirsch, S. Sivaramakrishnan, S. Jayasuriya, A. Wang, A. Molnar, R. Raskar, G. Wetzstein. A Switchable Light Field Camera Architecture with Angle Sensitive Pixels and Dictionary-based Sparse Coding. Computational Photography (ICCP), 2014 IEEE International Conference on. IEEE, 2014.

Bibtex

@inproceedings{hirsch2014switchable,
  title={{A} {S}witchable {L}ight {F}ield {C}amera {A}rchitecture with {A}ngle {S}ensitive {P}ixels and {D}ictionary-based {S}parse {C}oding},
  author={Hirsch, Matthew and Sivaramakrishnan, Sriram and Jayasuriya, Suren and Wang, Albert and Molnar, Alyosha and Raskar, Ramesh and Wetzstein, Gordon},
  booktitle={Computational Photography (ICCP), 2014 IEEE International Conference on},
  pages={1--10},
  year={2014},
  organization={IEEE}
}

Additional Information

Schematic of a single angle sensitive pixel. Two interleaved photodiodes capture a projection of the light field incident on the sensor (left). The angular responses of these diodes are complementary: a conventional 2D image can be synthesized by summing their measurements digitally (right).

Illustration of ASP sensor layout (left) and sampled spatio-angular frequencies (right). The pictured sensor interleaves three different types of ASPs. Together, they sample all frequencies contained in the dashed green box (right). A variety of light field reconstruction algorithms can be applied to these measurements, as described in the text.

Microscopic image of a single 6 × 4 pixel tile of the ASP sensor (left). We also show captured angular point spread functions (PSFs) of each ASP pixel type (right).

Comparison of different reconstruction techniques for the same captured data. We show reconstruction of a 2D image (bottom right), a low-resolution light field via linear reconstruction (bottom left and center), and a high-resolution light field via sparsity-constrained optimization with overcomplete dictionaries (top). Whereas linear reconstruction trades angular for spatial resolution — thereby decreasing image fidelity — nonlinear reconstructions can achieve an image quality that is comparable to a conventional, in-focus 2D image for each of 25 recovered views.

Overview of captured scenes showing mosaics of light fields reconstructed via sparsity-constrained optimization (top), a single view of these light fields (center), and corresponding 2D images (bottom). These scenes exhibit a variety of effects, including occlusion, refraction, specularity, and translucency. The resolution of each of the 25 light field views is similar to that of the conventional 2D images.

Best Paper Award

The authors accept the best paper award at ICCP 2014. Right to left, Ramesh Raskar, Suren Jayasuriya, Matthew Hirsch, Gordon Wetzstein, and Sriram Sivaramakrishnan