SIGGRAPH 2012 (August 7, 2012)
This was a successful year at SIGGRAPH. Together with Gordon Wetzstein and Matthew Hirsch, I presented "Tensor Displays" as both a technical paper and an emerging technologies demonstration. This work unifies our earlier multilayer display projects, including "High-Rank 3D (HR3D)" and "Layered 3D". This paper describes how a stack of three or more LCD panels can be controlled to allow high-fidelity light-field display (otherwise known as a glasses-free 3D display). The key contribution is identifying how multilinear algebra can be applied to solve for the optimal patterns to display on each panel during each frame (solving the multilayer-multiframe decomposition problem). Matt headed the demonstration, carefully packaging the prototype for shipment and manning the booth over the week. In return, he was rewarded with amble press coverage, some of which is tabulated here. Gordon, Matt, and I also contributed to the courses program, presenting the "Computational Plenoptic Imaging" and "Computational Displays" courses.
Joining NVIDIA Research (June 25, 2012)
This month I'm finishing my postdoc and moving back to California. I'll be joining NVIDIA Research to work in the "Computer Graphics" and "New Experiences" groups led by David Luebke. The latter group deserves some explanation; recent SIGGRAPH publications increasingly emphasize camera and display technologies. My focus will be to continue my systems-oriented imaging research, with the aim of bringing some of these "new experiences" out of the laboratory and into user's hands.
Videos of 2011 Invited Talks (September 19, 2011)
This fall marks the beginning of the second year of my post-doc at the MIT Media Lab. During the last year I've enjoyed traveling frequently to present my work, as well as that of the Camera Culture group, at various computer vision, graphics, and optics venues. A complete list of these talks is available here. I've included links to videos of my NIPS 2010 and Photographic Universe 2011 talks. The former talk focuses on light field capture and display, summarizing work developed in my dissertation, and the latter talk summarizes the larger field of computational photography.
Layered 3D and SIGGRAPH 2011 Course (May 4, 2011)
Our SIGGRAPH 2011 technical paper, entitled "Layered 3D: Tomographic Image Synthesis for Attenuation-based Light Field and High Dynamic Range Displays" is now available online here. This is one of the three SIGGRAPH technical papers being featured this month in the SIGGRAPHITTI newsletter. In other news, our SIGGRAPH course entitled "Build Your Own Glasses-free 3D Display" was also accepted. This year we'll focus at an intermediate-level audience and describe practical OpenGL multi-view rendering and GLSL fragment shaders for antialiasing and interlacing operations. This course will be jointly presented with Matthew Hirsch and Gregg Favalora from Optics for Hire.
Eurographics 2011 STAR and SIGGRAPH 2011 Acceptance (April 6, 2011)
At Eurographics 2011 Gordon Wetzstein and I will co-present a State of the Art Report (STAR) on Computational Plenoptic Imaging. This STAR describes the plenoptic function: a ray-based model for light that includes the color spectrum as well as spatial, temporal, and directional variation, as introduced by Adelson and Bergen in 1991. We review approaches that optically encode dimensions of the plenoptic function beyond those captured by traditional photography and reconstruct the recorded information computationally. In other news, my SIGGRAPH 2011 technical paper, entitled "Tomographic Image Generation for Attenuation-based Light Field and High Dynamic Range Displays" was conditionally accepted. This project is joint work with Gordon Wetzstein and Wolfgang Heidrich, both from University of British Columbia, and Ramesh Raskar. It is my third paper to focus on "computational displays", following the BiDi Screen (SIGGRAPH Asia 2009) and Content-Adaptive Parallax Barriers (SIGGRAPH Asia 2010).
SIGGRAPH 2010 (July 25, 2010)
At SIGGRAPH 2010 Matthew Hirsch and I will co-present Build Your Own 3D Display. This course provides a beginner with the necessary mathematics, software, and practical details to construct their own 3D displays, including LCD shutter glasses and automultiscopic LCD displays. I will also present a talk and poster on Content-Adaptive Parallax Barriers; this project emerged from a collaboration between Matt, Yunhee Kim, Szymon Jakubczak, and Ramesh Raskar, and indirectly extends our prior work on Shield Fields and the BiDi Screen.
Ph.D. Thesis Defense (July 6, 2010)
Today I passed my Ph.D. thesis defense. My dissertation committee consisted of Gabriel Taubin (director), Joseph Mundy, and Ramesh Raskar. The title of my dissertation is "Mask-based Light Field Capture and Display". It is the culmination of five years of graduate study, three journal articles published in the ACM Transactions on Graphics, and countless months of writing. Above all, I am grateful for the endless support my fiancée Erika Dowd provided. In a few weeks I'll be joining the Camera Culture group at the MIT Media Lab (supervised by Ramesh Raskar) as a Postdoctoral Associate.
SIGGRAPH Asia 2009: BiDi Screen and 3D Scanning (December 9, 2009)
At SIGGRAPH Asia 2009 I am co-presenting a technical paper on the BiDi Screen: A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields. This paper describes how to transform an off-the-shelf LCD into a BiDirectional (BiDi) screen allowing 2D multi-touch and 3D gestures. Similar to my earlier work on Shield Fields, also with Ramesh Raskar, the BiDi screen uses an image sensor placed behind a liquid crystal layer to form a mask-based light field camera. I will co-present a live demonstration of the BiDi screen at E-Tech with Matt Hirsch and Henry Holtzman. In addition, Gabriel Taubin and I will co-instruct the Build Your Own 3D Scanner course, which we previously presented at SIGGRAPH 2009.
SIGGRAPH 2009: Build Your Own 3D Scanner and BiDi Screen (June 3, 2009)
At SIGGRAPH 2009 Gabriel Taubin and I will co-present Build Your Own 3D Scanner: 3D Photography for Beginners (preview the lecture notes here). This course provides a beginner with the necessary mathematics, software, and practical details to leverage projector-camera systems in their own projects. In addition, Matt Hirsch will present a talk and poster on the BiDi Screen; this project emerged from a collaboration between Matt, Ramesh Raskar, Henry Holtzman, and myself, and extends our prior work on Shield Fields.
Image Destabilization for Programmable Defocus (April 20, 2009)
I recently traveled to San Francisco to present Image Destabilization: Programmable Defocus using Lens and Sensor Motion at ICCP 2009. This project was completed in collaboration with Ankit Mohan, Shinsaku Hiura, and Ramesh Raskar, while I was a visiting student at the MIT Media Lab last fall. In this work we propose a novel camera in which both the lens and sensor are perturbed during image exposure. Using this setup we simulate a lens with a larger effective aperture and demonstrate depth-independent defocus blur size. These techniques could ultimately achieve SLR-quality photos (with a narrow depth of field and pleasing bokeh), even with pinhole-like mobile phone cameras. Such applications were discussed in the Photo Future article in the May/June issue of Technology Review, which surveys our work.
Shield Fields: Modeling and Capturing 3D Occluders (November 24, 2008)
Over the last year I have been collaborating with Prof. Ramesh Raskar and Amit Agrawal at MIT Media Lab and MERL, respectively. Our paper on heterodyne light field capture and visual hull reconstruction will be presented at SIGGRAPH Asia 2008. The key theoretical result is that shield fields, a closely-related concept to light fields, can be used to decouple the effects of occluders and incident illumination. We first describe the properties of shield fields in the frequency-domain and briefly analyze the "forward" problem of efficiently computing cast shadows. Afterwards, we apply the shield field signal-processing framework to make several new observations regarding the "inverse" problem of reconstructing 3D occluders from cast shadows. Building on the previous work by Veeraraghavan et al., this paper also introduces a new family of optimal tiled-broadband codes for efficient light field capture. We apply these codes to develop the first single-camera, single-shot approach to capture visual hulls without requiring moving or programmable illumination. For more information, please see the project website here.
Modeling and Synthesis of Aperture Effects in Cameras (June 20, 2008)
My recent work on coded aperture imaging was presented at the International Symposium on Computational Aesthetics in Graphics, Visualization, and Imaging
(CAe 2008). This paper focuses on the capture, analysis, and synthesis of vignetting and depth-of-field effects in conventional cameras. In order to simultaneously estimate vignetting-correction tables and spatially-varying blur kernels, we propose a simple calibration pattern consisting of a two-dimensional array of point light sources. In addition, this paper introduces the Bokeh Brush: a novel, post-capture method for full-resolution control of the shape of out-of-focus highlights. Typical results are summarized in the animations below. More information, including a copy of the paper and the presentation slides, is available here.
Surround Structured Lighting for Full Object Scanning (August 21, 2007)
I recently presented my paper Surround Structured Lighting for Full Object Scanning at the 6th International Conference on 3-D Digital Imaging and Modeling (3DIM 2007) in Montréal. This paper introduces a new system for acquiring complete 3D surface models using a single structured light projector, a pair of planar mirrors, and one or more synchronized cameras. Using an orthographic projector composed of a Fresnel lens and a DLP projector, we display a single Gray code sequence to encode all the illumination planes within the scanning volume. More information is available here.
Structured Light for 3D Scanning (May 15, 2007)
My research during the last semester has focused on the topic of 3D reconstruction using controlled illumination. As the head TA for CS 220, I created an assignment in which the students implemented Bouguet's desktop 3D scanner using only a webcam, a halogen lamp, and a stick. The results of my reference implementation are presented here. To support additional class projects, I also implemented a conventional 3D scanning system using structured light. A given object was scanned by illuminating with a temporally-multiplexed Gray code sequence. By observing the projected patterns from one or more viewpoints, each visible point was reconstructed in 3D by ray-plane triangulation. A complete Matlab projector-camera calibration toolbox was developed to estimate the relative projector-camera position/pose and intrinsic parameters. For more information, see my class presentation here. Finally, my own project on Surround Structured Lighting was recently accepted for publication and oral presentation at 3DIM 2007.
Spring Course Schedule (January 21, 2007)
This term I will be taking a variety of courses in Computer Science and Electrical Engineering, however my primary focus will be on expanding my research. See the webpages below for detailed course descriptions. In addition, I am currently working with the Department of Engineering to secure funding for a computer vision seminar series. For more information, see the Computer Vision Reading Group website.
- CS 220: 3D Photography and Geometry Processing
- EN 257: Applied Stochastic Processes
- EN 298: Special Projects, Reading, Research and Design
Downloads: course schedule
Exemplar-based Image Inpainting (December 19, 2006)
I recently completed my final project for EN 256: Computer Vision. Building on my prior Image Quilting implementation, I created a complete system for automatically inpainting large image regions. More specifically, I implemented Exemplar-based Image Inpainting by
Criminisi, Pérez and Toyama. As shown in the figure below, this approach involves copying small image fragments to complete a user-specified masked region. For the "Hollywood" example, we find that this system reliably removes a large region of the original image while preserving the underlying background textures. In general, this approach has broad applications extending from photographic restoration to removing wires from special effects sequences. For more information, see my final report and the accompanying movies
Texture Synthesis using Image Quilting (November 27, 2006)
For my final project in EN 256: Computer Vision, I am currently studying texture synthesis algorithms and their application to problems in computer vision and graphics. If we view a given image as a sample from a local stationary random process, then the goal of texture synthesis is to generate additional images
which appear (to a human observer) to be produced from the
same underlying distribution. Applications include non-periodic texture mapping for graphics, as well as inpainting for photographic restoration. For reasons discussed in my project proposal, I decided to implement Efros and Freeman's Image Quilting algorithm – a particularly simple, yet effective, patch-based solution. As shown in the figure below, the algorithm begins by selecting a random seed block from the sample texture. Afterwards, additional blocks are assigned (in raster order) such that the overlapping regions are as similar as possible. For more information regarding this technique, please consult my recent progress report. Source code will be made available in late December.
Unsupervised Learning of Handwriting Styles (November 10, 2006)
Recently, Bernoulli mixture models (BMM) have been used in a wide variety of applications including document understanding, image analysis, and financial modeling. I recently implemented the EM algorithm for multivariate BMM estimation. As an example, a 12 component model was trained on the first five digits in the MNIST handwriting dataset (see the 10,000 training samples here). The per-pixel values of the Bernoulli parameters, for the first 20 steps of the EM algorithm, are animated below. Note that the BMM automatically identified the five digit classes, as well as the dominant styles for writing each digit. Additionally, the mixture components can be used for text recognition; for instance, an optimal Bayes classifier built using the BMM obtains a test error rate on the order of 3% – comparable to the results reported by LeCun and Cortes here. For additional information, see the following report.
Bilateral Filtering and Image Abstraction (September 2, 2006)
I have recently implemented a form of automatic image abstraction inspired by the SIGGRAPH paper by Winnemöller et al. My source code for bilateral filtering and image/video abstraction is now available on Matlab Central. Results are presented in the following album and movies
Fall Course Schedule (September 1, 2006)
This term I am taking a variety of courses in the Engineering, Computer Science, and Applied Math. departments. Please see the following websites for course descriptions:
- EN 256: Computer Vision
- CS 195: Introduction to Machine Learning
- AM 255: Numerical Solution of Partial Differential Equations I
- EN 297: Special Projects, Reading, Research and Design
Downloads: course schedule
SIGGRAPH and France (August 5, 2006)
I recently attended SIGGRAPH 2006 to present my poster on Multi-Flash 3D Photography. There were a great variety of papers involving Computational Photography this year, including: Photosynth, Coded Exposure Photography, and Multiview Radial Catadioptric Imaging for Scene Capture.
Prior to attending SIGGRAPH, I was in France for a month working as a summer intern at INRIA Rhône-Alpes in the PERCEPTION team. (Note that the artistic rendering shown at the left was generated automatically by the NPR system presented by ETRI in the Guerilla Studio.)
Publications Update (June 7, 2006)
The Research and Courses/Projects pages have been updated. Three papers have been accepted for presentation at the Third International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT 2006). My SIGGRAPH poster has also been accepted. In addition, a variety of homework solutions, course projects, and other tutorials have been made available for download.
Matlab Central Contributions (January 15, 2006)
In my research I often implement a variety of basic image processing algorithms in Matlab. I've decided to
start posting these programs to the Matlab Central
Please see my author page for more information.
Update (January 5, 2006)
This term I will be focusing my research on visual sensor networks (VSN). In addition, I will continue my efforts on catadioptric mirror arrays, especially for 3D reconstruction. Coursework will include:
- EN 292: Video Processing
- CS 157: Design and Analysis of Algorithms
- EN 202: Mathematical Methods for Engineering and Physics II
- EN 298: Special Projects, Reading, Research and Design
Downloads: course schedule
November 14, 2012