Light field photography is an image-based technique for recording scene appearance. Unlike conventional photography, light fields permit manipulation of viewpoint and focus after the imagery has been recorded. Devices for capturing light fields range in scale from room-size arrays of cameras to a handheld camera in which a microlens array has been inserted between the main lens and sensor plane.
In this project, we are exploring light field photography at the microscopic scale. Specifically, by inserting a microlens array into the optical train of a conventional optical microscope, we can capture light fields of biological specimens in a single snapshot. Although diffraction places a limit on the product of spatial and angular resolution in these light fields, we can nevertheless produce useful perspective flyarounds and 3D focal stacks from them. Applying standard deconvolution algorithms to these focal stacks, we can reconstruct 3D volumes. Since microscope optics produce orthographic views, perspective flyarounds represent a new way to look at microscopic specimens. Focal stacks are not new, but manual techniques for capturing them are time-consuming and hence not applicable to moving or light-sensitive specimens.
Like any new scientific imaging instrument, we expect the light field microscope to have many applications in science, medicine, and industry. For example, since the light field microscope separates image acquisition from the selection of viewpoint and focus. it can be used as a "digital viewfinder" for a conventional microscope. Applying computer vision algorithms to these selection procedures may lead to greater automation of microscope operation when analyzing large numbers of specimens, such as in clinical pathology.
February 14, 2008
Microscope design: Marc Levoy
Specimen: Shinya Inoue (Marine Biological Laboratory)
At left above is our most compact light field microscope (LFM). As introduced on this web page, it consists of an ordinary research microscope (Nikon Eclipse 80i) and cooled scientific camera (Retiga 4000R, 2048 x 2048 pixels) with a microlens array inserted between the two (red circle). The image formed by the microlens array is conveyed to the sensor in the camera by a 1:1 relay lens system consisting of two nose-to-nose Nikon 50mm/1.4 photographic lenses. With this arrangement, instead of capturing an ordinary 2048 x 2048 pixel image, the camera captures a 2048 x 2048 sample light field, capable of producing 17 x 17 different oblique views, each about 120 x 120 pixels in size. We have fabricated several copies of this arrangement, which have been used by microscopists in several departments at Stanford University, at the Hopkins Marine Institute (Monterey, CA), the Marine Biological Laboratory (Woods Hole, MA), and ETH (Zurich).
At right above is a prototype high-resolution light field microscope. A 45-degree mirror (A) sends the microscope image through a 2:1 telecentric relay lens (B), our microlens array (C), a 1:1 telecentric relay lens (D), and a Canon 5D full-frame digital SLR (E). Since doubling the lateral magnification using a 2:1 relay lens roughly halves the angular aperture (this is called the LaGrangian invariant), we use an f/30 microlens array rather than the f/20 array we more commonly use. The camera was manually microstepped to place a green pixel beneath each pixel site, thereby avoiding color demosaicing and the consequent loss in spatial resolution. The light fields captured using this arrangement (see below) contain approximately 4500 x 3000 samples, capable of producing 15 x 15 oblique views, each about 300 x 200 pixels in size. The field of view is approximately 1.8mm horizontally for the 20x light field and 0.9mm for the 40x. For the objectives we used and the pixel resolution of our camera, these light fields are also diffraction-limited, meaning that no further spatial or angular resolution can be squeezed from the wavefront propagating through the microscope, at least not using conventional image capture protocols.
The two columns above represent two light fields captured using this high-resolution microscope. The specimen is a Golgi-stained slice of rat brain (courtesy of Shinya Inoue, Marine Biological Laboratory). The light field at left was photographed using a 20x/0.75NA objective, and the light field at right using a 40x/1.3NA oil immersion objective. Note the spatial detail in both movies; you can easily see the fine dendrites. Note also the large amount of parallax visible in the 40x movie, especially on the looping capillaries in the lower-left part of the field of view. This much parallax is available because a 1.3NA objective can capture rays leaving the specimen at angles up to 59 degrees on either side of the optical axis. For photographers, this would correspond to a lens having a relative aperture of f/0.38 - far faster than any commercially available photographic lens.
