Although we came to Italy mainly to digitize statues, we also had in mind
acquiring at least one really big light field (RBLF) during our year abroad.
After spending several weeks in the Medici Chapel struggling to scan
Michelangelo's four allegories - they are shiny, full of narrow crevices, and
and very close to the wall behind them, we decided it was time to try
something different.
Light fields, sometimes called Lumigraphs, are an example of image-based
rendering (IBR), a relatively new idea in the graphics community. They are
based on the notion of generating (i.e. rendering) new images of a virtual
scene from existing images rather than from 3D models. A light field can be
thought of as an array of perspective images of a scene taken from viewpoints
spaced closely together on a 2D plane. If the spacing between the viewpoints
is sufficiently dense, and the views themselves are sufficiently wide-angle,
then the array contains within it a measurement of the light leaving every
point in the scene and traveling in every direction, literally the field of
light surrounding the scene. By extracting pixels from these images -
sometimes only a few pixels from each, it is possible to construct correct
perspective images for viewpoints other than those present in the original
array.
For viewpoints spaced around a circle, a "flatland" light field can be
visualized using this simple diagram. The blue blob represents the scene. Each
red dot is the viewpoint corresponding to one image of this scene. Each black
line segment emanating from a red dot is a pixel in one of these images. It
represents a unique line of sight. The yellow dot shows a new viewpoint. Its
image can be constructed by borrowing pixels from the existing images, as the
diagram shows. If the red dots are sufficiently dense, then new images can be
constructed for any viewpoint lying outside the convex hull of the scene,
even viewpoints lying outside the circle.
The advantages of light fields over 3D computer models are that rendering is
cheap - just shuffling pixels, its cost is independent of object complexity,
and the resulting images are completely photorealistic. The disadvantages of
light fields are that the scene geometry and its lighting cannot change. Light
fields also require a lot of memory, although in some cases no more than a
canned video sequence, and they offer interactivity, which a canned video
lacks. For more information on light fields and light field rendering, look at
Levoy and Hanrahan, Proc. Siggraph '96, or click
here.
There are many technical choices to be made when designing a light field. In
our case the first choice to be made was: which statue? We looked for
one whose light field was intrinsically interesting and hard to capture with a
3D model. Michelangelo's allegorical statue of Night (shown at left) is both
highly polished and slightly transparent, leading to a variety of subtle,
hard-to-render visual effects. We also looked for a statue whose lighting we
could control and, more importantly, freeze for several days. We could do this
on Night, but to avoid changes in natural lighting and the inevitable
flashbulbs of tourists, we had to work only at night. Finally, we looked for a
statue that we were also planning to digitize with our laser scanner. We have
done this on Night. The resulting combination of a high-resolution 3D model
and a dense light field will provide a unique dataset for exploring a variety
of image-based rendering algorithms.
The next choice to make was: where should we stand? It wasn't practical to
completely encircle the statue with viewpoints. Instead, we decided to make a
90-degree arc around the statue while standing about 8 feet away. In this
photograph the arc can be seen taped out on the floor. To acquire the light
field, we removed the 3D scanner head from our motorized gantry and installed
in its place a high-quality digital still camera. The camera is positioned
roughly at eye level, which is also the height Michelangelo designed the statue
to be seen from.
We divided this arc into 7 "light field slabs", shown in this plan view by
yellow line segments. Each slab is an array of images taken from viewpoints
closely spaced on a vertical "camera plane" that coincides with the line
segment. A conventional optical system with fixed field of view and fixed
focus was used, and the directions of view, shown by orange arrows on the
figure, were chosen by hand to keep the statue nicely framed. Thus, unlike the
light field slabs described in our 1996 paper, the perspective views were
conventional rather than off-axis, the focal plane rotated instead of being
fixed, and the center of attention moved instead of being stationary. It is
possible to generate canonical two-plane light field slabs from this data, but
it would require reprojection and resampling of each acquired image.
The camera plane measured 775mm wide x 700mm high, overlapping its neighbor
75mm. This is large enough to allow a reasonable range of observer motion
during light field rendering: up, down, forward (i.e. towards the statue)
backward, as well as left and right along the arc. Within this plane, we moved
the camera through a grid of 62 x 56 viewpoints spaced 12.5mm apart. The total
number of images acquired was thus 3,472 per light field slab, or 24,304 for
the entire light field. Since the physical aperture of our camera was
considerably smaller than 12.5mm, the light field is not correctly
"anti-aliased", but we could not avoid this without acquiring far more images
than we did.
Although the exposure time per frame was short, 1/12 second, additional time
was required to download and store the image and move the camera to the next
position. Since the direction of view varied with each image, all four motion
axes (horizontal, vertical, pan, and pitch, shown here by green arrows) were in
play at all times. Fortunately, our gantry is stiff, so we didn't have to wait
for vibrations to die down before shooting the next frame. Nevertheless, each
light slab took almost 5 hours to acquire. Shooting 2 slabs per night (working
from dusk to dawn), the entire light field took 4 nights to acquire. Image
resolution was 1300 x 1030 RGB pixels, or 95GB uncompressed. We used JPEG
compression at 6:1, reducing our storage requirements to 16GB. We expect that
vector quantization (VQ) can do much better, but it will more seriously
compromise image quality.
Results
Actual acquisition of the light field took place during the nights of March 26
- 29, 1999. Like our other acquisition projects, this one was more challenging
than we expected. In particular, controlling the lighting was hard. To avoid
changes in natural lighting, we worked only at night. To reduce the harsh
shadows of the existing electric lighting, we installed additional spotlights.
To keep the moving gantry from changing the lighting, we kept it far back from
the statue. Nevertheless, spotlights aged and blew out, the gantry cast
shadows on the floor and tomb near the statue, and the sun rose inconveniently
early each morning. We also found that we could not position the gantry
relative to our taped layout more accurately than about 1cm. Even if we could,
the floor of the chapel is not level. This forced us to enlarge the overlap
between slabs, which in turn increased acquisition time, which in turn pushed
us closer to dawn each night.
Nevertheless, we managed to acquire a reasonably nice dataset. Here are
representative images, slightly reduced in size, from each of the 7 light field
slabs. Histogram matching has been used to eliminate changes in lighting (see
above), but otherwise the images appear exactly as acquired.
Click here for a full-size (1300 x 1030) version of the
third image. Click here for a movie of one row of images from the fourth
slab, at half-size
(.avi (6.5MB) or
.rm (900KB)).
We are currently assembling these images into a light field
that will be viewable using our downloadable
light field viewer software. When we finish the light field, we'll put it
here.
© 1999
Stanford Computer Graphics Laboratory
webmaster@graphics.stanford.edu
