Understanding and Recreating Visual Appearance Under Natural Illumination

The appearance of an outdoor scene is determined to a great extent by the prevailing illumination conditions. However, most practical computer vision applications treat illumination more as a nuisance rather than a source of signal. In this dissertation, we suggest that we should instead embrace illumination, even in the challenging, uncontrolled world of consumer photographs.

Our first main contribution is an understanding of natural illumination from images. This is, in general, a hard problem given the wide appearance variation in scenes. Fortunately, natural illumination, while complex, is far from being completely arbitrary. It has a structure that is well understood in atmospheric optics, but which has hardly been exploited in vision and graphics. We introduce methods for automatically estimating the illumination conditions from two types of uncontrolled outdoor image datasets: webcams and single images. The variation in sun position and sky appearance over time can be exploited to obtain viewing and illumination geometry in webcam sequences. For single images, the sky is combined in a probabilistic way with other scene features such as cast shadows and shading on vertical surfaces and convex objects, as well as with illumination priors from large image collections.

Our second main contribution is to exploit the knowledge of illumination in order to synthesize novel, realistic visual content. Instead of creating appearance using the traditional computer graphics pipeline, we propose to borrow the appearance of the world that is contained in existing photo collections and webcam datasets. We also demonstrate realistic 3-D object insertion by creating plausible high-dynamic range environment maps. This can be done in image sequences, and even in single images, completely automatically. Addressing such questions has implications in a broad range of applications including intelligent transportation, surveillance, human-robot interaction, and digital entertainment.