Natural Video Matting using Camera Arrays

 

Neel Joshi Wojciech Matusik Shai Avidan
University of California, San Diego MERL MERL

SIGGRAPH 2006

 

(a) We use a camera array to capture a collection of images of a scene. Here we show a single camera’s image. (b) We synthetically refocus the data and compute the variance image of the refocused images (darker means lower variance). (c) From the “variance image” we automatically compute a trimap. (d) We extrapolate the variances into the unknown region of the trimap and use these measurements to solve for the alpha matte. (e) We then compute the alpha multiplied foreground and composite it over a new background.

 

Abstract

 

We present an algorithm and a system for high-quality natural video matting using a camera array. The system uses high frequencies present in natural scenes to compute mattes by creating a synthetic aperture image that is focused on the foreground object, which reduces the variance of pixels reprojected from the foreground while increasing the variance of pixels reprojected from the background. We modify the standard matting equation to work directly with variance measurements and show how these statistics can be used to construct a trimap that is later upgraded to an alpha matte. The entire process is completely automatic, including an automatic method for focusing the synthetic aperture image on the foreground object and an automatic method to compute the trimap and the alpha matte. The proposed algorithm is very effcient and has a per-pixel running time that is linear in the number of cameras. Our current system runs at several frames per second, and we believe that it is the first system capable of computing high-quality alpha mattes at near real-time rates without the use of active illumination or special backgrounds.

 

Citation

Neel Joshi, Wojciech Matusik, and Shai Avidan. Natural Video Matting using Camera Arrays. ACM Transactions on Graphics. 25(3) July 2006.

Paper

Adobe Acrobat PDF (4.27 MB)

Addendum: Reduction from our method to Triangulation Matting

Adobe Acrobat PDF

SIGGRAPH 2006 Talk

PPT Slides Only (8.76 MB) | Zip File of Slides and Videos (58.3 MB)

Overview Video

Divx (54.3 MB) | Quicktime MPEG4 (56.3 MB)

Supplementary Videos (all Divx)

Smoke: central view | synthetic aperture | variance | trimap | alpha | alpha multiplied foreground | composite

Coffee: real-time screen capture

Coffee: central view | synthetic aperture | variance | trimap | alpha | composite

Hair: real-time screen capture

Dynamic Backgrounds: real-time screen capture

Dynamic Backgrounds: central view | variance | alpha | composite

Datasets

UCSD/MERL Light Field Archive

   

Copyright 2006 by Neel Joshi, MERL, and the ACM

Use of images, videos, and slides for non-commercial, academic, or news related purposes is allowed with proper attribution. Contact Neel Joshi for inquiries about other uses.