Omnimatte: Associating Objects and Their Effects in Video
We pose a novel problem: automatically associating subjects in videos with ‘effects’ related to them in the scene. Given an input video (top) and rough masks of subjects of interest (middle), our method estimates an
omnimatte – an alpha matte and foreground color that includes the subject itself along with all scene elements associated with it (bottom). The associated elements can be other objects attached to the subject or moving with it, or complex effects such as shadows, reflections, smoke, or ripples the subject creates in water.
Abstract
Computer vision is increasingly effective at segmenting objects in images and videos; however, scene effects related to the objects shadows, reflections, generated smoke, etc are typically overlooked. Identifying such scene effects and associating them with the objects producing them is important for improving our fundamental understanding of visual scenes, and can also assis