Siggraph 2008 is taking place this week, and there’s a ton of really great research papers that are being talked about and published. One such paper was published by Microsoft Research, and it details how they took standard DV footage and then used it’s pixel motion to unwrap the footage into a useable texture map. You can then paint on top of it, recomposite it over your footage and wahla! Pretty cool stuff.
Though in the video they presented they just used the process to put some funny eyebrows and a big bushy mustache on the actor above, I can’t help but to think how useful this would be for generating clean plates, digital double texturing, and rotoing elements in a plate as well.
We introduce a new representation for video which facilitates a number of common editing tasks. The representation has some of the power of a full reconstruction of 3D surface models from video, but is designed to be easy to recover from a priori unseen and uncalibrated footage. By modelling the image-formation process as a 2D-to-2D transformation from an object’s texture map to the image, modulated by an object-space occlusion mask, we can recover a representation which we term the “unwrap mosaic”. Many editing operations can be performed on the unwrap mosaic, and then re-composited into the original sequence, for example resizing objects, repainting textures, copying/cutting/pasting objects, and attaching effects layers to deforming objects.