Media production with correlation of image stream and abstract objects in a three-dimensional virtual stage

DETAILED DESCRIPTION Rather than working solely with flattened two-dimensional (2D) images that can only be combined using 2D techniques, the invention allows the application of both three-dimensional (3D) and 2D techniques for integration of different media elements within a common virtual stage.
To that end, the 3D characteristics of live/recorded elements are reconstructed for use in the virtual stage.
Similarly, 3D models of synthetic objects can be directly incorporated into the virtual stage.
In that virtual stage, 3D representations of both physical and synthetic objects can be choreographed, and the resulting 2D images may be rendered in an integrated fashion based on both 3D and 2D data.
Accordingly, the present invention utilizes a data processing system in media production.
Representations of objects, including representations derived from at least one image stream captured from physical objects, are provided in a three-dimensional virtual stage.
A correlation is maintained between representations of objects in the virtual stage and corresponding segments of the at least one image stream.
Representations of objects are choreographed within the virtual stage, and a choreography specification is provided for generation of a two-dimensional image stream of the choreographed objects within the virtual stage.
Representations of objects in the virtual stage include both 3D representations of physical objects and 3D representations of synthetic objects.
2D representations of these and other objects on the stage may also be included.
Representations of a virtual camera and lighting corresponding to the camera and lighting used to capture the image stream from the physical objects can also be provided as objects in the virtual stage, and the positions and orientations of the virtual camera and virtual lighting can be manipulated within the virtual stage.
A 3D path within the virtual stage may represent the motion associated with at least one feature of an object represented in the virtual stage