Mosaic generation and sprite-based coding with automatic foreground and background separation

DETAILED DESCRIPTION Automatic object segmentation generates high quality mosaic (panoramic) images and operates with the assumption that each of the objects present in the video scene exhibits dynamical modes which are distinct from the global motion induced by the camera.
Image segmentation, generation of a background mosaic and coding are all intricately linked.
Image segmentation is progressively achieved in time and based on the quality of prediction signal produced by the background mosaic.
Consequently, object segmentation is embedded in the coder/decoder (codec) as opposed to being a separate pre or post-processing module, reducing the overall complexity and memory requirements of the system.
In the encoder, foreground and background objects are segmented by first encoding and decoding a first image at a first time reference.
The method used to encode and decode this first image does not need to be specified for the purpose of this invention.
The second image at a second time reference is divided into non-overlapping macroblocks (tiles).
The macroblocks are matched to image sample arrays in the decoded first image or in the mosaic.
In the first case, the encoder uses local motion vectors to align an individual macroblock with one or several corresponding image sample array in the previous decoded image.
In the second case, the encoder uses parameters of a global motion model to align an individual macroblock with a corresponding mosaic sample array.
The encoder evaluates the various residuals and selects the proper prediction signal to use according to a pre-specified policy.
This decision is captured in the macroblock type.
The macroblock types, the global motion parameters, the local motion vectors and the residual signals are transmitted to the decoder.
Frame residuals represent the difference between the macroblocks and corresponding image arrays in the previously decoded image matched by using local motion vectors.
Macroblocks having a single local motion vector are identified as INTER1V-type macroblocks