Apparatus and Method for Generating a Coded Video Sequence
and for Decoding a Coded Video Sequence by Using an
Intermediate Layer Residual Value Prediction
5
Description
The present invention relates to video coding systems and
particularly to scalable video coding systems, which can be
10 used in connection with the video coding standard H.264/AVC
or with new MPEG video coding systems.
The standard H.264/AVC is the result of a video
standardization project of the ITU-T video coding expert
15 group VCEG and the ISO/IEC motion picture expert group
(MPEG). The main goals of this standardization project are
to provide a clear video coding concept with very good
compression behavior and at the same time to generate a
network-friendly video representation, which comprise both
20 application with "conversation character", such as video
telephony, as well as applications without conversion
character (storage, broadcast, stream transmission).
Apart from the above-mentioned standard ISO/IEC 14496-10,
25 there is also a plurality of publications relating to the
standard. Merely exemplarily, reference is made to "The
Emerging H.264-AVC standard", Ralf Schafer, Thomas Wiegand
and Heiko Schwarz, EBU Technical Review, January 2003.
Additionally, the expert publication "Overview of the
30 H.264/AVC Video Coding Standard", Thomas Wiegand, Gary J.
Sullivan, Gesle Bjontegaard and Ajay Lothra, IEEE
Transactions on Circuits and Systems for Video Technology,
July 2003 as well as the expert publication ,,Context-based
adaptive Binary Arithmethic Coding in the H.264/AVC Video
35 Compression Standard", Detlev Marpe, Heiko Schwarz and
Thomas Wiegand, IEEE Transactions on Circuits and Systems
for Video Technology, September 2003, comprise a detailed

- 2 -
overview over different aspects of the video coding
standard.
However, for a better understanding, an overview over the
5 video coding/decoding algorithm will be given with
reference to Figs. 9 to 11.
Fig. 9 shows a full structure of a video coder, which
generally consists of two different stages. Generally, the
10 first stage,. which generally operates video-related,
generates output data, which are then subject to an entropy
coding by a second stage, which is designated by 80 in Fig.
9. The data are data 81a, quantized transformation
coefficients 81b as well as motion data 81c, wherein these
15 data 81a, 81b, 81c are supplied to the entropy coder 80 to
generate a coded video signal at the output of the entropy
coder 80.
Specifically, the input video signal is partitioned and
20 splitted, respectively, into macroblocks, wherein every
macroblock has 16 x 16 pixels. Then, the association of the
macroblocks to slice groups and slices is chosen, according
to which every macroblock of every slice is processed by
the net of operation blocks as illustrated in Fig. 8. It
25 should be noted that an efficient parallel-processing of
macroblocks is possible when different slices exist in a
video picture. The association of macroblocks to slice
groups and slices is performed via a block coder control 82
in Fig. 8. There are different slices, which are defined as
30 follows:
I slice: The I slice is a slice wherein all macroblocks of
the slice are coded by using an intra prediction.
35 P slice: Additionally to the coding types of the I slices,
certain macroblocks of the P slice can also be coded by
using an inter prediction with at least one motion-
compensated prediction signal per prediction block.

- 3 -
B slice: Additionally to the coder types available in the P
slice, certain macroblocks of the B slice can also be coded
by using an inter prediction with two motion-compensated
5 prediction signals per prediction block.
The above three coder types are very similar to the ones in
earlier standards, but with the exception of using
reference pictures, as will be described below. The
10 following two coder types for slices are new in the
standard H.264/AVC:
SP slice: It is also referred to as switch P slice, which
is coded such that efficient switching between different
15 precoded pictures is made possible.
SI slice: The SI slice is also referred to as switch I
slice, which allows an exact adaptation of the macroblocks
in a SP slice for a direct random access and for error
20 recovery purposes.
All in all, slices are a sequence of macroblocks, which are
processed in the order of a raster scan, if not a property
of the flexible macroblock ordering FMO is used, which is
25 also defined in the standard. A picture can be partitioned
into one or several slices, as illustrated in Fig. 11.
Thus, a picture is a collection of one or several slices.
In that sense, slices are independent of one another, since
their syntax elements can be analyzed (parsed) from the bit
30 stream, wherein the values of the samples can be decoded
correctly in the range of the picture represented by the
slice, without requiring data from other slices, provided
that used reference pictures are identical both in the
coder and in the decoder. However, certain information from
35 other slices can be required to apply the deblocking filter
across slice borders.

- 4 -
The FMO characteristic modifies the way how pictures are
partitioned into slices and macroblocks, by using the
concept of slice groups. Every slice group is a set of
macroblocks defined by a macroblock to slice group mapping,
5 which is specified by the content of a picture parameter
set and by certain information from slice headers. This
macroblock to slice group mapping consists of a slice group
identification number for every macroblock in the picture,
wherein it is specified to which slice group the associated
10 macroblock belongs. Every slice group can be partitioned
into one or several slices, so that a slice is a sequence
of macroblocks within the same slice group, which is
processed in the order of a raster sampling within the set
of macroblocks of specific slice group.
15
Every macroblock can be transmitted in one or several coder
types, depending on the slice coder type. In all slice
coder types, the following types of intra coding are
supported, which are referred to as intra-4x4 or intra-16x16,
20 wherein additionally a chroma prediction mode and an I-PCM
prediction mode are supported.
The intra-4x4 mode is based on the prediction of every 4x4
chroma block separately and is very well suited for coding
25 parts of a picture with outstanding details. The intra_16x16
mode, on the other hand, performs a prediction of the whole
16x16 chroma block and is more suited for coding "soft"
regions of a picture.
30 Additionally to these two chroma prediction types, a
separate chroma prediction is performed. As an alternative
for intra-4X4 and intra-16X16, the I-4X4 coder type allows that
the coder simply skips the prediction as well as the
transformation coding and instead transmits the values of
35 the coded samples directly. The I-PCM mode has the following
purposes: It allows the coder to represent the values of
the samples precisely. It provides a way to represent the
values of very abnormal picture content exactly without

- 5 -
data enlargement. Further, it allows to determine a hard
boundary for the number of bits, which a coder needs to
have for macroblock handling without loss of coding
efficiency.
5
In contrary to earlier video coding standards (namely H.263
plus and MPEG-4 visual), where the intra prediction has
been performed in the transformation domain, the intra
prediction in H.264/AVC is always performed in the spatial
10 domain, by referring to adjacent samples of previously
coded blocks, which are on the left of and above,
respectively, the block to be predicted (Fig. 10) . In
certain environments, where transmission errors occur, this
can cause an error propagation, wherein this error
15 propagation takes place due to the motion compensation in
intra coded macroblocks. Thus, a limited intra coding mode
can be signaled, which enables a prediction of only intra
coded adjacent macroblocks.
20 When the intra-4X4 mode is used, every 4x4 block of
spatially adjacent samples is predicted. The 16 samples of
the 4x4 block are predicted by using previously decoded
samples in adjacent blocks. One of 9 prediction modes can
be used for every 4x4 block. Additionally to the "DC
25 prediction" (where a value is used to predict the whole 4x4
block), 8 direction prediction modes are specified. These
modes are suitable to predict direction structures in a
picture, such as edges in different angles.
30 Additionally to the intra macroblock coder types, different
predictive or motion-compensated coder types are specified
as P macroblock types. Every P macroblock type corresponds
to a specific partition of the macroblock into the block
forms, which are used for a motion-compensated prediction.
35 Partitions with luma block sizes of 16x16, 16x8, 8x8 or
8x16 samples are supported by the syntax. In the case of
partitions of 8x8 samples, an additional syntax element is
transmitted for every 8x8 partition. This syntax element

— 6 —
specifies whether the respective 8x8 partition is further
partitioned into partitions of 8x4, 4x8 or 4x4 luma samples
and corresponding chroma samples.
5 The prediction signal for every prediction-coded MxM luma
block is obtained by shifting a region of the respective
reference picture specified by a translation motion vector
and a picture reference index. Thus, if the macroblock is
coded by using four 8x8 partitions, and when every 8x8
10 partition is further partitioned into four 4x4 partitions,
a maximum amount of 16 motion vectors for a single P
macroblock can be transmitted within the so-called motion
field.
15 The quantization parameter slice QP is used to determine
the quantization of the transformation coefficients in
H.264/AVC. The parameter can assume 52 values. These values
are disposed such that an increase of 1 with regard to the
quantization parameter means an increase of the
20 quantization step width by about 12 %. This means that an
increase of the quantization parameter by 6 causes an
increase of the quantizer step width by exactly a factor of
2. It should be noted that a change of the step size by
about 12 % also means a reduction of the bit rate by about
25 12 %.
The quantized transformation coefficients of a block are
generally sampled in zigzag path and processed by using
entropy coding methods. The 2x2 DC coefficients of the
30 chroma component are sampled in raster scan sequence and
all inverse transformation operations within H.264/AVC can
be implemented by using only additions and shift operations
of 16 bit integer values.
35 With reference to Fig. 9, the input signal is first
partitioned picture by picture in a video sequence, for
every picture, into the macroblocks with 16x16 pixels.
Then, every picture is supplied to a subtractor 84, which

- 7 -
subtracts the original picture, which is supplied by a
decoder 85, which is contained in the coder. The
subtraction result, which means the residual signals in the
spatial domain, are now transformed, scaled and quantized
5 (block 86) to obtain the quantized transformation
coefficients on line 81b. For generating the subtraction
signal, which is fed into the subtractor 874, the quantized
transformation coefficients are first again scaled and
inverse transformed (block 87), to be supplied to an adder
10 88, the output of which feeds the deblocking filter 89,
wherein the output video signal, as, for example, will be
decoded by a decoder, can be monitored at the output of the
deblocking filter, for example for control purposes (output
90) .
15
By using the decoded output signal at output 90, a motion
estimation is performed in block 91. For motion estimation
in block 90, a picture of the original video signal is
supplied, as seen from Fig. 9. The standard allows two
20 different motion estimations, namely a forward motion
estimation and a backward motion estimation. In the forward
motion estimation, the motion of the current picture is
estimated with regard to the previous picture. In the
backward motion estimation, however, the motion of the
25 current picture is estimated by using the future picture.
The results of the motion estimation (block 91) are
supplied to a motion compensation block 92, which performs
a motion-compensated inter prediction, particularly when a
switch 93 is switched to the inter prediction mode, as it
30 is the case in Fig. 9. If, however, the switch 93 is
switched to intra frame prediction, an intra frame
prediction is performed by using a block 490. Therefore,
the motion data are not required, since no motion
compensation is performed for an intra frame prediction.
35
The motion estimation block 91 generates motion data and
motion fields, respectively, wherein motion data and motion
fields, respectively, which consist of motion vectors, are

- 8 -
transmitted to the decoder so that a corresponding inverse
prediction, which means reconstruction by using the
transformation coefficients and the motion data, can be
performed. It should be noted that in the case of a forward
5 prediction, the motion vector can be calculated from the
immediately previous picture and from several previous
pictures, respectively. Above that, it should be noted that
in the case of a backward prediction, a current picture can
be calculated by using the immediately adjacent future
10 picture and of course also by using further future
pictures.
It is a disadvantage of the video coding concept
illustrated in Fig. 9 that it provides no simple
15 scalability possibility. As known in the art, the term
"scalability" means a coder/decoder concept where the coder
provides a scaled data stream. The scaled data stream
comprises a base scaling layer as well as one or several
enhancement scaling layers. The base scaling layer
20 comprises a representation of the signal to be coded,
generally with lower quality, but also with lower data
rate. The enhancement scaling layer contains a further
representation of the video signal, which provides a
representation with improved quality with regard to the
25 base scaling layer, typically together with the
representation of the video signal in the base scaling
layer. On the other hand, the enhancement scaling layer
has, of course, individual bit requirements, so that the
number of bits for representing the signal to be coded
30 increases with every enhancement layer.
Depending on design and possibilities, a decoder will
decode, either only the base scaling layer to provide
comparatively qualitatively bad representation of the
35 picture signal represented by the coded signal. With every
"addition" of a further scaling layer, however, the decoder
can improve the quality of the signal step by step (at the
expense of the bit rate).

- 9 -
Depending on the implementation and the transmission
channel from a coder to a decoder, at least the base
scaling layer is transmitted, since the bit rate of the
5 base scaling layer is typically so low that also a so far
limited transmission channel will be sufficient. If the
transmission channel allows no more bandwidth for the
application, only the base scaling layer but no enhancement
scaling layer will be transmitted. As a consequence, the
10 decoder can generate merely a low quality representation of
the picture signal. Compared to the unscaled case, where
the data rate would have been so high that a transmission
system would not have been possible, the low quality
representation is advantageous. If the transmission channel
15 allows the transmission of one or several enhancement
layers, the coder will transmit one or several enhancement
layers to the decoder, so that it can increase the quality
of the output video signal step by step, depending on the
request.
20
With regard to the coding of video sequences, two different
scalings can be distinguished. One scaling is a temporal
scaling, in so far that not all video frames of a video
sequence are transmitted, but that for reducing the data
25 rate, for example, only every second frame, every third
frame, every fourth frame, etc. is transmitted.
The other scaling is the SNR scalability (SNR = signal to
noise ratio), wherein every scaling layer, e.g. both the
30 base scaling layer and the first, second, third,
enhancement scaling layer comprise all time information,
but with varying quality. Thus, the base scaling layer
would have a low data rate, but a low signal noise ratio,
wherein this signal noise ratio can then be improved step
35 by step by adding one enhancement scaling layer each.
The coder concept illustrated in Fig. 9 is problematic in
that it is based on the fact that merely residual values

- 10 -
are generated by the subtracter 84, and are then processed.
These residual values are calculated based on prediction
algorithms, in the arrangement shown in Fig. 9, which forms
a closed loop by using the blocks 86, 87, 88, 89, 93, 94
5 and 84, wherein a quantization parameter enters the closed
loop, which means in blocks 86, 87. If now a simple SNR
scalability would be implemented in that for example every
predicted residual signal is quantized first with a coarse
quantizer step width, and then quantized step by step with
10 finer quantizer step widths, by using enhancement layers,
this would have the following consequences. Due to the
inverse quantization and the prediction, particularly with
regard to the motion estimation (block 91) and the motion
compensation (block 92), which take place by using the
15 original picture on the one hand and the quantized picture
on the other hand, a "diverging" of the quantizer step
widths results both in the coder and the decoder. This
leads to the fact that the generation of the enhancement
scaling layers on the coder side becomes very problematic.
20 Further, processing the enhancement scaling layers on the
decoder side becomes impossible, at least with regard to
the elements defined in the standard H.264/AVC. The reason
therefore is the closed loop in the video coder illustrated
with regard to Fig. 9, wherein the quantization is
25 contained.
In the standardization document JVT-I 032 tl titled "SNR-
Scalable Extension of H.264/AVC", Heiko Schwarz, Detlev
Marpe and Thomas Wiegand, presented in the ninth JVT
30 meeting from 2nd to 5th December 2003 in San Diego, a
scalable extension to H.264/AVC is presented, which
comprises a scalability both with regard to time and signal
noise ratio (with equal or different temporal accuracy).
Therefore, a lifting representation of time subband
35 partitions is introduced, which allows the usage of known
methods for motion-compensated prediction.

- 11 -
Wavelet based video coder algorithms, wherein lifting
implementations are used for the wavelet analysis and for
wavelet synthesis, are described in J.-R. Ohm, ,,Complexity
and delay analysis of MCTF interframe wavelet structures",
5 ISO/IECJTC1/WG11 Doc.M8520, July 2002. Comments on
scalability can also be found in D. Taubman, ,,Successive
refinement of video: fundamental issues, past efforts and
new directions", Proc. of SPIE (VCIP'03), vol. 5150, pp.
649-663, 2003, wherein, however, significant coder
10 structure alterations are required. According to the
invention, a coder/decoder concept is achieved, which has,
on the one hand, the scalability possibility and can, on
the other hand, be based on elements in conformity with the
standard, particularly, e.g., for the motion compensation.
15
Before reference will be made in more detail to a
coder/decoder structure with regard to Fig. 3, first, a
basic lifting scheme on the side of the coder and an
inverse lifting scheme on the side of the decoder,
20 respectively, will be illustrated with regard to Fig. 4.
Detailed explanations about the background of the
combination of lifting schemes and wavelet transformations
can be found in W. Sweldens, ,,A custom design construction
of biorthogonal wavelets'", J. Appl. Comp. Harm. Anal., vol.
25 3 (no. 2), pp. 186-200, 1996 and I. Daubechies and W.
Sweldens, ,,Factoring wavelet transforms into lifting
Steps", J. Fourier Anal. Appl., vol. 4 (no.3), pp. 247-269,
1998. Generally, the lifting scheme consists of three
steps, the polyphase decomposition step, the prediction
30 step and the update step.
The decomposition step comprises partitioning the input
side data stream into an identical first copy for a lower
branch 40a as well as an identical copy for an upper branch
35 40b. Further, the identical copy of the upper branch 40b is
delayed by a time stage (z-1), so that a sample S2k+1 with an
odd index k passes through a respective decimator and
downsampler 42a, 42b, respectively, at the same as a sample

- 12 -
with an even index S2k. The decimator 42a and 42b,
respectively, reduces the number of samples in the upper
and the lower branch 40b, 40a, respectively, by eliminating
every second sample.
5
The second region II, which relates to the prediction step,
comprises a prediction operator 43 as well as a subtracter
44. The third region, which means the update step,
comprises an update operator 45 as well as an adder 46. On
10 the output side, two normalizers 47, 48 exist, for
normalizing the high-pass signal hk (normalizer 47) and for
normalizing the low-pass signal 1k through the normalizer
48.

As a result of the prediction step, the odd samples are
replaced by their respective prediction residual values
15 Particularly, the polyphase decomposition leads to the
partitioning of even and odd samples of a given signal
s[k]. Since the correlation structure typically shows a
local characteristic, the even and odd polyphase components
are highly correlated. Thus, in a final step, a prediction
20 (P) of the odd samples is performed by using the integer
samples. The corresponding prediction operator (P) for
every odd sample Sodd[k] = s[2k+1] is a linear combination of
the adjacent even samples Seven[k]=S[2k],i.e.

It should be noted that the prediction step is equivalent
to performing a high-pass filter of a two channel filter
bank, as it is illustrated in I. Daubechies and W.
35 Sweldens, ,,Factoring wavelet transforms into lifting
steps", J. Fourier Anal. Appl. vol 4 (no.3), pp. 247-269,
1998.

13

In the third step of the lifting scheme, low-pass filtering
is performed, by replacing the even samples seven[k] by a
linear combination of prediction residual values h[k]. The
respective update operator U is given by

By replacing the even samples with

the given signal s[k] can finally be represented by l(k)
and h(k), wherein every signal has half the sample rate.
Since both the update step and the prediction step are
15 fully invertible, the corresponding transformation can be
interpreted as critically sampled perfect reconstruction
filter bank. Indeed, it can be shown that any biorthogonal
family of wavelet filters can be realized by a sequence of
one or several prediction steps and one or several update
20 steps. For a normalization of low-pass and high-pass
components, the normalizers 47 and 48 are supplied with
suitably chosen scaling factors F1 and Fh, as has been
explained.
25 The inverse lifting scheme, which corresponds to the
synthesis filter bank, is shown in Fig. 4 on the right hand
side. It consists simply of the application of the
prediction and update operator in inverse order and with
inverse signs, followed by the reconstruction by using the
30 even and odd polyphase components. Specifically, the right
decoder shown in Fig. 4 comprises again a first decoder
region I, a second decoder region II as well as a third
decoder region III. The first decoder region cancels the
effect of the update operator 45. This is effected by
35 supplying the high-pass signal, which has been re-
normalized by a further normalizer 50, to the update
operator 45. Then, the output signal of the decoder side
update operator 45 is supplied to a subtracter 52, in

- 14 -
contrary to the adder 46 in Fig. 4. Correspondingly, the
output signal of the predictor 43 is processed, the output
signal of which is now supplied to an adder 53 and not to a
subtracter as on the coder side. Now, an upsampling of the
5 signal by the factor 2 takes place in every branch (blocks
54a, 54b). Then, the upper branch is shifted by one sample
into the future, which is equivalent to delaying the lower
branch, to perform then an addition of the data streams on
the upper branch and the lower branch in an adder 55, to
10 obtain the reconstructed signal sk at the output of the
synthesis filter bank.
Several wavelets can be implemented by the predictor 43 and
the update-operator 45, respectively. If the so-called hair
15 wavelet is to be implemented, the prediction operator and
the update operator are given by the following equation:


25 correspond to the non-normalized high-pass and low-pass
(analysis) output signal, respectively, of the hair filter.
In the case of the 5/3 biorthogonal spline wavelet, the
low-pass and high-pass analysis filter of this wavelet have
30 5 and 3 filter taps, respectively, wherein the
corresponding scaling function is a second order B spline.
In coder applications for still pictures, such as JPEG
2000, this wavelet is used for a time subband coder scheme.
In a lifting environment, the corresponding prediction and
35 update operators of the 5/3 transformation are given as
follows:


- 15 -
Fig. 3 shows a block diagram of a coder/decoder structure
with exemplary four filter levels both on the side of the
coder and on the side of the decoder. From Fig. 3, it can
5 be seen that the first filter level, the second filter
level, the third filter level and the fourth filter level
are identical with regard to the coder. The filter levels
with regard to the decoder are also identical. On the coder
side, every filter level comprises a backward predictor M10
10 as well as a forward predictor M11 61 as central elements.
The backward predictor 60 corresponds in principle to the
predictor 43 of Fig. 4, while the forward predictor 61
corresponds to the update operator of Fig. 4.
15 In contrary to Fig. 4, it should be noted that Fig. 4
relates to a stream of samples, where a sample has an odd
index 2k+l, while another sample has an even index 2k.
However, as has already been explained with regard to Fig.
1, the notation in Fig. 3 relates to a group of pictures
20 instead of to a group of samples. If a picture has for
example a number of samples and pictures, respectively,
this picture is fed in fully. Then, the next picture is fed
in, etc. Thus, there are no longer odd and even samples,
but odd and even pictures. According to the invention, the
25 lifting scheme described for odd and even samples is
applied to odd and even pictures, respectively, each of
which has a plurality of samples. Now, the sample by sample
predictor 4 3 of Fig. 4 becomes the backward motion
compensation prediction 60, while the sample by sample
30 update operator 4 5 becomes the picture by picture forward
motion compensation prediction 61.
It should be noted that the motion filters, which consist
of motion vectors and represent coefficients for the block
35 60 and 61, are calculated for two subsequent related
pictures and are transmitted as side information from coder
to decoder. However, it is a main advantage of the
inventive concept that the elements 91, 92, as they are

- 16 -
described with reference to Fig. 9 and standardized in
standard H.264/AVC, can easily be used to calculate both
the motion fields M10 and the motion fields M11. Thus, no
new predictor/update operator has to be used for the
5 inventive concept, but the already existing algorithm
mentioned in the video standard, which is examined and
checked for functionality and efficiency, can be used for
the motion compensation in forward direction or backward
direction.
10
Particularly, the general structure of the used filter bank
illustrated in Fig. 3 shows a temporal decomposition of the
video signal with a group of 16 pictures, which are fed in
at an input 64. The decomposition is a dyadic temporal
15 decomposition of the video signal, wherein in the
embodiment shown in Fig. 3 with four levels 24=16 pictures,
which means a group size of 16 pictures, is required to
achieve the representation with the smallest temporal
resolution, which means the signals at the output 28a and
20 at the output 28b. Thus, if 16 pictures are grouped, this
leads to a delay of 16 pictures, which makes the concept
shown in Fig. 3 with four levels rather problematic for
interactive applications. Thus, if interactive applications
are aimed at, it is preferred to form smaller groups of
25 pictures, such as to group four or eight pictures. Then,
the delay is correspondingly reduced, so that the usage for
interactive applications becomes possible. In cases where
interactivity is not required, such as for storage
purposes, etc., the number of pictures in a group, which
30 means the group size, can be correspondingly increased,
such as to 32, 64, etc. pictures.
In that way, an interactive application of the hair-based
motion-compensated lifting scheme is used, which consists
35 of the backward motion compensation prediction (M10) , as in
H.264/AVC, and that further comprises an update step, which
comprises a forward motion compensation (M11) . Both the
prediction step and the update step use the motion

- 17 -
compensation process, as it is illustrated in H.264/AVC.
Further, not only the motion compensation is used, but also
the deblocking filter 89 designated with the reference
number 8 9 in Fig. 9.
5
The second filter level comprises again downsampler 66a,
66b, a subtracter 69, a backward predictor 67, a forward
predictor 68 as well as an adder 70 and a further
processing means to output the first and second high-pass
10 picture of the second level at an output of the further
processing means, while the first and second low-pass
picture of the second level are output at the output of the
adder 70.
15 Additionally, the coder in Fig. 3 comprises a third level
as well as a fourth level, wherein a group of 16 pictures
is fed into the fourth-level input 64. At a fourth-level
high-pass output 72, which is also referred to as HP4,
eight high-pass pictures quantized with a quantization
20 parameter Q and correspondingly processed are output.
Correspondingly, eight low-pass pictures are output at a
low-pass output 73 of the fourth filter level, which is fed
into an input 74 of the third filter level. This level,
again, is effective to generate four high-pass pictures at
25 a high-pass output 75, which is also referred to as HP3,
and to generate four low-pass pictures at a low-pass output
76, which are fed into the input 10 of the second filter
level and decomposed.
30 It should particularly be noted that the group of pictures
processed by a filter level does not necessarily have to be
video pictures originating from an original video sequence,
but can also be low-pass pictures, which are output by a
next higher filter level at a low-pass output of the filter
35 level.
Further, it should be noted that the coder concept shown in
Fig. 3 for 16 pictures can easily be reduced to eight

- 18 -
pictures, when simply the fourth filter level is omitted
and the group of pictures is fed into the input 74. In the
same way, the concept shown in Fig. 3 can also be extended
to a group of 32 pictures, by adding a fifth filter level
5 and by outputting then 16 high-pass pictures at a high-pass
output of the fifth filter level and feeding the sixteen
low-pass pictures at the output of the fifth filter level
into the input 64 of the fourth filter level.
10 The tree-like concept of the coder side is also applied to
the decoder side, but now no longer, like on the coder
side, from the high level to the lower level but, on the
decoder side, from the lower level to the higher level.
Therefore, the data stream is received from a transmission
15 medium, which is schematically referred to as network
abstraction layer 100, and the received bit stream is first
subject to an inverse further processing by using the
inverse further processing means, to obtain a reconstructed
version of the first high-pass picture of the first level
20 at the output of means 30a and a reconstructed version of
the first-level low-pass picture at the output of block 30b
of Fig. 3. Then, analogous to the right half of Fig. 4,
first the forward motion compensation prediction is
reversed via the predictor 61, to subtract then the output
25 signal of the predictor 61 from the reconstructed version
of the low-pass signal (subtracter 101).
The output signal of the subtracter 101 is fed into a
backward compensation predictor 60 to generate a prediction
30 result, which is added to the reconstructed version of the
high-pass picture in an adder 102. Then, both signals,
which means the signals in the lower branch 103a, 103b, are
brought to the double sample rate, by using the upsampler
104a, 104b, wherein then the signal on the upper branch is
35 either delayed or "accelerated", depending on the
implementation. It should be noted that the upsampling is
performed by the bridge 104a, 104b simply by inserting a
number of zeros which corresponds to the number of samples

- 19 -
for a picture. The shift by the delay of a picture by the
element shown with z-1 in the upper branch 103b against the
lower branch 103a effects that the addition by an adder 106
causes that the two second-level low-pass pictures occur
5 subsequently on the output side with regard to the adder
106.
The reconstructed versions of the first and second second-
level low-pass picture are then fed into the decoder-side
10 inverse filter of the second level and there they are
combined again with the transmitted second-level high-pass
pictures by the identical implementation of the inverse
filter bank to obtain a sequence of four third-level low-
pass pictures at an output 101 of the second level. The
15 four third-level low-pass pictures are then combined in an
inverse filter level of the third level with the
transmitted third-level high-pass pictures to obtain eight
fourth-level low-pass pictures in subsequent format at an
output 110 of the inverse third-level filter. These eight
20 third-level low-pass pictures will then be combined again
with the eight fourth-level high-pass pictures received
from the transmission medium 100 via the input HP4, in an
inverse fourth-level filter, as discussed with regard to
the first level, to obtain a reconstructed group of 16
25 pictures at an output 112 of the inverse fourth-level
filter.
Thus, in every stage of the analysis filter bank, two
pictures, either original pictures or pictures representing
30 low-pass signals and generated in a next higher level, are
decomposed into a low-pass signal and a high-pass signal.
The low-pass signal can be considered as representation of
the common characteristics of the input pictures, while the
high-pass signal can be considered as representation of the
35 differences between the input pictures. In the
corresponding stage of the synthesis filter bank, the two
input pictures are again reconstructed by using the low-
pass signal and the high-pass signal.

- 20 -
Since the inverse operations of the analysis step are
performed in the synthesis step, the analysis/synthesis
filter bank (without quantization, of course) guarantees a
5 perfect reconstruction.
The only occurring losses occur due to the quantization in
the further processing means, such as 26a, 26b, 18. If
quantization is performed very finely, a good signal noise
10 ratio is achieved. If, however, quantization is performed
very coarsely, a relatively bad signal noise ratio is
achieved, but with a low bit rate, which means low demand.
Without SNR scalability, a time scaling control could be
15 implemented already with the concept shown in Fig. 3.
Therefore, a time scaling control 120 is used, which is
formed to obtain the high-pass and low-pass output,
respectively, and the outputs of the further processing
means (26a, 26b, 18 ...) , respectively, at the input side to
20 generate a scaled data stream from these partial data
streams TPl, HP1, HP2, HP3, HP4, which has the processed
version of the first low-pass picture and the first high-
pass picture in a base scaling layer. Then, the processed
version of the second high-pass picture could be
25 accommodated in a first enhancement scaling layer. The
processed versions of the third-level high-pass pictures
could be accommodated in a second enhancement scaling
layer, while the processed versions of the fourth-level
high-pass pictures are introduced in a third enhancement
30 scaling layer. Thereby, merely based on the base scaling
layer, a decoder could already generate a sequence of
lower-level low-pass pictures with a lower time quality,
which means two first-level low-pass pictures per group of
pictures. With the addition of every enhancement scaling
35 layer, the number of reconstructed pictures per group can
always be doubled. The functionality of the decoder is
typically controlled by a scaling control, which is formed
to detect how many scaling layers are contained in the data

- 21 -
stream and how many scaling layers have to be considered by
the decoder during decoding, respectively.
The JVT document JVT-J 035 with the title "SNR-Scalable
5 Extension of H.264/AVC" Heiko Schwarz, Detlev Marpe and
Thomas Wiegand, presented during the tenth JVT meeting in
Waikoloa Hawaii, 8th to 12th December 2003, shows a SNR
scalable extension of the temporal decomposition scheme
illustrated in Figs. 3 and 4. Particularly, a time scaling
10 layer is partitioned into individual "SNR scaling
sublayers", wherein a SNR base layer is obtained in such
that a certain time scaling layer is quantized with a first
coarser quantizer step width to obtain the SNR base layer.
Then, among other things, an inverse quantization is
15 performed, and the result signal from the inverse
quantization is subtracted from the original signal to
obtain a difference signal, which is then quantized with a
finer quantizer step width to obtain the second scaling
layer. However, the second scaling layer is requantized
20 with the finer quantizer step width to subtract the signal
obtained after the requantization from the original signal
to obtain a further difference signal, which, again after
quantization, but now with a finer quantizer step width,
represents a second SNR scaling layer and an SNR
25 enhancement layer, respectively.
Thus, it has been found out that the above described
scalability schemes, which are based on the motion-
compensated temporal filtering (MCTF) , already provide a
30 high flexibility with regard to the temporal scalability
and also the SNR scalability. But there is still a problem
in that the bit rate of several scaling layers together is
still significantly above the bit rate, which can be
achieved when pictures of the highest quality would be
35 coded without scalability. Due to the side information for
the different scaling layers, scalable coders might never
obtain the bit rate of the unscaled case. However, the bit
rate of a data stream with several scaling layers should

- 22 -
approach the bit rate of the unsealed case as closely as
possible.
Further, the scalability concept should provide high
5 flexibility for all scalability types, which means a high
flexibility both with regard to time and space and also
with regard to SNR.
The high flexibility is particularly important where
10 already pictures with low resolution would be sufficient
but a higher temporal resolution is desirable. Such a
situation results, for example, when fast changes exist in
pictures, such as, for example, in videos of team sports,
where additionally to the ball, many persons move at the
15 same time.
It is the object of the present invention to provide a
concept for flexible coding/decoding, which provides a bit
rate, which is as low as possible, despite the fact that it
20 is a scalable concept.
This object is achieved by an apparatus for generating a
coded video sequence in accordance with claim 1, a method
for generating a coded video sequence in accordance with
25 claim 15, an apparatus for decoding a coded video sequence
in accordance with claim 16, a method for decoding a coded
video sequence in accordance with claim 26, a computer
program in accordance with claim 27 or a computer-readable
medium in accordance with claim 28.
30
The present invention is based on the knowledge that bit
rate reductions can be obtained not only with a motion-
compensated prediction performed within a scaling layer,
but that further bit rate reductions with constant picture
35 quality can be obtained by performing an intermediate
scaling layer prediction of the residual pictures after the
motion-compensated prediction of a lower layer, such as the

- 23 -
base layer, to a higher layer, such as the enhancement
layer.
It has been found out that within the same temporal scaling
5 layer, the residual values of the individual considered
other scaling layers, which are preferably scaled with
regard to resolution or with regard to the signal noise
ratio also have correlations between the residual values
after the motion-compensated prediction. According to the
10 invention, these correlations are advantageously used by
providing an intermediate layer predictor on the coder side
for the enhancement scaling layer, which corresponds to an
intermediate layer combiner on the decoder side.
Preferably, this intermediate layer predictor is designed
15 adaptively, to decide, e.g., for every macroblock whether
an intermediate layer prediction is worthwhile or whether
the prediction would rather lead to a bit rate increase.
The latter is the case when the prediction residual signal
becomes larger than the original motion compensation
20 residual signal of the enhancement layer with regard to a
subsequent entropy coder. However, this situation will not
occur in many cases, so that the intermediate layer
predictor is activated and leads to a significant bit rate
reduction.
25
Further, in a preferred embodiment of the present
invention, a prediction of the motion data of the
enhancement layer is also performed. Thus, it has further
shown that within different quality scaling layers, such as
30 with regard to as SNR or resolution, the motion fields in
different scaling layers also have correlations to one
another, which can be used advantageously for bit rate
reduction according to the invention by providing a motion
data predictor. In the implementation, the prediction can
35 be performed in that for no individual motion data are
calculated the enhancement layer, but the motion data of
the base layer are transmitted, eventually after an
upsampling. This can, however, lead to the fact that the

- 24 -
motion compensation residual signal in the enhancement
layer becomes larger than in the case where the motion data
are calculated specially for the enhancement layer.
However, this disadvantage does make no difference when the
5 saving due to the motion data saved for the enhancement
layer during the transmission is larger than the bit rate
increase caused by possibly larger residual values.
However, an individual motion field can be calculated for
10 the enhancement layer in the implementation, wherein the
motion field of the base layer is integrated into the
calculation or used as predictor to transmit only motion
field residual values. This implementation has the
advantage that the motion data correlation of the two
15 scaling layers is fully utilized and that the residual
values of the motion data are as small as possible after
the motion data prediction. The disadvantage of this
concept however, is the fact that additional motion data
residual values have to be transmitted.
20
In the preferred embodiment of the present invention,
additionally, a SNR scalability is used. This means that
quantization is performed in the base layer with a coarser
quantization parameter than in the enhancement layer. The
25 residual values of the base motion prediction quantized
with the coarser quantizer step width and reconstructed
again are thereby used as prediction signals for the
intermediate layer predictor. In the case of a pure SNR
scalability, it can be sufficient to calculate a single
30 motion field for all scaling layers on the coder side. With
reference to the motion data of the enhancement layer, this
means again that no further enhancement motion data have to
be transmitted, but that the enhancement motion data from
the base layer can be used fully on the coder side for the
35 inverse motion compensation for the enhancement layer.
However, different quantization parameter result in
different motion fields, when a calculation of the motion

- 25 -
data is used, into which the quantization parameter is
introduced.
If a spatial scalability is used, which means the base
5 scaling layer has a coarser spatial resolution than the
enhancement scaling layer, it is preferred to interpolate
the residual values of the base motion prediction, which
means to convert from the lower spatial resolution of the
enhancement scaling layer and to provide it then to the
10 intermediate layer predictor.
Further, it is preferred to perform an individual
calculation of the motion information for every scaling
layer. However, in a preferred embodiment of the present
15 invention, a motion data prediction is used here for a data
rate reduction, which can consist again either in the
complete transmission of the motion data of the lower
scaling layer (after scaling), or which can consist of
using the upsampled motor vectors of the lower scaling
20 layer for predicting the motion vectors of the higher
scaling layer, to transmit then only the motion data
residual values which will require a lower data rate than
non-predicted motion data. In this case, it is preferred to
design both the intermediate layer predictor and an
25 enhancement motion data predictor adaptively.
In a preferred embodiment of the present invention, a
combined scalability is used in that the base scaling layer
and the enhancement scaling layer differ both in spatial
30 resolution and in the used quantization parameter, which
means the used quantizer step width. In this case,
starting, e.g., from a previous quantization parameter for
the base scaling layer due to a Lagrange optimization, a
combination of quantization parameter for the base layer,
35 distortion and bit requirement for the motion data for the
base layer is calculated. The residual values obtained
after a motion-compensated prediction and the base motion
data used thereby will then be used for a prediction of the

- 26 -
respective data of a higher scaling layer, wherein again
starting from a finer scaling parameter for the higher
scaling layer, an optimum combination of bit requirement
for the motion data, quantization parameter and distortion,
5 enhancement motion data can be calculated.
Preferred embodiments of the present invention will be
explained in the following with reference to the
accompanying drawings, in which:
10
Fig. la is a preferred embodiment of an inventive coder;
Fig. lb is a detailed representation of a base picture
coder of Fig. la;
15
Fig. lc is a discussion of the functionality of an
intermediate layer prediction flag;
Fig. Id is a description of a motion data flag;
20
Fig. le is a preferred implementation of the enhancement
motion compensator 1014 of Fig. la;
Fig. 1f is a preferred implementation of the enhancement
25 motion data determination means 1078 of Fig. 2;
Fig. lg is an overview representation of three preferred
embodiments for calculating the enhancement motion
data and for enhancement motion data processing
30 for the purpose of signalization and residual data
transmission, if necessary;
Fig. 2 is a preferred embodiment of an inventive decoder;
35 Fig. 3 is a block diagram of a decoder with four levels;
Fig. 4 is a block diagram for illustrating the lifting
decomposition of a time subband filter bank;

- 27 -
Fig. 5a is a representation of the functionality of the
lifting scheme shown in Fig. 4;
5 Fig. 5b is a representation of two preferred lifting
specifications with unidirectional prediction
(hair wavelet) and bidirectional prediction (5/3
transformation);
10 Fig. 5c is a preferred embodiment of the prediction and
update operators with motion compensation and
reference indices for an arbitrary choice of the
two pictures to be processed by the lifting
scheme;
15
Fig. 5d is a representation of the intra mode where
original picture information can be inserted
macroblock by macroblock into high-pass pictures;
20 Fig. 6a is a schematic representation for signalizing a
macroblock mode;
Fig. 6b is a schematic representation for upsampling of
motion data in a spatial scalability according to
25 a preferred embodiment of the present invention;
Fig. 6c is a schematic representation of the data stream
syntax for motion vector differences;
30 Fig. 6d is a schematic representation of a residual value
syntax enhancement according to a preferred
embodiment of the present invention;
Fig. 7 is an overview diagram for illustrating the time
35 shift of a group of, for example, 8 pictures;
Fig. 8 is a preferred time placement of low-pass pictures
for a group of 16 pictures;

- 28 -
Fig. 9 is an overview block diagram for illustrating the
basic coder structure for a coder according to the
standard H.2 64/AVC for a macroblock;
5
Fig. 10 is a context arrangement consisting of two
adjacent pixel elements A and B on the left and
above a current syntax element C, respectively,
and
10
Fig. 11 is a representation of the partition of a picture
into slices.
Fig. la shows a preferred embodiment of an apparatus for
15 generating a coded video sequence, which has a base scaling
layer and an enhancement scaling layer. An original video
sequence with a group of 8, 16 or any number of pictures is
fed in via an input 1000. On the output side, the coded
video sequence contains the base scaling layer 1002 and the
20 enhancement scaling layer 1004. The enhancement scaling
layer 1004 and the base scaling layer 1002 can be supplied
to a bit stream multiplexer, which generates a single
scalable bit stream on the output side. Depending on the
implementation, however, a separate transmission of the two
25 scaling layers is also possible and useful in some cases.
Fig. la shows a coder for generating two scaling layers,
which means the base scaling layer and an enhancement
scaling layer. In order to obtain a coder, which, if
necessary, generates one or several further enhancement
30 layers, the functionality of the enhancement scaling layer
is to be repeated, wherein a higher enhancement scaling
layer is always supplied with data by the next lower
enhancement scaling layer, as the enhancement scaling layer
1004 shown in Fig. 1 is supplied with data by the base
35 scaling layer 1002.
Before reference will be made to different scaling types in
detail, such as a SNR scalability or a spatial scalability

- 29 -
or a combined scalability of spatial and SNR scalability,
first, the basic principle of the present invention will be
illustrated. First, the coder comprises a base motion
compensator or base motion estimator 1006 for calculating
5 base motion data, which indicates how a macroblock has
moved in a current picture in relation to another picture
in a group of pictures, which the base motioned compensator
1006 obtains on the input side. Techniques for calculating
motion data, particularly for calculating a motion vector
10 for a macroblock, which is basically a region of pixels in
a digital video picture, are known. Preferably, the motion
compensation calculation is used, as it is standardized in
the video coding standard H.264/AVC. Thereby, a macroblock
of a later picture is considered and it is determined, how
15 the macroblock "moved" in comparison to an earlier picture.
This motion (in xy direction) is indicated by a two-
dimensional motion vector, which is calculated by block
1006 for every macroblock and supplied to a base picture
coder 1010 via a motion data line 1008. Then, it is
20 calculated for the next picture, how a macroblock has moved
from the previous picture to the next picture.
In one implementation, this new motion vector, which, in a
way, indicates the motion from second to a third picture,
25 can be transmitted again as two-dimensional vector. For
efficiency reasons, however, it is preferred to transmit
only a motion vector difference, which means the difference
of the motion vector of a macroblock from the second to the
third picture and the motion vector of the macroblock from
30 the first to the second picture. Alternative referencings
and motion vector differences, respectively, to not
immediately previous pictures, but to further preceding
pictures can also be used.
35 The motion data, which have been calculated by block 1006,
will then be supplied to a base motion predictor 1012,
which is designed to calculate a base sequence of residual
error pictures for using the motion data and the group of

- 30 -
pictures. Thus, the base motion predictor performs the
motion compensation, which has, in a way, been prepared by
the motion compensator and motion estimator, respectively.
This base sequence of residual error pictures will then be
5 supplied to the base picture coder. The base picture coder
is formed to provide the base scaling layer 1002 at its
output.
Further, the inventive coder comprises an enhancement
10 motion compensator or enhancement motion estimator 1014 for
detecting enhancement motion data. These enhancement motion
data are supplied to an enhancement motion predictor 1016,
which generates an enhancement sequence of residual error
pictures on the output side and supplies them to a
15 downstream intermediate layer predictor 1018. Thus, the
enhancement motion predictor performs the motion
compensation, which, in a way, has been prepared by the
motion compensator and motion estimator, respectively.
20 The intermediate layer predictor is formed to calculate
enhancement prediction residual error pictures on the
output side. Depending on the implementation, the
intermediate layer predictor uses additionally to the data,
which it obtains from block 1016, which means additionally
25 to the enhancement sequence of residual error pictures, the
base sequence of residual error pictures, as it is provided
by block 1012 via a dotted bypass line 1020. Alternatively,
the block 1018 can also use an interpolated sequence of
residual error pictures, which is provided at the output of
30 block 1012 and interpolated by an interpolator 1022. Again
alternatively, the intermediate layer predictor can also
provide a reconstructed base sequence of residual error
pictures, as it is provided to an output 1024 of the base
picture coder 1010. As can be seen from Fig. la, this
35 reconstructed base sequence of residual error pictures can
be interpolated 1022 or not interpolated 1020. Thus,
generally, the intermediate layer predictor operates by
using the base sequence of residual error pictures, wherein

- 31 -
the information at the intermediate layer predictor input
1026 is derived, e.g. by a reconstruction or interpolation
of the base sequence of residual error pictures at the
output of block 1012.
5
Downstream to the intermediate layer predictor 1018, there
is an enhancement picture coder 1028, which is formed to
code the enhancement prediction residual error pictures to
obtain the coded enhancement scaling layer 1004.
10
In a. preferred embodiment of the present invention, the
intermediate layer predictor is formed to subtract the
signal at its output 1026 macroblock by macroblock and
picture by picture from the respective signal, which the
15 intermediate layer predictor 1018 obtains from the
enhancement motion predictor 1016. The result signal
obtained in this subtraction represents then a macroblock
of a picture of the enhancement prediction residual error
pictures.
20
In a preferred embodiment of the present invention, the
intermediate layer predictor is formed adaptively. For
every macroblock, an intermediate layer prediction flag
1030 is provided, which indicates the intermediate layer
25 predictor that it has to perform a prediction, or which
indicates in its other state that no prediction is to be
performed, but that the corresponding macroblock at the
output of the enhancement motion predictor 1016 is to be
supplied to the enhancement picture coder 1028 without
30 further prediction. This adaptive implementation has the
advantage that an intermediate layer prediction is only
performed where it is useful, where the prediction residual
signal leads to a lower output picture rate compared to the
case where no intermediate layer prediction has been
35 performed, but where the output data of the enhancement
motion predictor 1016 have been coded directly.

- 32 -
In the case of a spatial scalability, a decimator 1032 is
provided between the enhancement scaling layer and the base
scaling layer, which is formed to convert the video
sequence at its input, which has a certain spatial
5 resolution, to a video sequence at its output, which has a
lower resolution. If a pure SNR scalability is intended,
which means if the base picture coder 1010 and 1028 for the
two scaling layers operate with different quantization
parameters 1034 and 1036, respectively, the decimator 1032
10 is not provided. This is illustrated schematically in Fig.
la by the bypass line 1038.
Further, in the case of spatial scalability, the
interpolator 1022 has to be provided. In the case of a pure
15 SNR scalability, the interpolator 1022 is not provided.
Instead, the bypass line 1020 is taken, as illustrated in
Fig. la.
In one implementation, the enhancement motion compensator
20 1014 is formed to fully calculate an individual motion
field, or to use the motion field calculated by the base
motion compensator 1006 directly (bypass line 1040) or
after upsampling by an upsampler 1042. In the case of a
spatial scalability, the upsampler 1042 has to be provided
25 to upsample a motion vector of the base motion data to the
higher resolution, which means, for example, to scale. If,
for example, the enhancement resolution is twice as high
and wide as the base resolution, a macroblock (16x16
luminance samples) in the enhancement layer covers a region
30 of a picture, which corresponds to a sub-macroblock (8x8
luminance samples) in the base layer.
Thus, in order to be able to use the base motion vector for
the macroblock of the enhancement scaling layer, the base
35 motion vector is doubled in its x component and its y
component, which means scaled by the factor 2. This will be
discussed in more detail with reference to Fig. 6b.

- 33 -
If, however, there is merely an SNR scalability, the motion
field is the same for all scaling layers. Thus, it has to
be calculated only once and can be directly used by every
higher scaling layer in the way it has been calculated by
5 the lower scaling layer.
For intermediate layer prediction, the signal at the output
of the base motion predictor 1012 can also be used.
Alternatively, the reconstructed signal on line 1024 can be
10 used. The selection, which of these two signals is used for
prediction, is made by a switch 1044. The signal on line
1024 differs from the signal at the output of block 1012 by
the fact that it has already experienced a quantization.
This means that the signal on line 1024 has a quantization
15 error in comparison to the signal at the output of block
1012. The alternative of using the signal on line 1024 for
intermediate layer prediction is particularly advantageous
when an SNR scalability is either used alone or in
connection with a spatial scalability, since then the
20 quantization error made by the base picture coder 1010 is
then "taken along" to the higher scaling layer, since the
output signal at block 1018 will then contain the
quantization error made by the first scaling layer, which
will then be quantized at the input 1036 by the enhancement
25 picture coder with a typically finer quantizer step width
and a changed quantization parameter 2, respectively, and
will be written into the enhancement scaling layer 1004.
Analogous to the intermediate layer prediction flag 1030, a
30 motion data flag 1048 is fed into the picture coder, so
that a corresponding information about that is contained in
the enhancement scaling layer 1004, which will then be used
by the decoder, which will be discussed with reference to
Fig. 2.
35
If a pure spatial scalability is used, the output signal of
the base motion predictor 1012, which means the base
sequence of residual error pictures, can be used instead of

- 34 -
the signal on line 1024, which means instead of the
reconstructed sequence of base residual error pictures.
Depending on the implementation, the control of this switch
5 can take place manually or based on a prediction benefit
function.
Here, it should be noted that preferably all predictions,
which means the motion prediction, the enhancement motion
10 data prediction and the intermediate layer residual value
prediction are designed adaptively. This means that motion
data prediction residual values do not necessarily have to
be present for every macroblock or sub-macroblock in a
picture of the base sequence of residual error pictures,
15 for example. Thus, a picture of the base sequence of
residual error pictures can also contain non-predicted
macroblocks and sub-macroblocks, respectively, despite the
fact that it is referred to as "residual error picture".
This situation will occur when it has been found out that,
20 e.g., a new object occurs in a picture. Here, a motion-
compensated prediction would be useless, since the
prediction residual signal would become larger than the
original signal in the picture. In the enhancement motion
prediction in block 1016, in such a case, both the
25 prediction operator and eventually the update operator for
this block (e.g. macroblock or sub-macroblock) would be
deactivated.
Still, for clarity reasons, e.g. a base sequence of
30 residual error pictures is mentioned, despite maybe only a
single residual error picture of the base sequence of
residual error pictures has a single block, which actually
includes motion prediction residual signals. In typical
application cases, however, every residual error picture
35 will actually have a high number of blocks with motion
prediction residual data.

- 35 -
In the sense of the present invention, this applies also
for the enhancement sequence of residual error pictures. In
that way, the situation in the enhancement layer will be
similar to the situation in the base layer. Thus, in the
5 sense of the present invention, an enhancement sequence of
residual error pictures is already a sequence of pictures,
wherein in the extreme case only a single block of a single
"residual error picture" will have motion prediction
residual values, while in all other blocks of this picture
10 and even in all other "residual error pictures" actually no
residual errors exist, .' since the motion-compensated
prediction and, if necessary, the motion-compensated update
have been deactivated for all these pictures/blocks.
15 According to the present invention, this applies also for
the intermediate layer predictor, which calculates
enhancement prediction residual error pictures. Typically,
the enhancement prediction residual error pictures will be
present in a sequence. However, the intermediate layer
20 predictor is also preferably formed adaptively. If, for
example, it has been found out that a residual data
prediction of a base layer from the base layer to the
enhancement layer has been useful only for a single block
of a single "residual error picture", while for all other
25 blocks of this picture and, if necessary, even for all
other pictures of the sequence of enhancement prediction
residual error pictures, the intermediate layer residual
data prediction has been deactivated, in the present
context, for clarity reasons, the sequence will still be
30 referred to as enhancement prediction residual error
picture. In this connection, it should be noted that the
intermediate layer predictor can only predict, residual
data, when in a corresponding block of a residual error
picture in the base layer motion compensation residual
35 values have already been calculated, and when for a block
corresponding to this block (e.g. at the same x, y
position) a motion-compensated prediction has also been
performed in a residual error picture of the enhancement

- 36 -
sequence, so that in this block, residual error values
exist in the enhancement layer due to a motion-compensated
prediction. Only when actual motion-compensated prediction
residual values exist in both blocks to be considered, the
5 intermediate layer predictor will preferably become active
to use a block of residual error values in a picture of the
base layer as predictor for a block of residual error
values in a picture of the enhancement layer and then to
transmit only the residual values of this prediction, which
10 means enhancement prediction residual error data in this
block of the considered picture to the enhancement picture
coder.
In the following, a detailed illustration of the base
15 picture coder 1010 or the enhancement picture coder 1028
and any picture coder, respectively, will be discussed with
reference to Fig. lb. On the input side, the picture coder
receives the group of residual error pictures and supplies
them macroblock by macroblock to a transformation 1050. The
20 transformed macroblocks will then be scaled in a block 1052
and quantized by using a quantization parameter 1034, 1036,
... At the output of block 1052, the used quantization
parameter, which means the used quantizer step width for a
macroblock as well as quantization indices for the spectral
25 values of the macroblock, will be output. This information
will then be supplied to an entropy coder stage not shown
in Fig. lb, which comprises a Huffman coder or preferably
an arithmetic coder, which operates with the known CABAC
concept according to H.264/AVC. The output signal of means
30 1052 will also be supplied to block 1054, which performs an
inverse scaling and requantization to convert the
quantization indices together with the quantization
parameter again into numerical values, which will then be
supplied to an inverse transformation in block 1056 to
35 obtain a reconstructed group of residual error pictures,
which will now have a quantization error at the input of
the transformation block 1050 compared to the original
group of residual error pictures, which depends on the

- 37 -
quantization parameters and the quantizer step width,
respectively. Depending on the control of the switch 1044,
either the one signal or the other signal is supplied to
the interpolator 1022 or already to the intermediate layer
5 predictor 1018 in order to perform the inventive residual
value prediction.
A simple implementation of the intermediate layer predictor
flag 1030 is illustrated in Fig. lc. If the intermediate
10 layer prediction flag is set, the intermediate layer
predictor 1018 is activated. However, if the flag is not
set, the intermediate layer predictor is deactivated, so
that a simulcast operation is performed for this macroblock
or a sub-macroblock subordinate to this macroblock. The
15 reason therefore could be that the coder gain by the
prediction is actually a coder loss, which means that a
transmission of the corresponding macroblock at the output
of block 1016 provides a better coder gain in the
subsequent entropy coding than when prediction residual
20 values would be used.
A simple implementation of the motion data flag 1048 is
shown in Fig. 1d. If the flag is set, motion data of the
enhancement layer are derived from upsampled motion data of
25 the base layer. In the case of an SNR scalability, the
upsampler 1042 is not required. Here, when the flag 1048 is
set, the motion data of the enhancement layer can be
derived directly from the base motion data. It should be
noted that this motion data "derivation" can be the direct
30 takeover of the motion data or a real prediction wherein
block 1014 subtracts the motion vectors obtained from the
base layer from corresponding motion vectors for the
enhancement scaling layer calculated by block 1014, to
obtain motion data prediction values. The motion data of
35 the enhancement layer (if no prediction of any type has
been performed) or the residual values of the prediction
(if a real prediction has been performed) will be supplied
to the enhancement picture coder 1028 via an output shown

- 38 -
in Fig. la, so that they will be contained, in the
enhancement scaling layer bit stream 1004 in the end. If,
however, a full take over of the motion data from the base
scaling layer with or without scaling is performed, no
5 enhancement motion data have to be written into the
enhancement scaling layer bit stream 1004. It is merely
sufficient to signalize this fact by the motion data flag
1048 in the enhancement scaling layer bit stream.
10 Fig. 2 shows an apparatus for decoding a coded video
sequence, which comprises the base scaling layer 1002 and
the enhancement scaling layer 1004. The enhancement scaling
layer 1004 and the base scaling layer 1002 can originate
from a bit stream demultiplexer, which demultiplexes a
15 scalable bit stream with both scaling layers
correspondingly, to extract both the base scaling layer
1002 and the enhancement scaling layer 1004 from the common
bit stream. The base scaling layer 1002 is supplied to a
base picture decoder 1060, which is formed to decode the
20 base scaling layer to obtain a decoded base sequence of
residual error pictures and base motion data, which are
applied to an output line 1062. The output signals at line
1062 will then be supplied to a base motion combiner 1064,
which cancels the base motion predictor introduced in the
25 coder in block 1012, to output decoded pictures of the
first scaling layer on the output side. Further, the
inventive decoder comprises an enhancement picture decoder
1066 for decoding the enhancement scaling layer 1004 to
obtain enhancement prediction residual error pictures at an
30 output line 1068. Further, the output line 1068 comprises
motion data information, such as the motion data flag 1070
or, if actually enhancement motion data or enhancement
motion data residual values existed in the enhancement
scaling layer 1004, these enhancement motion data. Now, the
35 decoded base sequence on the line 1062 will either be
interpolated by an interpolator 1070 or supplied unchanged
(line 1072) to an intermediate layer combiner 1074 in order
to cancel the intermediate layer prediction performed by

- 39 -
the intermediate layer predictor 1018 of Fig. la. Thus, the
intermediate layer combiner is formed to combine the
enhancement prediction residual error pictures with
information about the decoded base sequence on line 1062,
5 either interpolated (1070) or not (1072), to obtain an
enhancement sequence of residual error pictures, which will
finally be provided to an enhancement motion combiner 1076,
which, like the base motion combiner 1064, cancels the
motion compensation performed in the enhancement layer. The
10 enhancement motion combiner 1076 is coupled to a motion
data determination means 1078, to provide the motion data
for the motion combination in block 1076. The motion data
can actually be full enhancement motion data for the
enhancement layer provided by the enhancement picture
15 decoder at output 1068. Alternatively, the enhancement
motion data can also be motion data residual values. In
both cases, the corresponding data will be supplied to the
motion data determination means 1078 via an enhancement
motion data line 1080. If, however, the motion data flag
20 1070 signals that no individual enhancement motion data
have been transmitted for the enhancement layer, necessary
motion data will be taken from the base layer via a line
1082, depending on the used scalability either directly
(line 1084) or after upsampling by an upsampler 1086.
25
Further, in the case of an intermediate layer prediction of
intrablocks, which means no motion data residual values, a
corresponding connection between the enhancement motion
combiner 107 6 and the base motion combiner 1064 is provided
30 on the decoder side, which has, depending on spatial
scalability, an interpolator 1090 or a bypass line when
only an SNR scalability has been used. In the case of an
optional intrablock prediction between two layers, merely a
prediction residual signal will be transmitted to the
35 enhancement layer for this intramacroblock, which will be
indicated by corresponding signalization information in bit
stream. In this case, the enhancement motion combiner will
also perform a summation for this one macroblock,

- 40 -
additionally to the below explained functionality, which
means to perform a combination between the macroblock
residual values and the macroblock values from the lower
scaling layer and to supply the obtained macroblock to the
5 actual inverse motion compensation processing.
In the following, with reference to Figs. 3 to 5d, a
preferred embodiment of the base motion predictor 1012 or
the enhancement motion predictor 1016 and the inverse
10 element, respectively, which means the enhancement motion
combiner 1076 or the base motion compensator 1064 will be
explained.
Basically, any motion-compensated prediction algorithm can
15 be used, which means also the motion compensation algorithm
illustrated at 92 in Fig. 9. Thus, the conventional motion
compensation algorithm also follows the systematic shown in
Fig. 1, wherein, however, the update operator U illustrated
in Fig. 4 with reference number 45, is deactivated. This
20 leads to the fact that a group of pictures is converted
into an original picture and residual pictures and
prediction residual signals, respectively, or residual
error pictures depending thereon. If, however, an
enhancement is implemented in the known motion compensation
25 scheme in that the update operator, as illustrated in Fig.
4, is active and is calculated, for example as it is
illustrated with regard to Figs. 5a to 5d, the normal
motion-compensated prediction calculation becomes the so-
called MCTF processing, which is also referred to as
30 motion-compensated time filtering. Here, the normal picture
and intra picture of the conventional motion compensation,
respectively, becomes a low-pass picture through the update
operation, since the original picture combined with the
prediction residual signal weighted by the update operator.
35
As has already been described with regard to Figs, la and
2, in a preferred embodiment of the present invention, such
an MCTF processing is performed for every scaling layer,

- 41 -
wherein the MCTF processing is preferably performed as it
is described with reference to Figs. 3 to 5d and 7 to 8.
In the following, the preferred embodiment of the motion-
5 compensated prediction filter will be described with
reference to Fig. 4 and the subsequent Figs. 5a - 5d. As
has already been explained, the motion-compensated temporal
filter (MCTF) consists of a general lifting scheme with
three steps, namely the polyphase decomposition, the
10 prediction and the update. The corresponding
analysis/synthesis filter bank structure is shown in Fig.
4. On the analysis side, the odd samples of a given signal
are filtered by a linear combination of the even samples by
using the prediction operator P and the high-pass signal H
15 to the prediction residual values. A corresponding low-pass
signal 1 is formed by adding a linear combination of the
prediction residual values h with the even samples of the
input signal s by using the update operator. The equation
connection of the variables h and 1 shown in Fig. 4 as well
20 as the basic embodiments of the operators P and U is shown
in Fig. 5a.
Since both the prediction step and the update step can be
fully inverted, the corresponding transformation can be
25 considered as critically sampled perfect reconstruction
filter bank. The synthesis filter bank comprises the
application of the prediction operator and the update
operator in inverse sequence with the inverted signs in the
summation process, wherein the even and odd polyphase
30 components are used. For a normalization of the high-
pass/low-pass components, corresponding scaling factors F1
and Fh are used. These scaling factors do not necessarily
have to be used, but they can be used when quantizer step
sizes are chosen during coding.
35
f[x,k] shows a video signal with the space coordinates x =
(x,y)T, wherein k is the time coordinate. The prediction
operator P and the update operator U for the temporal

- 42 -
decomposition by using the lifting representation of the
hair wavelet is given as shown on the left hand side in
Fig. 5b. For the 5/3 transformation, corresponding
operators result as shown on the right hand side in Fig.
5 5b. The enhancement to the motion-compensated temporal
filtering is obtained by modification of the prediction
operator and the update operator, as shown in Fig. 5c.
Particularly, reference will be made to the reference
indices r > 0, which allow a general picture adaptive
10 motion-compensated filtering. Through these reference
indices, it can be ensured that in the scenario illustrated
in Fig. 4 not only merely two temporally immediately
subsequent pictures are decomposited into a high-pass
picture and a low-pass picture, but that, for example, a
15 first picture can be filtered in a motion compensated way
with a third picture of a sequence. Alternatively, the
appropriate choice of reference indices allows that, e.g.,
one and the same picture of a sequence of sequences can be
used to serve as base for the motion vector. This means
20 that the reference indices allow for example in a sequence
of eight pictures that all motion vectors are related, e.g.
to the fourth picture of the sequence, so that a single
low-pass picture results at the end by processing these
eight pictures through the filter scheme in Fig. 4, and
25 that seven high-pass pictures (enhancement pictures) result
and that all motion vectors relate to one and the same
picture of the original sequence where one enhancement
picture is associated to every motion vector.
30 If thus one and the same picture of a sequence is used as
reference for filtering several further pictures, this
leads to a temporal resolution scaling not obeying to the
factor of 2, which can be advantageous for certain
applications. Always the same picture, namely, for example,
35 the fourth picture of the sequence of eight pictures, is
fed into the lower branch of the analysis filter bank in
Fig. 4. The low-pass picture is the same in every
filtering, namely the finally desired single low-pass

- 43 -
picture of the sequence of pictures. When the update
parameter is zero, the base picture is simply "passed
through" through the lower branch. In comparison, the high-
pass picture is always dependent on the corresponding other
5 picture of the original sequence and the prediction
operator, wherein the motion vector associated to this
input picture is used in the prediction. Thus, in this case
it can be said that the finally obtained low-pass picture
is associated to a certain picture of the original sequence
10 of pictures, and that also every high-pass picture is
associated to a picture of the original sequence, wherein
exactly the deviation of the original picture correspond to
the sequence (a motion compensation) from the chosen base
picture of the sequence (which is fed into the lower branch
15 of the analysis filter bank of Fig. 4) . When every update
parameter M01, M11, M21 and M31 is equal to zero, this leads
to the fact that the picture fed into the lower branch 73
of the fourth level is simply "passed through" towards the
bottom. In a way, the low-pass picture TP1 is fed
20 "repeatedly" into the filter bank, while the other pictures
- controlled by the reference indices - are introduced one
after the other into the input 64 of Fig. 3.
As can be seen from the previous equations, the prediction
25 and update operators for the motion-compensated filtering,
respectively, provide different predictions for the two
different wavelets. When the hair wavelet is used, a
unidirectional motion-compensated prediction is achieved.
If, however, the 5/3 spline wavelet is used, the two
30 operators specify a bidirectional motion-compensated
prediction.
Since the bidirectional compensated prediction generally
reduces the energy of the prediction residual value, but
35 increases the motion vector rate compared to an
unidirectional prediction, it is desirable to switch
dynamically between the unidirectional and the
bidirectional prediction, which means that one can switch

- 44 -
between a lifting representation of the hair wavelet and
the 5/3 spline wavelet dependent on a picture dependent
control signal. The inventive concept, which uses no closed
feedback loop for temporal filtering, easily allows this
5 macroblock by macroblock switching between two wavelets,
which again supports flexibility and particularly data rate
saving, which can be performed optimally in a signal-
adapted way.
10 In order to represent the motion fields or generally the
prediction data fields MP and MU, ideally, the existing
syntax of the B slices in H.264/AVC can be used.
By cascading the pair-wise picture decomposition stages, a
15 dyadic tree structure is obtained, which decomposits a
group of 2n pictures into 2n-1 residual pictures and a single
low-pass (or intra) picture, as it is illustrated with
regard to Fig. 7 for a group of eight pictures.
Particularly, Fig. 7 shows the first-level high-pass
20 picture HP1 at the output 22 of the filter of the first
level as well as the first-level low-pass picture at the
output 24 of the first-level filter. The two low-pass
pictures TP2 at the output 16 of the second-level filter as
well as the high-pass pictures obtained from the second
25 level are shown in Fig. 7 as second level pictures. The
third level low-pass pictures are applied to the output 7 6
of the third level filter, while the third level high-pass
pictures are applied to the output 75 in processed form.
The group of eight pictures could originally comprise eight
30 video pictures, wherein then the decoder of Fig. 3 would be
used without fourth filter level. If, however, the group of
eight pictures is a group of eight low-pass pictures, as
they are used at the output 73 of the fourth level filter,
the inventive MCTF decomposition can be used as base motion
35 predictor, enhancement motion predictor and as base motion
combiner or enhancement motion combiner, respectively.

- 45 -
Thus, generally, in this decomposition a group of 2n
pictures, (2n+1-2) motion field descriptions, (2n~1) residual
pictures as well as a single low-pass (or intra) picture
are transmitted.
5
Both the base motion compensator and the enhancement motion
compensator are preferably controlled by a base control
parameter and an enhancement control parameter,
respectively, to calculate an optimum combination of a
10 quantization parameter (1034 or 1036) and motion
information, which is fixed in dependence on a certain
rate. This is performed according to the following method
to obtain an optimum ratio with regard to a certain maximum
bit rate. Thus, it has been found out that for lower bit
15 rates, which means relatively coarse quantization
parameters, the motion vectors count more than for higher
scaling layers, where relatively fine quantization
parameters are taken. Thus, for cases of coarse quantizing
and thus lower bit rate, less motion data are calculated
20 than for higher scaling layers. Thus, it is preferred in
higher scaling layers to move to sub-macroblock modes to
calculate rather a lot of motion data for a good quality
and for an optimum situation in the high bit rate, than in
the case of a lower bit rate, where the motion data
25 proportionally count more with regard to the residual data
than in the case of a higher scaling layer. This will be
discussed below.
Pictures A and B are given, which are either original
30 pictures or pictures representing low-pass signals, which
are generated in a previous analysis stage. Further, the
corresponding arrays of luma samples a[] and b[] are
provided. The motion description M10 is estimated in a
macroblock by macroblock way as follows:
35
For all possible macroblock and sub-macroblock partitions
of a macroblock i within a picture B, the associated motion
vectors

- 46 -
mi = [mx, my]T
wherein the deterioration term is given as follows:
are determined by minimizing the Lagrange function


Here, S specifies the motion vector search region within
the reference picture A. P is the region covered by the
considered macroblock partition or sub-macroblock
15 partition. R(i,m) specifies the number of bits, which are
required to transmit all components of the motion vector m,
wherein λ is a fixed Lagrange multiplier.
First, the motion search proceeds across all integer sample
20 exact motion vectors in the given search region S. Then, by
using the best integer motion vector, the eight surrounding
half sample exact motion vectors are tested. Finally, by
using the best half sample exact motion vector, the eight
surrounding quarter sample exact motion vectors are tested.
25 For the half and quarter half exact motion vector
improvement, the term

30 is interpreted as interpolation operator.
Generally, the mode decision for the macroblock mode and
the sub-macroblock mode follows the same approach. The mode

- 47 -
P1, which minimizes the following Lagrange function, is
chosen from a given set of possible macroblock or sub-
macroblock modes Smode:

wherein P specifies the macroblock or sub-macroblock
region, and wherein m[p,x,y] is the motion vector which is
associated to the macroblock or sub-macroblock mode p and
the partition or sub-macroblock partition, which comprises
15 the luma position (x,y).
The rate term R(i,p) represents the number of bits, which
are associated to the choice of the coder mode p. For the
motion compensated coder modes, the same comprises the bits
20 for the macroblock mode (if applicable), the sub-macroblock
mode and modes (if applicable), respectively, and the
motion vector and vectors, respectively. For the intra
mode, the same comprises the bits for the macroblock mode
and the arrays of quantized luma and chroma transformation
25 coefficient levels.
The set of possible sub-macroblock modes is given by
{P_8x8, P_8x4, P_4x8, P_4x4}.
30
The set of possible macroblock modes is given by
{P_16xl6, P_16x8, P_8xl6, P_8x8, INTRA},
35 wherein the INTRA mode is only used when a motion field
description M10 used for the prediction step is estimated.

- 48 -
The Lagrange multiplier A is set according to the following
equation in dependence on the base layer quantization
parameter for the high-pass picture or pictures QPHi of the
decomposition stage, for which the motion field is
5 estimated:

According to the invention, the decomposition scheme shown
10 in Fig. 8 is used, which is assumed to enable a sensible
compromise between temporal scalability and coder
efficiency. The sequence of the original pictures is
treated as sequence of input pictures A, B, A, B, A, B, ...,
A, B. Thus, this scheme provides a stage with optimum
15 temporal scalability (equal distance between the low-pass
pictures). The sequence of low-pass pictures, which are
used as input signal to all following decomposition stages,
are treated as sequences of input pictures B, A, A, B, B, A
... A, B, whereby the spaces between the low-pass pictures
20 which are decomposited, are kept small in the following two
channel analysis scheme, as can be seen in Fig. 8.
In the following, reference will be made to preferred
implementations of both the motion data intermediate layer
25 prediction and the residual data intermediate layer
prediction with regard to Figs. 6a to 6d. To obtain a
spatial and an SNR scalability, respectively, basically,
motion data and texture data of a lower scaling layer are
used for prediction purposes for a higher scaling layer.
30 Here, particularly in the spatial scalability, an
upsampling of the motion data will be required, before they
can be used as prediction for the decoding of spatial
enhancement layers. The motion prediction data of a base
layer representation are transmitted by using a subset of
35 the existing B slice syntax of AVC. Preferably, two
additional macroblock modes are introduced for coding the
motion field of an enhancement layer.

- 49 -
The first macroblock mode is "base_layer_mode" and the
second mode is the "qpel_refinement_mode". For signalizing
these two additional macroblock modes, two flags, namely
5 the BLFlag and the QrefFlag are added to the macroblock
layer syntax, prior to the syntax element mb_mode, as shown
..in Fig. 1. Thus, the first flag BLFlag 1098 signalizes the
base layer mode, while the other flag 1100 symbolizes the
qpel refinement mode. If such a flag is set, it has the
10 value 1, and the data stream is as shown in Fig. 6a. Thus,
if the flag 1098 has the value 1, the flag 1100 and the
syntax element macroblock mode 1102 have no further
importance. If, however, the flag 1098 has the value zero,
it is not set, and the flag 1100 will be used, which, when
15 it is set, again bridges the element 1102. If, however,
both flags 1098 and 1100 have a value zero, which means
they are both not set, the macroblock mode will be
evaluated in the syntax element 1102.
20 When BLFlag = 1, the base layer mode is used, and no
further information is used for the corresponding
macroblock. This macroblock mode indicates that the motion
prediction information including the macroblock partition
of the corresponding macroblock of the base layer is
25 directly used in that way for the enhancement layer. It
should be noted that here and in the whole specification,
the term "base layer" is to represent a next lower layer
with regard to the currently considered layer, which means
the enhancement layer. When the base layer represents a
30 layer with half the spatial resolution, the motion vector
field, which means the field of motion vectors including
the macroblock partition is scaled correspondingly, as it
is illustrated in Fig. 6b. In this case, the current
macroblock comprises the same region as an 8x8 sub-
35 macroblock of the base layer motion field. Thus, if the
corresponding base layer macroblock is coded in a direct,
16x16, 16x8 or 8x16 mode, or when the corresponding base
layer sub-macroblock is coded in the 8x8 mode or in the

- 50 -
direct 8x8 mode, the 16x16 mode is used for the current
macroblock. If, on the other hand, the base layer sub-
macroblock is coded in the 8x4, 4x8 or 4x4 mode, the
macroblock mode for the current macroblock = 16x8, 8x16 or
5 8x8 (with all sub-macroblock modes = 8x8) . When the base
layer macroblock represents an INTRA macroblock, the
current macroblock is set to INTRA_BASE, which means that
it is a macroblock with a prediction from the base layer.
For the macroblock partitions of the current macroblock,
10 the same reference indices are used as for the
corresponding macroblock/sub-macroblock partitions of the
base layer block. The associated motion vectors are
multiplied by a factor of 2. This factor applies for the
situation shown in Fig. 6b, where a base layer 1102
15 comprises half the region and number of pixels,
respectively, than the enhancement layer 1104. If the ratio
of the spatial resolution of the base layer to the spatial
resolution of the enhancement layer is unequal to 1/2,
corresponding scaling factors are used for the motion
20 vector.
If, however, the flag 1098 equals zero and flag 1100 equals
1, macroblock mode qpel_refinement_mode is signalized. The
flag 1100 is preferably only present when the base layer
25 represents a layer with half the spatial resolution of the
current layer. Otherwise, the macroblock mode
(qpel_ref inement_mode) is not contained in the set of
possible macroblock modes. This macroblock mode is similar
to the base layer mode. The macroblock partition as well as
30 the reference indices and the motion vectors are derived as
in the base layer mode. However, for every motion vector,
there is an additional quarter sample motion vector
refinement -1.0 or +1 for every motion vector component,
which is transmitted additionally and added to the derived
35 motion vector.
When the flag 1098 = zero and the flag 1100 = zero, or when
the flag 1100 is not present, the macroblock mode as well

- 51 -
as the corresponding reference indices and motion vector
differences are specified as usual. This means that the
complete set of motion data is transmitted for the
enhancement layer the same way as for the base layer.
5 However, according to the invention, the possibility is
provided to use the base layer motion vector as predictor
for the current enhancement layer motion vector (instead of
the spatial motion vector predictor). Thus, the list X
(wherein X lies between 0 and 1) is to specify the
10 reference index list of the considered motion vector. If
all subsequent conditions are true, a flag MvPrdFlag is
transmitted, as shown in Fig. 6c, for every motion vector
difference:
15 - the base layer macroblock comprising the current
macroblock/sub-macroblock partitions is not coded in an
INTRA macroblock mode;
the base layer macroblock/sub-macroblock partition
20 covering the upper left sample of the current
macroblock/sub-macroblock partition uses the list X or
a biprediction;
the list X reference index of the base layer
25 macroblock/sub-macroblock partition, which comprises
the upper left sample of the current macroblock/
sub-macroblock partition is equal to the list X
reference index of the current macroblock/sub-
macroblock partition.
30
If the flag 1106 of Fig. 6c is not present, or if this flag
1106 = zero, the spatial motion vector predictor is
specified as it is the case in the standard AVC. Otherwise,
when the flag 1106 is present and = 1, the corresponding
35 base layer vector is used as motion vector predictor. In
this case, the list X motion vector (wherein X = 0 or 1) of
the current macroblock/sub-macroblock partition is obtained
by adding the transmitted list X motion vector difference

- 52 -
to the possibly scaled list X motion vector of the base
layer macroblock/sub-macroblock partition.
Thus, the flags 1098, 1100 and 1106 represent together a
5 possibility to implement the motion data flag 1048
generally indicated in Fig. la and generally a motion data
control signal 1048, respectively. There are, of course,
different other possibilities of signalizing, wherein
naturally a fixed agreement between transmitter and
10 receiver can be used, which allows a reduction of
signalizing information.
In summary, a detailed implementation of the enhancement
motion compensator 1014 of Fig. la and the enhancement
15 motion data determination means 1078 of Fig. 2,
respectively, is illustrated in more detail with regard to
Figs, le, 1f and lg.
With reference to Fig. le, it can be seen that the
20 enhancement motion compensator 1014 basically has to do two
things. Thus, it first has to calculate the enhancement
motion data, typically the whole motion vectors and supply
them to the enhancement motion predictor 1016, so that the
same can use these vectors in uncoded form to obtain the
25 enhancement sequence of residual error pictures which are,
in the prior art, typically performed adaptively and block
by block. Another matter, however, is the enhancement
motion data processing, which means how the motion data
used for a motion-compensated prediction will now be
30 compressed as much as possible and written into a bit
stream. In order for something to be written into the bit
stream, respective data have to be brought to the
enhancement picture coder 1028, as it is illustrated with
regard to Fig. le. Thus, the enhancement motion data
35 processing means 1014b has the function to reduce the
redundancy contained in the enhancement motion data, which
the enhancement motion data calculation means 1014a has

- 53 -
determined, with regard to the base layer as much as
possible.
According to the invention, the base motion data or the
5 upsampled base motion data can be used both by the
enhancement motion data calculation means 1014a for
calculating the actually to be used enhancement motion data
or can also be used only for enhancement motion data
processing, which means for enhancement motion data
10 compression, while they are of no importance for the
calculation of the enhancement motion data. While the two
possibilities 1.) and 2.) of Fig. lg show embodiments where
the base motion data and the upsampled base motion data are
already used in the enhancement motion data calculation,
15 the embodiment 3.) of Fig. lb shows a case where
information about the base motion data are not used for
calculating the enhancement motion data but merely for
coding and capture of residual data, respectively.
20 Fig. If shows the decoder side implementation of the
enhancement motion data determination means 107 8, which has
a control module 1078a for block by block control, which
contains the signalizing information from the bit stream
and from the enhancement picture decoder 1066,
25 respectively. Further, the enhancement motion data
determination means 1078 comprises an enhancement motion
data reconstruction means 1078b, which actually determines
the motion vectors of the enhancement motion data field,
either only by using the decoded base motion data or
30 decoded upsampling base motion data or by combining
information about the decoded base motion data and from the
residual data extracted from the enhancement motion decoder
1066 from the enhancement scaling layer 1004, which can
then be used by the enhancement motion combiner 1076, which
35 can be formed as common combiner to reverse the coder side
motion-compensated prediction.

- 54 -
In the following, reference will be made to the different
embodiments as they are illustrated in Fig. lg in overview.
As has already been illustrated with regard to Fig. 6a, the
BLFlag 1098 signalizes a complete takeover of the upscaled
5 base motion data for the enhancement motion prediction. In
that case, means 1014a is formed to completely take over
the base motion data and in the case of different
resolutions of the different layers, to take over the
motion data in upscaled form and transmit them to means
10 1016, respectively. However, no information about motion
fields or motion vectors is transmitted to the enhancement
picture coder. Instead, merely an individual flag 1098 is
transmitted for every block, either macroblock or a sub-
macroblock.
15
On the decoder side, this means that means 1078a of Fig. If
decodes the flag 1098 for one block and, if it was active,
uses the decoded base motion data present from the base
layer or the decoded upsampled base motion data to
20 calculate the enhancement motion data, which are then
supplied to block 1076. In this case, the means 1078
requires no motion vector residual data.
In the second embodiment of the present invention, which is
25 signalized by the flag QrefFlag 1100, the base motion
vector is integrated into the enhancement motion data
calculation, which is performed by means 1014a. As it is
illustrated in Fig. lg in portion 2.) and described above,
the motion data calculation and the calculation of the
30 motion vector m, respectively, is performed by searching
the minimum of the term
(D + λ R).
35 The difference between a block of a current picture B and a
block of a previous and/or later picture shifted by a
certain potential motion vector is introduced into the

- 55 -
distortion term D. The quantization parameter of the
enhancement picture coder indicated in Fig. la by 1036 is
introduced into the factor X. The term R provides
information about the number of bits used for coding a
5 potential motion vector.
Normally, a search is performed among different potential
motion vectors, wherein the distortion term D is calculated
for every new motion vector, and the rate term R is
10 calculated, and wherein the enhancement quantization
parameter 1036, which is preferably fixed, but could also
vary, is considered. The described sum term is evaluated
for different potential motion vectors, whereupon the
motion vector is used, which provides the minimum result of
15 the sum.
Now, according to the invention, the base motion vector of
the corresponding block from the base layer is also
integrated into this iterative search. If it fulfills the
20 search criteria, again merely the flag 1100 has to be
transmitted, but no residual values or anything else for
this block has to be transmitted. Thus, when the base
motion vector fulfills the criterion (minimum of the
previous term) for a block, means 1014a uses the base
25 motion vector in order to transmit it to means 1016.
However, merely the flag 1100 is transmitted to the
enhancement picture coder.
On the decoder side, this means that the means 1078a
30 controls the means 1078b when it decodes the flag 1100 to
determine the motion vector for this block from the base
motion data, since the enhancement picture decoder has
transmitted no residual data.
35 In a variation of the second embodiment, not only the base
motion vector but also a plurality of base motion vectors
derived from the base motion vector and (slightly) altered
are integrated into the search. Depending on the

- 56 -
implementation, any component of the motion vector can be
independently increased or decreased by one increment, or
be left the same. This increment can represent a certain
granularity of a motion vector, e.g. a resolution step, a
5 half resolution step or a quarter resolution step. If such
an altered base motion vector fulfills the search criteria,
the alteration, which means the increment, which means +1,
0 or -1 is transmitted as "residual data", additionally to
the flag 1100.
10
Activated by flag 1100, a decoder will then search for the
increment in the data stream and further recover the base
motion vector or the upsampled base motion vector and
combine the increment with the corresponding base motion
15 vector in block 1078b, to obtain the motion vector for the
corresponding block in the enhancement layer.
In the third embodiment, which is signalized by the flag
1106, the determination of the motion vectors can basically
20 be performed arbitrarily. With regard to the full
flexibility, the means 1014a can determine the enhancement
motion data e.g. according to the minimization object
mentioned in connection with the second embodiment. Then,
the determined motion vector is used for coder side motion-
25 compensated prediction, without considering information
from the base layer. However, in that case, the enhancement
motion data processing 1014a is formed to incorporate the
base motion vectors into the motion vector processing for
redundancy reduction, which means prior to the actual
30 arithmetic coding.
Thus, according to the standard H.264/AVC, a transmission
of motion vector differences is performed, wherein
differences between adjacent blocks are determined within a
35 picture. In the implementation, the difference can be
formed between different adjacent blocks, to select then
the smallest difference. Now, according to the invention,
the base motion vector for the corresponding block in a

- 57 -
picture is incorporated into this search for the most
favorable predictor for the motion vector difference. If it
fulfills the criterion that it provides the smallest
residual error value as predictor, this is signalized by
5 the flag 1106 and merely the residual error value is
transmitted to block 1028. If the base motion vector does
not fulfill this criterion, the flag 1106 is not set, and a
spatial motion vector difference calculation is performed.
10 For simpler coder implementations, however, instead of the
iterative search, simply always and for adaptively
determined blocks the base motion vector, respectively, and
an upsampled version of the same, respectively, can serve
as predictor.
15
According to the invention, an intermediate layer
prediction of residual data will also be performed. This
will be discussed below. When the motion information is
changed from one layer to the next, it can be favorable or
20 unfavorable to predict residual information and, in the
case of a MCTF decomposition, high-pass information of the
enhancement layer, respectively, from the base layer. When
the motion vectors for a block of the current layer are
similar to the motion vectors of the corresponding base
25 layer and macroblock by macroblock to corresponding motion
vectors of the corresponding base layer, it is likely that
the coder efficiency can be increased when the coded base
layer residual signal (high-pass signal) is used as
prediction for the enhancement residual signal (enhancement
30 high-pass signal), whereby only the difference between the
enhancement residual signal and the base layer
reconstruction (line 1024 of Fig. la) is coded. However,
when the motion vectors are not similar, it is very
unlikely that a prediction of the residual signal will
35 improve the coder efficiency. Consequently, an adaptive
approach is used for the prediction of the residual signal
and high-pass signal, respectively. This adaptive approach,
which means whether the intermediate layer predictor is

- 58 -
active or not, can be performed by an actual calculation of
the benefit based on the difference signal or can be
performed based on an estimation, how different the motion
vector of a base scaling layer for a macroblock is to a
5 corresponding macroblock in the enhancement scaling layer.
If the difference is smaller than a certain threshold, the
intermediate layer predictor is activated via the control
line 130. However, if the difference is higher than a
certain threshold, the intermediate layer predictor for
10 this macroblock is deactivated.
A flag ResPrdFlag 1108 is transmitted. When the flag 1108 =
1, the reconstructed residual signal of the base layer is
used as prediction for the residual signal of the current
15 macroblock of the enhancement layer, wherein only an
approximation of the difference between the current
residual signal of the enhancement layer and its base layer
reconstruction will be coded. Otherwise, the flag 1108 does
not exist or equals zero. Here, the residual signal of the
20 current macroblock in the enhancement layer will then be
coded without prediction from the base layer.
When the base layer represents a layer with half the
spatial resolution of the enhancement layer, the residual
25 signal is upsampled by using an interpolation filter,
before the upsampled residual signal of the base layer is
used as prediction signal. This filter is an interpolation
filter with six taps, such that for interpolating a value
of the higher spatial resolution of the enhancement layer,
30 which was not present in the base layer due to the lower
resolution, values from the surroundings are used to obtain
an interpolation result, which is as good as possible.
If, however, values at the edge of a transformation block
35 are interpolated, and the interpolation filter would use
only values of another transformation block for
interpolation, it is preferred not to do this, but to
synthesize the values of the interpolation filter outside

- 59 -
the considered block so that an interpolation with as
little artifacts as possible takes place.
Based on a so-called core experiment, it was found out that
5 the intermediate layer prediction of motion and residual
values significantly improves the coder efficiency of the
AVC based MCTF approach. For certain test points, PSNR
gains of more than 1 dB were obtained. Particularly with
very low bit rates for every spatial resolution (with the
10 exception of the base layer), the improvement of the
reconstruction quality was clearly visible.
Depending on the circumstances, the inventive method can be
implemented in hardware or in software. The implementation
15 can be performed on a digital storage medium, particularly
a disc or CD with electronically readable control signals,
which can cooperate with a programmable computer system
such that the method is performed. Thus, generally, the
invention consist also in a computer program product with a
20 program code for performing the inventive method stored on
a machine readable carrier, when the computer program
product runs on a computer. In other words, the invention
can also be realized as computer program with a program
code for performing the method when the computer program
25 runs on a computer.
Further, the present invention concerns a computer readable
medium, whereon a scalable data stream with a first scaling
layer and a second scaling layer together with the
30 associated control characters are stored for the different
decoder-side means. Thus, the computer readable medium can
be a data carrier or the internet whereon a data stream is
transmitted from a provider to a receiver.
85

- 60 -
Claims
1. Apparatus for generating a coded video sequence having
a base scaling layer (1002) and an enhancement scaling
5 layer (1004), comprising:
a base motion compensator (1006) for calculating base
motion data, which indicate how a macroblock in a
current picture has moved in relation to another
10 picture in a group of pictures;
a base motion predictor (1012) for calculating a base
sequence of residual error pictures by using the base
motion data;
15
a base picture coder (1010) formed to calculate the
coded base scaling layer (1002) from the base sequence
of residual error pictures;
20 an enhancement motion compensator (1014) for
determining enhancement motion data;
an enhancement motion predictor (1016) for calculating
an enhancement sequence of residual error pictures;
25
an intermediate layer predictor (1018) for calculating
enhancement prediction residual error pictures by
using the enhancement sequence of residual error
pictures and by using information (1026) about the
30 base sequence of residual error pictures; and
an enhancement picture coder (1028) for coding the
enhancement prediction residual error pictures to
obtain the coded enhancement scaling layer (1004) .
35
2. Apparatus according to claim 1,

- 61 -
wherein the base picture coder (1010) is formed to
perform a quantization with a base quantization
parameter (1034);
5 wherein the enhancement picture coder (1028) is formed
to perform a quantization with an enhancement
quantization parameter (1036), wherein the enhancement
quantization parameter (1036) can result in a finer
quantization than the base quantization parameter
10 (1034);
wherein the base picture coder (1010) is formed to
reconstruct the base sequence of residual error
pictures quantized with the first quantization
15 parameter to obtain a reconstructed base sequence and
wherein the intermediate layer predictor (1026) is
formed to calculate the enhancement prediction
residual error pictures by using the enhancement
20 sequence of residual error pictures and the
reconstructed base sequence of residual error pictures
as information about the base sequence of residual
error pictures.
25 3. Apparatus according to claim 1 or 2, further
comprising:
a decimator (1032) for decimating a resolution of the
group of pictures, wherein the decimator (1032) is
30 formed to provide a group of pictures with a base
resolution to the base compensator (1006), which is
smaller than an enhancement resolution of a group of
pictures which is provided to the enhancement motion
compensator (1014); and
35
an interpolator (1022) for spatially interpolating the
base sequence of residual error pictures or a
reconstructed base sequence of residual error pictures

- 62 -
to obtain an interpolated base sequence of residual
error pictures, which can be supplied to the
intermediate layer predictor (1018) as information
(1026) about the base sequence of residual error
5 pictures.
4. Apparatus according to claim 3, further comprising a
motion data upsampler (1042) to adapt the base motion
data to the enhancement resolution.
10
5. Apparatus according to one of the preceding claims,
wherein the base motion compensator (1006) is formed
to calculate a two-dimensional motion vector for a
macroblock.
15
6. Apparatus according to one of the preceding claims,
wherein the base motion predictor (1012) is formed to
subtract a macroblock predicted by using the base
motion data from a current macroblock to obtain a
20 macroblock of a residual error picture of the base
sequence of residual error pictures.
7. Apparatus according to one of the preceding claims,
wherein the intermediate layer predictor (1018) is
25 formed to decide adaptively for blocks, whether a
coder gain can be increased by applying a prediction
in comparison to a usage of a block from the
enhancement sequence of residual error pictures.
30 8. Apparatus according to one of the preceding claims,
wherein the intermediate layer predictor (1018) is
formed to subtract a macroblock predicted by using a
macroblock from the base sequence of residual error
pictures or the reconstructed base sequence of
35 residual error pictures or the interpolated base
sequence of residual error pictures or the
reconstructed and interpolated base sequence of
residual error pictures from a current macroblock of

- 63 -
the enhancement sequence of residual error pictures to
obtain a macroblock of an enhancement prediction
residual error picture.
5 9. Apparatus according to claim 2, wherein the base
motion compensator (1006) is formed to calculate the
base motion data in dependence on the base
quantization parameter (1034), a distortion term and a
data rate for transmitting the base motion data.
10
10. Apparatus according to claim 2 or 9, wherein the
enhancement motion compensator (1006) is formed to
calculate the enhancement motion data in dependence on
the enhancement quantization parameter (1036), a
15 distortion term and a data rate for transmitting the
enhancement motion data.
11. Apparatus according to one of the preceding claims,
wherein the enhancement motion compensator (1006) is
20 formed to use the base motion data or scaled motion
data in dependence on a mode control signal.
12. Apparatus according to one of the preceding claims,
wherein the base scaling layer (1002) has a lower
25 resolution than the enhancement scaling layer (1004),
and wherein the enhancement motion compensator (1014)
is formed to determine residual motion data in
dependence on a control signal (1048), and wherein the
enhancement picture coder (1028) is formed to code the
30 residual motion data into the enhancement scaling
layer (1004).
13. Apparatus according to one of the preceding claims,
wherein the base motion predictor (1012) and the
35 enhancement motion predictor (1016) are formed to
perform a motion-compensated prediction operation.

- 64 -
14. Apparatus according to one of claims 1 to 12, wherein
the base motion predictor (1012) and the enhancement
motion predictor are formed to perform a motion-
compensated update operation apart from the motion-
5 compensated prediction operation, to obtain a motion-
compensated temporal decomposition into at least a
low-pass and several high-pass pictures.
15. Method for generating a coded video sequence having a
10 base scaling layer (1002) and an enhancement scaling
layer (1004), comprising the steps of:
calculating (1006) base motion data, which indicate
how a macroblock in a current picture has moved in
15 relation to another picture in a group of pictures;
calculating (1012) a base sequence of residual error
pictures by using the base motion data;
20 coding (1010) information about the base sequence of
residual error pictures to calculate the coded base
scaling layer (1002) from the base sequence of
residual error pictures;
25 determining (1014) enhancement motion data;
calculating (1016) an enhancement sequence of residual
error pictures;
30 calculating (1018) enhancement prediction residual
error pictures by using the enhancement sequence of
residual error pictures and by using information
(1026) about the base sequence of residual error
pictures; and
35
coding (1028) the enhancement prediction residual
error pictures to obtain the coded enhancement scaling
layer (1004) .

- 65 -
16. Apparatus for decoding a coded video sequence having a
base scaling layer (1002) and an enhancement scaling
layer (1004), comprising:
5
a base picture decoder (1060) for decoding the base
scaling layer to obtain a decoded base sequence of
residual error pictures and base motion data;
10 a base motion combiner (1064) to obtain a sequence of
pictures of the base scaling layer by using the base
motion data and the decoded sequence of residual error
pictures;
15 an enhancement picture decoder (1066) for decoding the
enhancement scaling layer to obtain enhancement
prediction residual error pictures;
an intermediate layer combiner (1074) for combining
20 the decoded base sequence of residual error pictures
or an interpolated base sequence of residual error
pictures with the enhancement prediction residual
error pictures to obtain an enhancement sequence of
residual error pictures;
25
an enhancement motion combiner (1076), which is formed
to obtain a sequence of pictures of the enhancement
scaling layer by using the enhancement sequence of
residual error pictures and enhancement motion data.
30
17. Apparatus according to claim 16, wherein the base
picture decoder (1060) is formed to dequantize by
using a base quantization parameter which is larger
than an enhancement quantization parameter, and
35
wherein the intermediate layer combiner (1074) is
formed to use the decoded base sequence of residual
error pictures or an interpolated version of the same.

- 66 -
18. Apparatus according to claim 16 or 17, wherein the
pictures of the base scaling layer (1002) can have a
lower resolution than the pictures of the enhancement
5 scaling layer (1004), and
further having a residual value interpolator (1070) to
interpolate the decoded base sequence of residual
error pictures to the higher resolution.
10
19. Apparatus according to one of claims 16 to 18, wherein
the pictures of the base scaling layer have a lower
resolution than the pictures of the enhancement
scaling layer, and
15
further having a motion data upsampler (1086) to
convert the base motion data to the enhancement
resolution to obtain the enhancement motion data or a
prediction of the enhancement motion data.
20
20. Apparatus according to claim 19, wherein the coded
video sequence has residual motion data, and
having a motion data determiner (1078) to combine the
25 residual motion data with the prediction of the
enhancement motion data to determine the enhancement
motion data.
21. Apparatus according to one of claims 16 to 20, wherein
30 the base motion combiner (1064) and the enhancement
motion combiner are formed to perform only an inverse
prediction operation or an inverse prediction
operation and an inverse update operation.
35 22. Apparatus according to one of claims 18 to 20, wherein
an interpolator (1070) is formed to use an
interpolation filter operating by using a plurality of

- 67 -
pixels in an environment of the pixel to be
interpolated.
23. Apparatus according to claim 22, wherein the
5 interpolation filter is formed to use a synthesized
value instead of the pixel of the other transformation
block when a pixel of another transformation block is
present in the proximity of the pixel to be
interpolated.
10
24. Apparatus according to one of claims 16 to 23, wherein
the coded video sequence has enhancement motion data
for macroblocks, wherein a motion data control signal
(1070) is associated to the enhancement motion data,
15 which indicates whether a prediction signal is
underlying the enhancement motion data, which is
derived from a scaled motion vector of the base motion
data, or whether a prediction signal is underlying the
enhancement motion data, which is not derived from the
20 base motion data, and wherein the apparatus is formed
to read the motion data control signal (1070) and to
calculate the enhancement motion data in dependence on
the control signal by using the scaled version of the
base motion data.
25
25. Apparatus according to one of claims 16 to 24, wherein
the coded video sequence for macroblocks has an
intermediate layer prediction control signal (1030) in
the enhancement scaling layer, which indicates whether
30 a macroblock has been generated due to an intermediate
layer prediction or without an intermediate layer
prediction, and
wherein the apparatus is further formed to activate
35 the intermediate layer combiner (1074) only when the
intermediate layer prediction control signal (1030)
indicates an intermediate layer prediction for a
considered macroblock.

- 68 -
26. Method for decoding a coded video sequence with a base
scaling layer (1002) and an enhancement scaling layer
(1004), comprising the steps of:
5
decoding (1060) the base scaling layer to obtain a
decoded base sequence of residual error pictures and
base motion data;
10 performing a base motion combination (1064) to obtain
a sequence of pictures of the base scaling layer by
using the base motion data and the decoded sequence of
residual error pictures;
15 decoding (1066) the enhancement scaling layer to
obtain enhancement prediction residual error pictures;
combining (1074) the decoded base sequence of residual
error pictures or an interpolated base sequence of
20 residual error pictures with the enhancement
prediction residual error pictures to obtain an
enhancement sequence of residual error pictures;
performing an enhancement motion combination (1076) to
25 obtain a sequence of pictures of the enhancement
scaling layer by using the enhancement sequence of
residual error pictures and enhancement motion data.
27. Computer program for performing a method according to
30 claim 15 or 26, when the method runs on a computer.
28. Computer readable medium with a coded video sequence
having a base scaling layer (1002) and an enhancement
scaling layer (1004), wherein the coded video sequence
35 is formed such that it results in a decoded first
scaling layer and a decoded second scaling layer when
it is decoded in an apparatus for decoding according
to claim 16.

A video coder performs a motion-compensated prediction
(1906, 1012, 1014, 1016) both in the base layer (1002) and
in an enhancement layer (1004) to determine motion data of
the enhancement layer by using the motion data from the
base layer and/or to predict sequences of residual error
pictures after the motion-compensated prediction in the
enhancement layer by using sequences of residual error
pictures from the base layer via an intermediate layer
predictor (1018). On the decoder side, an intermediate
layer combiner is used for canceling this intermediate
layer prediction. Thereby, the data rate is improved
compared to scalability schemes without intermediate layer
prediction with the same picture quality.