DESCRIPTION
IMAGE PROCESSING DEVICE AND IMAGE PROCESSING METHOD
TECHNICAL FIELD
5 [0001]
This technique relates to image processing devices
and image processing methods. More particularly, this
technique provides an image processing device and an
image processing method for performing image processing
10 compatible with extended macroblocks.
BACKGROUND ART
[0002]
In recent years, apparatuses that handle image
15 information as digital information and achieve highefficiency
information transmission and accumulation in
doing so, or apparatuses compliant with a standard such
as MPEG for compression through orthogonal transforms
like discrete cosine transforms and motion compensation,
20 have.been spreading among broadcast stations and general
households.
[0003]
Particularly, MPEG2 (ISO/IEC 13818-2) is defined as
a general-purpose image encoding technique, and is
25 currently used for a wide range of applications for
professionals and general consumers.
[0004]
Also, a standard called H.26L (ITU-T Q6/16 VCEG) is
being developed for encoding images to be used in video
30 conferences. Compared with the conventional encoding
techniques such as MPEG2 and MPEG4, H.26L requires a
1
SP306420WOOO
larger amount of calculation in encoding and decoding,
but is known to achieve higher encoding efficiency. As
part of the MPEG4 activities, a standard for achieving
even higher encoding efficiency based on H.26L is being
5 currently developed as Joint Model of Enhanced-
'
Compression Video Coding, and has already been set as an
international standard under the name of H.264 and MPEG-4
Part 10 (hereinafter referred to as H.264/AVC (Advanced
Video Coding)) .
10 [0005]
Further, there is an increasing demand for encoding
at a higher compression rate so as to compress 4000 x
2000 pixels images and the like, or distribute highdefinition
images in today's circumstances where
15 transmission capacities are limited as in the Internet.
In view of this, Non-Patent Document 1 suggests an
increase of the macroblock size to 32 x 32 pixels, for
example, which is larger than that in MPEG2 and H.264/AVC.
That is, in Non-Patent Document 1, a hierarchical
20 structure is used for macroblocks so that blocks of 16 x
16 pixels and smaller blocks are compatible with the
macroblocks in H.264/AVC. As the supersets of those
blocks, larger blocks than those blocks are defined.
25 CITATION LIST
NON-PATENT DOCUMENT
[0006]
Non-Patent Document 1: "Video Coding Using Extended Block
Sizes" (Study Group 16, Contribution 123, ITU, January
30 2009)
2
SUMMARY OF THE INVENTION
PROBLEMS TO BE SOLVED BY THE INVENTION
[0007]
SP306420WOOO
In H.264/AVC, a macroblock formed with 16 x 16
5 pixels is divided into 16 x 16, 16 x 8, 8 x 16, or 8 x 8
sections, and the sections can have motion vector
information independently of one another. Each 8 x 8
section is further divided into 8 x 8, 8 x 4, 4 x 8, or 4
x 4 sub-sections, and the sub-sections can have motion
10 vector information independently of one another. As such
a motion prediction/compensation operation is performed,
the number of sections in a macroblock is increased, and
the amount of motion vector information becomes enormous.
If the motion vector information is encoded as it is, the
15 encoding efficiency becomes lower. Therefore, in
H.264/AVC, the motion compensation block size is
determined so that the total number of motion vectors in
two successive macroblocks does not exceed a
predetermined upper limit value. In view of this, when
20 the upper limit value defined in the H.264/AVC standard
is used in a case where the macroblock size is extended,
the block size cannot be made smaller due to the
limitation on the number of motion vectors, and image
quality degradation or the like might occur.
25 [0008]
30
Therefore, the present technique aims to provide
image processing devices and image processing methods for
performing image processing compatible with extended
macroblocks.
SOLUTION TO PROBLEMS
3
SP306420WOOO
[0009]
A first aspect of this technique is an image
processing device that includes: a setting unit that sets
the upper limit value of the number of motion vectors in
5 accordance with the block size of a block that is a unit
for encoding to be performed when image data is encoded;
a motion prediction/compensation unit that detects a
prediction mode when the number of motion vectors of the
block is equal to or smaller than the upper limit value
10 set by the setting unit, and performs motion compensation
on a reference image by using the motion vectors of the
detected prediction mode, to generate a predicted image;
and an encoding unit that encodes the image data to
generate an encoded stream by using the predicted image
15 generated by the motion prediction/compensation unit.
[0010]
According to the present technique, the motion
vector number limit setting unit sets the upper limit
value of the number of motion vectors in accordance with
20 macroblock size. In a case where the macroblock is
larger than a predetermined size such as 16 x 16 pixels,
the upper limit value is set at a larger value than the
upper limit value in a 16 x 16 pixel macroblock. In a
case where the upper limit value of the number of motion
25 vectors is p when the macroblock size is 16 x 16 pixels,
and the size of the extended macroblock is (16 x m) x (16
x n) pixels, the upper limit value is set at (p x m x n).
The motion prediction/compensation unit detects an
optimum prediction mode by excluding each prediction mode
30 having a larger number of motion vectors than the upper
limit value from the candidates for the optimum
4
• SP306420WOOO
prediction mode. Using the motion vectors of the
detected optimum prediction mode, the motion
prediction/compensation unit performs motion compensation
on a reference image, to generate a predicted image. The
5 motion prediction/compensation unit also monitors the
number of times control is performed to restrict the
number of motion vectors to a smaller number than the
upper limit value in each frame. When the number of
times control is performed exceeds a predetermined number
10 of times, the macroblocks in the next frame are made
larger. When a scene change is detected by a scene
change detection unit, the macroblocks are adjusted to a
predetermined size set in advance. Further, in a case
where an operation mode to generate a predicted image by
15 detecting an optimum prediction mode, regardless of the
number of motion vectors, and performing motion
compensation on a reference image with the use of the
motion vectors of the detected optimum prediction mode is
set, the motion prediction/compensation unit incorporates
20 determination information indicating that the number of
motion vectors is larger than the upper limit value into
the compressed image information generated by using the
predicted image when the number of motion vectors in the
optimum prediction mode is larger than the upper limit
25 value in the operation mode. Also, a block having a
larger size than the predetermined size is the largest
unit for encoding with the largest size among the units
for encoding in a hierarchical structure, for example.
[0011]
30 A second aspect of this technique is an image
processing method that includes the steps of: setting the
5
• SP306420W000
upper limit value of the number of motion vectors in
accordance with the block size of a block that is a unit
for encoding to be performed when image data is encoded;
generating a predicted image by detecting a prediction
5 mode when the number of motion vectors of the block is
equal to or smaller than the set upper limit value, and
performing motion compensation on a reference image with
the use of the motion vectors of the detected prediction
mode; and generating an encoded stream by encoding the
10 image data with the use of the generated predicted image.
[0012]
A third aspect of this technique is an image
processing device that includes: a setting unit that sets
the upper limit value of the number of motion vectors in
15 accordance with the block size of a block that is a unit
for decoding to be performed when an encoded stream
generated by encoding image data is decoded; a motion
compensation unit that detects a prediction mode when the
number of motion vectors of the block is equal to or
20 smaller than the upper limit value set by the setting
unit, and performs motion compensation on a reference
image by using the motion vectors of the detected
prediction mode, to generate a predicted image; and a
decoding unit that decodes the encoded stream by using
25 the predicted image generated by the motion compensation
unit.
[0013]
According to the present technique, the motion
vector number limit setting unit sets the upper limit
30 value of the number of motion vectors in accordance with
the macroblock size when an encoded stream generated by
6
• SP306420WOOO
encoding image data is decoded. In a case where the
macroblock is larger than a predetermined size such as 16
x 16 pixels, the upper limit value is set at a larger
value than the upper limit value in a 16 x 16 pixel
5 macroblock. In a case where the upper limit value of the
number of motion vectors is p when the macroblock size is
16 x 16 pixels, and the size of the extended macroblock
is (16 x m) x (16 x n) pixels, the upper limit value is
set at {p x m x n) . When generating a predicted image by
10 performing motion compensation on a reference image based
on motion vector information contained in compressed
image information, the motion compensation unit detects a
prediction mode while the number of motion vectors is
equal to or smaller than the upper limit value. Using
15 the motion vectors of the detected prediction mode, the
motion compensation unit performs motion compensation on
a reference image, to generate a predicted image. Using
the generated predicted image, the decoding unit decodes
the encoded stream. When detecting a prediction mode,
20 the motion compensation unit excludes each prediction
mode having a larger number of motion vectors than the
upper limit value from the candidates for the prediction
mode to be detected. Further, the motion compensation
unit monitors the number of times control is performed to
25 restrict the number of motion vectors to a smaller value
than the upper limit value in each frame. When the
number of times control is performed exceeds a
predetermined number of times, the blocks in the next
frame are made larger in size. In a case where a scene
30 change is detected in image data, the block size is
adjusted to a predetermined size set in advance. In an
7
• SP306420WOOO
operation mode to generate a predicted image by detecting
an optimum prediction mode, regardless of the number of
motion vectors, and performing motion compensation on a
reference image with the use of the motion vectors of the
5 detected optimum prediction mode, determination
information indicating that the number of motion vectors
is larger than the upper limit value is obtained from the
encoded stream. Size identification information for
identifying the block size of each block having an upper
10 limit value set thereon is also obtained from the encoded
stream.
[0014]
A fourth aspect of this technique is an image
processing method that includes: setting the upper limit
15 value of the number of motion vectors in accordance with
the block size of a block that is a unit for decoding to
be performed when an encoded stream generated by encoding
image data is decoded; generating a predicted image by
detecting a prediction mode when the number of motion
20 vectors of the block is equal to or smaller than the
upper limit value set by the setting unit, and performing
motion compensation on a reference image with the use of
the motion vectors of the detected prediction mode; and
decoding the encoded stream by using the generated
25 predicted image.
EFFECTS OF THE INVENTION
[0015]
· According to this technique, the upper limit value
30 of the number of motion vectors is set in accordance with
macroblock size. An optimum prediction mode having a
8
• SP306420WOOO
smaller number of motion vectors than the upper limit
value is detected, and motion compensation using the
motion vectors of the detected optimum prediction mode is
performed on a reference image, to generate a predicted
5 image. Accordingly, when macroblocks of an extended size
are used, the number of motion vectors is not restricted
to a smaller number than necessary, and image processing
compatible with the extended macroblocks can be performed.
10 BRIEF DESCRIPTION OF DRAWINGS
[0016]
Fig. 1 is a diagram illustrating the structure of
an image encoding device.
Figs. 2 are diagrams showing macroblocks in
15 H.264/AVC.
Fig. 3 is a diagram showing a motion compensation
block and adjacent blocks.
Fig. 4 is a diagram for explaining Multi-Reference
Frame.
20 Fig. 5 is a diagram for explaining a temporal
direct mode.
Fig. 6 is a diagram for explaining a motion
prediction/compensation operation with 1/4 pixel
precision.
25 Figs. 7 are diagrams showing pixels to be used in
30
motion prediction/compensation operations with 1/4 pixel
precision using a 6-tap FIR filter.
Fig. 8 is a diagram for explaining the upper limit
value of the number of motion vectors.
Figs. 9 illustrate a hierarchical structure in a
case where macroblock sizes are extended.
9
• SP306420WOOO
Fig. 10 is a diagram showing the structure of the
motion prediction/compensation unit and the motion vector
number limit setting unit.
Fig. 11 is a flowchart showing an operation of the
5 image encoding device.
Fig. 12 is a flowchart showing prediction
operations.
Fig. 13 is a flowchart showing intra prediction
operations.
10 Fig. 14 is a flowchart showing inter prediction
operations.
Fig. 15 is a flowchart showing the setting of the
upper limit value of the number of motion vectors.
Fig. 16 is a diagram illustrating the structure of
15 another image encoding device.
Fig. 17 is a diagram showing the structure of
another motion prediction/compensation unit and the
motion vector number limit setting unit.
Fig. 18 is a diagram illustrating the structure of
20 an image decoding device.
Fig. 19 is a diagram showing the structure of the
motion compensation unit and the motion vector number
limit setting unit.
Fig. 20 is a flowchart showing an operation of the
25 image decoding device.
Fig. 21 is a flowchart showing a predicted image
generating operation.
Fig. 22 is a flowchart showing an inter-predicted
image generating operation.
30 Fig. 23 is a diagram showing an example of slice
headers.
10
•
5
SP306420WOOO
Fig. 24 is a diagram schematically showing an
example structure of a television apparatus.
Fig. 25 is a diagram schematically showing an
example structure of a portable telephone device.
Fig. 26 is a diagram schematically showing an
example structure of a recording/reproducing apparatus.
Fig. 27 is a diagram schematically showing an
example structure of an imaging apparatus.
10 MODE FOR CARRYING OUT THE INVENTION
[0017]
The following is a description of embodiments.
According to the present technique, when an upper limit
value that is set in accordance with a level as in
15 H.264/AVC is used in a case where macroblocks are
extended in size, the number of motion vectors is limited
to a smaller number than necessary, and image quality
might be degraded. Therefore, in a case where
macroblocks of an extended size are used, the upper limit
20 value is set in accordance with the macroblock size, so
as to guarantee real-time operations in view of memory
bandwidth, with the number of motion vectors not being
restricted to a smaller number than necessary. Also, the
number of motion vectors is not restricted to a smaller
25 number than necessary, and image quality degradation is
prevented. Explanation will be made in the following
order.
1. Structure of an Image Encoding Device
2. Operations of the Image Encoding Device
30 3. Structure of Another Image Encoding Device
4. Other Operations of an Image Encoding Device
11
• SP306420WOOO
5. Structure of an Image Decoding Device
6. Operations of the Image Decoding Device
7. Case of Software Processing
8. Applications to Electronic Apparatuses
5 [0018]
[1. Structure of an Image Encoding Device]
Fig. 1 illustrates the structure of an image
encoding device that is an image processing device
performing image encoding. The image encoding device 10
10 includes an analog/digital conversion unit (an A/D
conversion unit) 11, a picture rearrangement buffer 12, a
subtraction unit 13, an orthogonal transform unit 14, a
quantization unit 15, a lossless encoding unit 16, an
accumulation buffer 17, and a rate control unit 18. The
15 image encoding device 10 further includes an inverse
quantization unit 21, an inverse orthogonal transform
unit 22, an addition unit 23, a deblocking filter 24, a
frame memory 25, an intra prediction unit 31, a motion
prediction/compensation unit 32, and a predicted
20 image/optimum mode select unit 33.
[0019]
The A/D conversion unit 11 converts analog image
signals into digital image data, and outputs the image
data to the picture rearrangement buffer 12.
25 [0020]
The picture rearrangement buffer 12 rearranges the
frames of the image data output from the A/D conversion
unit 11. The picture rearrangement buffer 12 rearranges
the frames in accordance with the GOP (Group of Pictures)
30 structure related to encoding operations, and outputs the
rearranged image data to the subtraction unit 13, the
12
•
intra prediction unit 31, and the motion
prediction/compensation unit 32.
[0021]
SP306420WOOO
The subtraction unit 13 receives the image data
5 output from the picture rearrangement buffer 12 and
predicted image data selected by the later described
predicted image/optimum mode select unit 33. The
subtraction unit 13 calculates prediction error data that
is the difference between the image data output from the
10 picture rearrangement buffer 12 and the predicted image
data supplied from the predicted image/optimum mode
select unit 33, and outputs the prediction error data to
the orthogonal transform unit 14.
15
[0022]
The orthogonal transform unit 14 performs an
orthogonal transform operation, such as a discrete cosine
transform (DCT) or a Karhunen-Loeve transform, on the
prediction error data output from the subtraction unit 13.
The orthogonal transform unit 14 outputs transform
20 coefficient data obtained by performing the orthogonal
transform operation to the quantization unit 15.
[0023]
The quantization unit 15 receives the transform
coefficient data output from the orthogonal transform
25 unit 14 and a rate control signal supplied from the later
described rate control unit 18. The quantization unit 15
quantizes the transform coefficient data, and outputs the
quantized data to the lossless encoding unit 16 and the
inverse quantization unit 21. Based on the rate control
30 signal supplied from the rate control unit 18, the
quantization unit 15 switches quantization parameters
13
• SP306420WOOO
(quantization scales), to change the bit rate of the
quantized data.
[0024]
The lossless encoding unit 16 receives the
5 quantized data output from the quantization unit 15, and
prediction mode information supplied from the later
described intra prediction unit 31, the motion
prediction/compensation unit 32, and the predicted
image/optimum mode select unit 33. The prediction mode
10 information contains information indicating prediction
mode, macroblock size, macroblock type, and motion
vectors, in accordance with whether the prediction mode
is an intra prediction mode or an inter prediction mode.
The prediction mode information may also include the
15 later described determination information, size
identification information, or the like. The lossless
encoding unit 16 performs a lossless encoding operation
on the quantized data through variable-length coding or
arithmetic coding or the like, to generate and output
20 compressed image information to the accumulation buffer
17. Further, the lossless encoding unit 16 performs
lossless encoding on the prediction mode information, and
adds the encoded prediction mode information to the
. compressed image information. For example, the lossless
25 encoding unit 16 adds the encoded prediction mode
information to the header information in an encoded
stream that is the compressed image information.
[0025]
The accumulation buffer 17 stores the compressed
30 image information supplied from the lossless encoding
unit 16. The accumulation buffer 17 also outputs the
14
• SP306420WOOO
stored compressed image information at the transmission
rate suitable for the transmission path.
[0026]
The rate control unit 18 monitors the free space in
5 the accumulation buffer 17. The rate control unit 18
generates the rate control signal in accordance with the
free space, and outputs the rate control signal to the
quantization unit 15. The rate control unit 18 obtains
information indicating the free space from the
10 accumulation buffer 17, for example. In a case where the
remaining free space is small, the rate control unit 18
lowers the bit rate of the quantized data through the
rate control signal. In a case where the remaining free
space in the accumulation buffer 17 is sufficiently large,
15 the rate control unit 18 increases the bit rate of the
quantized data through the rate control signal.
[0027]
The inverse quantization unit 21 inversely
quantizes the quantized data supplied from the
20 quantization unit 15. The inverse quantization unit 21
outputs the transform coefficient data obtained by
performing the inverse quantization operation to the
inverse orthogonal transform unit 22.
25
[0028]
The inverse orthogonal transform unit 22 performs
an inverse orthogonal transform operation on the
transform coefficient data supplied from the inverse
quantization unit 21, and outputs the resultant data to
the addition unit 23.
30 [0029]
The addition unit 23 adds the data supplied from
15
SP306420WOOO
the inverse orthogonal transform unit 22 to the predicted
image data supplied from predicted image/optimum mode
select unit 33, to generate decoded image data. The
addition unit 23 then outputs the decoded image data to
5 the deblocking filter 24 and the frame memory 25. The
decoded image data is used as the image data of a
reference image.
[0030]
The deblocking filter 24 performs a filtering
10 operation to reduce block distortions that occur at the
time of image encoding. The deblocking filter 24
performs a filtering operation to remove block
distortions from the decoded image data supplied from the
addition unit 23, and outputs the filtered decoded image
15 data to the frame memory 25.
[0031]
The frame memory 25 stores the decoded image data
that has been subjected to the filtering operation and is
supplied from the deblocking filter 24. The decoded
20 image data stored in the frame memory 25 is to be used as
reference image data by the motion
prediction/compensation unit 32.
[0032]
The intra prediction unit 31 performs intra
25 prediction operations in all candidate intra prediction
modes by using the image data that is output from the
picture rearrangement buffer 12 and is about the images
to be encoded, and the decoded image data supplied from
the addition unit 23. The intra prediction unit 31
30 further calculates a cost function value in each of the
intra prediction modes, and selects an optimum intra
16
• SP306420WOOO
prediction mode that is the intra prediction mode with
the smallest cost function value calculated or the intra
prediction mode with the highest encoding efficiency.
The intra prediction unit 31 outputs the predicted image
5 data generated in the optimum intra prediction mode, the
prediction mode information about the optimum intra
prediction mode, and the cost function value in the
optimum intra prediction mode, to the predicted
image/optimum mode select unit 33.
10 [Q033]
The motion prediction/compensation unit 32 performs
a motion prediction/compensation operation in each
motion-compensated block size. Using the reference image
data that is read from the frame memory 25, the motion
15 prediction/compensation unit 32 detects motion vectors
from the images to be encoded that are read from the
picture rearrangement buffer 12 in each prediction mode
or from each motion compensation block in each mode.
Based on the detected motion vectors, the motion
20 prediction/compensation unit 32 further performs a motion
compensation operation on the decoded image, to generate
predicted image data.
[0034]
The motion prediction/compensation unit 32 also
25 calculates a cost function value in each prediction mode.
The motion prediction/compensation unit 32 selects an
optimum inter prediction mode that is the motioncompensated
block size that has the smallest cost
function value calculated, and has a total number of
30 motion vectors of two successive macroblocks that is not
larger than the upper limit value set by the later
17
• SP306420WOOO
described motion vector number limit setting unit 35.
The motion prediction/compensation unit 32 outputs the
predicted image data generated in the optimum inter
prediction mode, the prediction mode information about
5 the optimum inter prediction mode, and the cost function
value and the motion vector information in the optimum
inter prediction mode, to the predicted image/optimum
mode select unit 33. The motion prediction/compensation
unit 32 also performs predictions in a skipped macroblock
10 mode and a direct mode as inter prediction modes.
[0035]
The predicted image/optimum mode select unit 33
compares the cost function value supplied from the intra
prediction unit 31 with the cost function value supplied
15 from the motion prediction/compensation unit 32, and
selects the smaller cost function value as the optimum
mode with the highest encoding efficiency. The predicted
image/optimum mode select unit 33 also outputs the
predicted image data generated in the optimum mode to the
20 subtraction unit 13 and the addition unit 23. Further,
the predicted image/optimum mode select unit 33 outputs
the prediction mode information about the optimum mode to
the lossless encoding unit 16. The predicted
image/optimum mode select unit 33 performs intra
25 predictions or inter predictions on the slice basis.
[0036]
Not complying with H.264/AVC, the motion vector
number limit setting unit 35 sets the upper limit value
of the number of motion vectors in accordance with
30 macroblock size, and outputs the upper limit value to the
motion prediction/compensation unit 32. Limiting of the
18
• SP306420WOOO
number of motion vectors in H.264/AVC is now described.
[0037]
In H.264/AVC, a macroblock formed with 16 x 16
pixels is divided into 16 x 16, 16 x 8, 8 x 16, or 8 x 8
5 sections, as shown in Fig. 2(A), and the sections can
have motion vector information independently of one
another. Each 8 x 8 section is further divided into 8 x
8, 8 x 4, 4 x 8, or 4 x 4 sub-sections, as shown in Fig.
2(B), and the sub-sections can have motion vector
10 information independently of one another. As such a
motion prediction/compensation operation is performed,
the amount of motion vector information becomes enormous.
If the motion vector information is encoded as it is, the
encoding efficiency becomes lower. Therefore, in
15 H.264/AVC, the amount of motion vector information is
reduced in the following manner.
[0038]
In Fig. 3, block "E" is the motion compensation
block to be encoded, and blocks "A" through "D" are
20 already encoded blocks and are adjacent to the block "E".
[ 0039]
Here, x is A, B, C, D, or E, and mvx represents the
motion vector information about x. First, by using the
motion vector information about the blocks A, B, and C,
25 predicted motion vector information pmvE about the motion
compensation block E is generated through a median
operation according to the equation (1):
30
pmvE = med (mvA, mvB, mvC) ( 1)
[0040]
In a case where the information about the block "C"
is "unavailable" for the block "C" is located at a corner
19
SP306420WOOO
of the image frame or the like, the information about the
block "D" is used instead.
[0041]
In the compressed image information, the data mvdE
5 to be encoded as the motion vector information about the
motion compensation block "E" is generated by using pmvE
according to the equation (2):
mvdE = mvE - pmvE (2)
In an actual operation, processing is performed on the
10 horizontal component and the vertical component of the
motion vector information independently of each other.
[0042]
By using the data mvdE, which is the difference
from the predicted motion vector information pmvE, as the
15 motion vector information as described above, a reduction
of the amount of motion vector information is achieved.
[0043]
In H.264/AVC, a technique called Multi-Reference
Frame, which is not specified in image information
20 encoding techniques such as MPEG-2 and H.263, is
specified. Fig. 4 is a diagram for explaining MultiReference
Frame. In H.264/AVC, reference frames are
stored in a memory, and it is possible to refer to a
different frame for each motion compensation block, as
25 shown in Fig. 4.
[0044]
For B pictures in H.264/AVC, a direct mode is set.
In the direct mode, motion vector information is not
stored in the compressed image information, but the
30 motion vector information about each motion compensation
block is extracted from the motion vector information
20
SP306420WOOO
about an adjacent block or a co-located block in an image
decoding device.
[0045]
In the direct mode, a spatial direct mode and a
5 temporal direct mode are set, and one of the two modes
can be selected for each slice.
[0046]
In the spatial direct mode, motion vector
information generated through median prediction is
10 applied to each motion compensation block, as shown in
the equation (3) :
mvE = pmvE ( 3)
[0047]
Fig. 5 1s a diagram for explaining the temporal
15 direct mode. In Fig; 5, the block located at the address
in the same space as a motion compensation block in a LO
reference picture is set as a Co-Located block, and the
motion vector information in the Co-Located block is set
as mvcol. Also, TOB represents the distance between the
20 picture and the LO reference picture on the temporal axis,
and TOO represents the distance between the LO reference
picture and the L1 reference picture on the temporal axis.
[0048]
Here, LO motion vector information mvLO and L1
25 motion vector information mvL1 in the picture are
generated as indicated by the equations (4) and (5):
mvLO (TOB/TOO)mvcol (4)
mvL1 ((TOO-TOB)/TOO)mvcol ( 5)
In the compressed image information, the information
30 indicating the distance on the temporal axis does not
exist, and therefore, a calculation is performed by using
21
• SP306420WOOO
POC (Picture Order Count).
[0049]
In H.264/AVC, a motion prediction/compensation
operation with 1/4 pixel precision is performed by using
5 a 6-tap FIR (Finite Impulse Response) filter, to increase
encoding efficiency. Fig. 6 is a diagram for explaining
the motion prediction/compensation operation with 1/4
pixel precision.
10
[0050]
In Fig. 6, "A" represents the location of each
integer precision pixel stored in the frame memory, "b",
"c", and "d" represent the locations of 1/2 pixel
precision pixels, "e1", "e2", and "e3" represent the
locations of 1/4 pixel precision pixels.
15 [ 0051]
Clip1() is defined as shown in the equation (6):
[Mathematical Formula 1]
{
-O;if(a max_pix)
( 6)
In the equation (6), the value of max_pix is 255 in a
case where an input image has 8-bit precision.
20 [0052]
25
The pixel values at the locations "b" and "d" are
generated by using a 6-tap FIR filter as shown in the
equations (7) and (8):
F = A-2 - 5•A-l + 20•A0 + 20•Al - 5•A2 + A3 . . . (7)
b, d = Clip1((F + 16) >> 5) ( 8)
[0053]
22
.. SP306420WOOO
The pixel value at the location "c" is generated by
using a 6-tap FIR filter as shown in the equation (9) or
(10) and the equation (11):
F b-2 - 5•b-1 + 20•bo + 20•b1 - 5•b2 + b3 (9)
5 F d-2 - 5•d-1 + 20•do + 20•d1 - 5•d2 + d3 ( 10)
c = Clip1 ( (F+512) >> 10) (11)
The Clip1 processing is performed only once at last after
product-sum operations are performed both in the
horizontal direction and the vertical direction.
10 [0054]
15
The pixel values at the locations "e1" through "e3"
are generated by linear interpolations as shown in the
equations (12) through (14):
[ 0055]
e1 (A + b + 1) >> 1 (12)
e2
e3
(b + d + 1) >> 1
(b + c + 1) >> 1
(13)
(14)
In such a motion prediction/compensation operation
with 1/4 pixel precision, the pixel value of the 21 x 21
20 pixel block shown in Fig. 7(A) is used in a case where
the size of the motion compensation block is 16 x 16
pixels. In a case where the size of the motion
compensation block is 4 x 4 pixels, the pixel value of
the 9 x 9 pixel block shown in Fig. 7(B) is used. As
25 described above, in a motion prediction/compensation
operation with 1/4 pixel precision using a 6-tap FIR
filter, a 5-pixel overhead is required for each block
size in both the horizontal direction and the vertical
direction. Therefore, in a case where the block size is
30 16 x 16 pixels, a pixel value ((21x21)/(16x16)) times
larger is required, and in a case where the block s.ize is
23
• SP306420WOOO
4 x 4 pixels, a pixel value ((9x9)/(4x4)) times larger is
required. That is, if the image encoding device selects
a larger number of small block sizes, the pixel values of
a larger number of pixels are required, and a greater
5 memory bandwidth is required.
[0056]
Therefore, in H.264/AVC, the motion compensation
block size is determined so that the total number of
motion vectors in two successive macroblocks does not
10 exceed a predetermined upper limit value, as shown in Fig.
8. In this manner, increases in memory bandwidth are
prevented. Specifically, the total number of the motion
vector number N(a) in a macroblock MB(a) and the motion
vector number N(a+l) in a macroblock MB(a+l) is
15 restrained from exceeding an upper limit value NHL (16 in
the case of level 4 (the resolution of HDTV) . Likewise,
the total number of the motion vector number N(a+l) in
the macroblock MB(a+l) and the motion vector number
N(a+2) in a macroblock MB(a+2) is restrained from
20 exceeding the upper limit value NHL. As described above,
the total number of motion vectors in two successive
macroblocks is restrained from exceeding the
predetermined upper limit value NHL. For example, the
macroblock size of a B picture is 16 x 16 pixels, and the
25 motion compensation block size is 4 x 4 pixels. In this
case, 16 motion compensation blocks are included in one
macroblock, and the number of motion vectors is 16 x 2,
which is larger than the upper limit value NHL (= 16) .
Therefore, the motion compensation block size is
30 determined so that the number of motion vectors does not
exceed the upper limit value.
24
• SP306420WOOO
[0057]
Figs. 9 illustrate a hierarchical structure in a
case where macroblock sizes are extended. 8 x 8 pixels
sub-macroblockOf Figs. 9, Figs. 9(C) and 9(D) show a 16 x
5 16 pixel macroblock and an 8 x 8 pixel sub-macroblock
defined in the H.264/AVC standard. As macroblocks having
sizes made larger from those in H.264/AVC, a 64 x 64
pixel macroblock shown in Fig. 9(A) and a 32 x 32 pixel
macroblock shown in Fig. 9(B) are defined. It should be
10 noted that, in Figs. 9, "skip/direct" indicates a block
size used in a case where a skipped macroblock mode or a
direct mode is selected. Also, "ME" indicates a motion
compensation block size. "P8 x 8" indicates that the
block can be further divided in a lower hierarchical
15 level with a smaller block size.
[0058]
As described above, when the upper limit value
defined in the H.264/AVC standard is also used in a case
where the macroblock size is extended, the number of
20 motion vectors is limited to a smaller number than
necessary, and image quality might be degraded. For
example, if the upper limit value that is used in cases
where each macroblock is 16 x 16 pixels in size is used
in a 64 x 64 pixel macroblock, the number of motion
25 vectors is limited to a smaller number than necessary,
since the upper limit value remains the same though the
block size is four times larger. As a result, the motion
compensation block size cannot be made smaller, and image
quality degradation or the like might occur.
30 [ 0059]
Therefore, the motion vector number limit setting
25
• SP306420WOOO
unit 35 sets the upper limit value of the number of
motion vectors in accordance with macroblock size. In a
case ·where the size is larger than a macroblock of a
predetermined size (N), the motion vector number limit
5 setting unit 35 sets a larger upper limit value than the
upper limit value in a macroblock of the predetermined
size (N), to guarantee real-time operations in view of
memory bandwidth, with the number of motion vectors not
being restricted to a smaller number than necessary. For
10 example, in a case where the upper limit value of the
number of motion vectors is p when the macroblock size is
16 x 16 pixels (N = 16), and the macroblock size is (16 x
m) x (16 x n) pixels, the motion vector number limit
setting unit 35 sets the upper limit value at (m x n x p),
15 which is in predetermined ratio to the macroblock size.
In this manner, real-time operations are guaranteed in
view of memory bandwidth, with the number of motion
vectors not being restricted to a smaller number than
necessary.
20 [0060]
25
30
Fig. 10 is a diagram showing the structure of the
motion prediction/compensation unit and the motion vector
number limit setting unit.
[0061]
The motion prediction/compensation unit 32 includes
a motion search unit 321, a mode determination unit 322,
a motion compensation processing unit 323, and a motion
vector buffer 324.
[0062]
Rearranged image data supplied from the picture
rearrangement buffer 12, and reference image data read
26
• SP306420WOOO
from the frame memory 25 are supplied to the motion
search unit 321.
[0063]
The motion search unit 321 performs motion search
5 operations in all candidate inter prediction modes, and
outputs motion vector information indicating the motion
vectors detected through the motion search operations to
the mode determination unit 322. Based on the detected
motion vectors, the motion search unit 321 performs
10 motion compensation .on the reference image data, to
generate and output predicted image data to the mode
determination unit 322. The motion search unit 321
further outputs information indicating the macroblock
size, or size identification information for identifying
15 the block size of a block on which an upper limit value
is set, to the motion vector number limit setting unit 35.
[0064]
Based on the size identification information, the
motion vector number limit setting unit 35 sets the upper
20 limit value of the number of motion vectors in accordance
with the block size, and notifies the mode det~rmination
unit 322 of the set upper limit value.
[0065]
The motion vector information and predicted image
25 data about all the candidate inter prediction modes are
supplied from the motion search unit 321 to the mode
determination unit 322. Motion vector information that
is generated at an encoded adjacent block is also
supplied from the later described motion vector buffer
30 324 to the mode determination unit 322. Further,
rearranged image data supplied from the picture
27
• SP306420WOOO
rearrangement buffer 12 is also supplied to the mode
determination unit 322 to calculate cost functions.
[0066]
The mode determination unit 322 calculates a cost
5 function value in each of the candidates for the optimum
inter prediction mode by using the image data supplied
from the picture rearrangement buffer 12 and the
predicted image data supplied from the motion search unit
321. The mode determination unit 322 selects the mode
10 with the smallest cost function value as the optimum
inter prediction mode for the macroblock being encoded.
Using the motion vector information about the adjacent
macroblock supplied from the motion vector buffer 324,
the mode determination unit 322 calculates the total
15 number of motion vectors in the two successive
macroblocks. The mode determination unit 322 excludes
each mode with the calculated value larger than the upper
limit value set by the motion vector number limit setting
unit 35, from the candidate modes, and then determines
20 the optimum inter prediction mode.
[0067]
Further, the mode determination unit 322 outputs
the prediction mode information, the motion vector
information, and the cost function value of the mode
25 determined to be the optimum mode, to the motion
compensation processing unit 323. The prediction mode
information may contain the size identification
information for identifying the block size of each block
having an upper limit value set thereon. Based on the
30 motion vector information supplied from the motion vector
buffer 324, the mode determination unit 322 also
28
• SP306420WOOO
calculates predicted motion vectors. The mode
determination unit 322 calculates difference motion
vectors that are the differences between the calculated
predicted motion vectors and the motion vectors detected
5 by the motion search unit 321, and outputs difference
motion vector information indicating the difference
motion vectors containedlin the motion vector information
to the motion compensation processing unit 323.
[0068]
10 Using the image data about reference images
supplied from the frame memory 25, the motion
compensation processing unit 323 performs motion
compensation based on the motion vector information
supplied from the mode determination unit 322, to
15 generate predicted image data. The motion compensation
processing unit 323 outputs the generated predicted image
data, the cost function value, and the prediction mode
information (containing the macroblock size, the
macroblock type, the prediction mode, the difference
20 motion vector information, and the like) to the predicted
image/optimum mode select unit 33. The motion
compensation processing unit 323 also outputs the motion
vector information to the motion vector buffer 324.
25
[0069]
The motion vector buffer 324 temporarily stores the
motion vector information supplied from the motion
compensation processing unit 323. The motion vector
buffer 324 also supplies the stored motion vector
information to the mode determination unit 322 to
30 calculate the total number 9f motion vectors in two
successive macroblocks and predicted motion vectors.
29
• SP306420WOOO
[0070]
[2. Operations of the Image Encoding Device]
Fig. 11 is a flowchart showing an operation of the
image encoding device. In step ST11, the A/D conversion
5 unit 11 performs an A/D conversion on an input image
signal.
[0071]
In step ST12, the picture rearrangement buffer 12
performs image rearrangement. The pictur~ rearrangement
10 buffer 12 stores the image data supplied from the A/D
conversion unit 11, and rearranges the respective
pictures in encoding order, instead of display order.
[0072]
In step ST13, the subtraction unit 13 generates
15 prediction error data. The subtraction unit 13 generates
the prediction error data by calculating the differences
between the image data of the images rearranged in step
ST12 and predicted image data selected by the predicted
image/optimum mode select unit 33. The prediction error
20 data has a smaller data amount than the original image
data. Accordingly, the data amount can be made smaller
tha~ in a case where images are directly encoded.
[0073]
In step ST14, the orthogonal transform unit 14
25 performs an orthogonal transform operation. The
orthogonal transform unit 14 orthogonally transforms the
prediction error data supplied from the subtraction unit
13. Specifically, orthogonal transforms such as discrete
cosine transforms or Karhunen-Loeve transforms are
30 performed on the prediction error data, and transform
coefficient data is output.
30
• SP306420WOOO
[0074]
In step ST15, the quantization unit 15 performs a
quantization operation. The quantization unit 15
quantizes the transform coefficient data. In the
5 quantization, rate control is performed as will be
described later in the description of step ST25.
[0075]
In step ST16, the inverse quantization unit 21
performs an inverse quantization operation. The inverse
10 quantization unit 21 inversely quantizes the transform
coefficient data quantized by the quantization unit 15,
having characteristics compatible with the
characteristics of the quantization unit 15.
15
[0076]
In step ST17, the inverse orthogonal transform unit
22 performs an inverse orthogonal transform operation.
The inverse orthogonal transform unit 22 performs an
inverse orthogonal transform on the transform coefficient
data inversely quantized by the inverse quantization unit
20 21, having the characteristics compatible with the
characteristics of the orthogonal transform unit 14.
[0077]
In step ST18, the addition unit 23 generates
reference image data. The addition unit 23 generates the
25 reference image data (decoded image data) by adding the
predicted image data supplied from the predicted
image/optimum mode select unit 33 to the data of the
location that corresponds to the predicted image and has
been subjected to the inverse orthogonal transform.
30 [0078]
In step ST19, the deblocking filter 24 performs a
31
•
5
SP306420WOOO
filtering operation. The deblocking filter 24 removes
block distortions by filtering the decoded image data
output from the addition unit 23.
[0079]
In step ST20, the frame memory 25 stores the
reference image data. The frame memory 25 stores the
filtered reference image data (the decoded image data).
[0080]
In step ST21, the intra prediction unit 31 and the
10 motion prediction/compensation unit 32 each perform
prediction operations. Specifically, the intra
prediction unit 31 performs intra prediction operations
in intra prediction modes, and the motion
prediction/compensation unit 32 performs motion
15 prediction/compensation operations in inter prediction
modes. The prediction operations will be described later
in detail with reference to Fig. 12. In this step,
prediction operations are performed in all candidate
prediction modes, and cost function values are calculated
20 in all the candidate prediction modes. Based on the
calculated cost function values, an optimum intra
prediction mode and an optimum inter prediction mode are
selected, and the predicted images generated in the
selected prediction modes, the cost functions, and the
25 prediction mode information are supplied to the predicted
image/optimum mode select unit 33.
[0081]
In step ST22, the predicted image/optimum mode
select unit 33 selects predicted image data. Based on
30 the respective cost function values output from the intra
prediction unit 31 and the motion prediction/compensation
32
• SP306420WOOO
unit 32, the predicted image/optimum mode select unit 33
determines the optimum mode to optimize the encoding
efficiency. The predicted image/optimum mode select unit
33 further selects the predicted image data in the
5 determined optimum mode, and outputs the selected
predicted image data to the subtraction unit 13 and the
addition unit 23. This predicted image is used in the
operations in steps ST13 and ST18, as described above.
The prediction mode information corresponding to the
10 selected predicted image data is output to the lossless
encoding unit 16.
[0082]
In step ST23, the lossless encoding unit 16
performs a lossless encoding operation. The lossless
15 encoding unit 16 performs lossless encoding on the
quantized data output from the quantization unit 15.
That is, lossless encoding such as variable-length coding
or arithmetic coding is performed on the quantized data,
to compress the data. At this point, lossless encoding
20 is also performed on the prediction mode information that
is input to the lossless encoding unit 16 in step ST22 as
described above. Further, the lossless-encoded data of
the prediction mode information is incorporated into the
compressed image information generated by performing
25 lossless encoding on the quantized data.
[0083]
In step ST24, the accumulation buffer 17 performs
an accumulation operation. The accumulation buffer 17
stores the compressed image information output from the
30 lossless encoding unit 16. The compressed image
information stored in the accumulation buffer 17 is read
33
• SP306420WOOO
and transmitted to the decoding side via .a transmission
path where necessary.
[0084]
In step ST25, the rate control unit 18 performs
5 rate control. The rate control unit 18 controls the
quantization operation rate of the quantization unit 15
so that an overflow or an underflow does not occur in the
accumulation buffer 17 when the accumulation buffer 17
stores compressed ima~e information.
10 [0085]
15
Referring now to the flowchart of Fig. 12, the
prediction operations in step ST21 of Fig. 11 are
described.
[0086]
In step ST31, the intra prediction unit 31 performs
intra prediction operations. The intra prediction unit
31 performs intra predictions on the image of the current
block in all the candidate intra prediction modes. The
image data of a decoded image to be referred to in each
20 intra prediction is decoded image data not yet subjected
to the blocking filtering at the deblocking filter 24.
In the intra prediction operations, ~ntra predictions are
performed in all the candidate intra prediction modes,
and cost function values are calculated in all the
25 candidate intra prediction modes. Based on the
calculated cost function values, the intra prediction
mode with the highest encoding efficiency is selected
from all the intra prediction modes.
[0087]
30 In step ST32, the motion prediction/compensation
unit 32 performs inter prediction operations. Using the
34
• SP306420WOOO
decoded image data that is stored in the frame memory 25
and has been subjected to the deblocking filtering, the
motion prediction/compensation unit 32 performs inter
prediction operations in the candidate inter prediction
5 modes. In the inter prediction operations, inter
prediction operations are performed in all the candidate
inter prediction modes, and cost function values are
calculated in all the candidate inter prediction modes.
Based on the calculated cost function values, the inter
10 prediction mode with the highest encoding efficiency is
selected from all the inter prediction modes.
[0088]
Referring now to the flowchart of Fig. 13, the
intra prediction operations in step ST31 of Fig. 11 are
15 described.
[0089]
In step ST41, the intra prediction unit 31 performs
intra prediction operations in the respective prediction
modes. Using the decoded image data not yet subjected to
20 the blocking filtering, the intra prediction unit 31
generates predicted image data in each intra prediction
mode.
[0090]
In step ST42, the intra prediction unit 31
25 calculates the cost function value in each prediction
mode. As specified in the JM (Joint Model), which is the
reference software in H.264/AVC, the cost function values
are calculated by the technique of High Complexity Mode
or Low Complexity Mode.
30 [0091]
Specifically, in the High Complexity Mode, the
35
• SP306420WOOO
operation that ends with the lossless encoding operation
is provisionally performed as the operation of the step
ST42 in each candidate prediction mode, to calculate the
cost function value expressed by the following equation
5 (15) in each prediction mode:
Cost (ModeEO) = D + A.•R (15)
[0092]
Here, 0 represents the universal set of the
candidate prediction modes for encoding the block or
10 macroblock. D represents the energy difference
(distortion) between the decoded image and an input image
in a case where encoding is performed in a prediction
mode. R represents the generated encoding amount
including orthogonal transform coefficients and
15 prediction mode information, and A. represents the
Lagrange multiplier given as the function of a
quantization parameter QP.
[0093]
That is, to perform encoding in the High Complexity
20 Mode, a provisional encoding operation needs to be
performed in all the candidate prediction modes to
calculate the above parameters D and R, and therefore, a
larger amount of calculation is required.
25
[0094]
In the Low Complexity Mode, on the other hand, the
generation of a predicted image and the calculation of
the header bit such as motion vector information and
prediction mode information are performed as the
operation of step ST42 in all the candidate prediction
30 modes, and the cost function value expressed by the
following equation (16) is calculated in each prediction
36
•
mode:
(16)
[0095]
Cost(ModeEO)
SP306420WOOO
D + QP2Quant(QP)•Header_Bit
5 Here, 0 represents the universal set of the
candidate prediction modes for encoding the block or
macroblock. D represents the energy difference
(distortion) between the decoded image and an input image
in a case where encoding is performed in a prediction
10 mode. Header_Bit represents the header bit corresponding
to the prediction mode, and QP2Quant is the function
given as the function of the quantization parameter QP.
[0096]
That is, in the Low Complexity Mode, a prediction
15 operation needs to be performed in each prediction mode,
but any decoded image is not required. Accordingly, the
amount of calculation can be smaller than that required
in the High Complexity Mode.
20
[0097]
In step ST43, the intra prediction unit 31
determines the optimum intra prediction mode. Based on
the cost function values calculated in step ST42, the
intra prediction unit 31 selects the one intra prediction
mode with the smallest cost function value among the
25 calculated cost function values, and determines the
selected intra prediction mode to be the optimum intra
prediction mode.
[0098]
Referring now to the flowchart of Fig. 14, the
30 inter prediction operations in step ST32 of Fig. 11 are
described. In step ST51, the motion vector number limit
37
SP306420WOOO
setting unit 35 sets the upper limit value of the number
of motion vectors.
[0099]
Fig. 15 is a flowchart showing the setting of the
5 upper limit value of the number of motion vectors. In
step ST61, the motion vector number limit setting unit 35
determines the macroblock size. The motion vector number
limit setting unit 35 obtains the size identification
information indicating the macroblock size for performing
10 inter predictions from the motion search unit 321, and
determines the macroblock size. The operation then moves
on to step ST62.
[0100]
In step ST62, the motion vector number limit
15 setting unit 35 calculates the upper limit value of the
number of motion vectors. In accordance with the
macroblock size, the motion vector number limit setting
unit 35 calculates the upper limit value of the number.
If the macroblock is larger than a 16 x 16 pixel
20 macroblock, the motion vector number limit setting unit
35 sets a larger upper limit value than the upper limit
value in a 16 x 16 pixel macroblock, to guarantee realtime
operations in view of memory bandwidth, with the
number of motion vectors not being restricted to a
25 smaller number than necessary. For example, in a case
where the upper limit value of the number of motion
vectors is p when the macroblock size is 16 x 16 pixels,
and the macroblock size is (16 x m) x (16 x n) pixels,
the motion vector number limit setting unit 35 sets the
30 upper limit value at (m x n x p). The operation then
returns to step ST52 of Fig. 14.
38
SP306420WOOO
[0101]
In step ST52, the motion prediction/compensation
unit 32 performs motion prediction operations. The
motion prediction/compensation unit 32 performs a motion
5 prediction in each prediction mode, to detect motion
vectors. The operation then moves on to step ST53.
[0102]
In step ST53, the motion prediction/compensation
unit 32 performs a motion vector encoding operation. The
10 motion prediction/compensation unit 32 generates motion
vector information indicating difference motion vectors
that are the differences between the motion vectors
detected in the respective prediction modes and the
motion vectors of encoded blocks. The motion
15 prediction/compensation unit 32 performs lossless
encoding on the generated motion vector information, and
then moves on to step ST54.
[0103]
In step ST54, the motion prediction/compensation
20 unit 32 determines motion vectors and a reference image
for each prediction mode. The motion
prediction/compensation unit 32 determines the motion
vectors in each mode for each motion compensation block
size, and the reference image used in the detection of
25 the motion vectors. The operation then moves on to step
ST55. The motion prediction/compensation unit 32 also
excludes each prediction mode having a larger total
number of motion vectors of two successive macroblocks
than the upper limit value, from the candidates for the
30 optimum inter prediction mode.
[0104]
39
SP306420WOOO
In step ST55, the motion prediction/compensation
unit 32 calculates a cost function value in each
prediction mode. Using the above mentioned equation (15)
or (16), the motion prediction/compensation unit 32
5 calculates the cost function values. The cost function
value calculations in the inter prediction modes involve
the evaluations of cost function values in the skipped
macroblock mode or the direct mode specified in H.264/AVC.
10
[0105]
In step ST56, the motion prediction/compensation
unit 32 determines the optimum inter prediction mode.
Based on the cost function values calculated in step ST55,
the motion prediction/compensation unit 32 selects the
one inter prediction mode with the smallest cost function
15 value among the calculated cost function values, and
determines the selected inter prediction mode to be the
optimum inter prediction mode.
[0106]
As described above, with the image processing
20 device illustrated in Fig. 1 and the image processing
method shown in Fig. 11, the upper limit value of the
number of motion vectors is set in accordance with
macroblock size. Accordingly, even though the number of
motion vectors is limited so as not to exceed the upper
25 limit value, real-time operations can be guaranteed in
view of memory bandwidth, with the number of motion
vectors not being restricted to a smaller number than
necessary. Also, the number of motion vectors is not
limited to a smaller number than necessary, so that image
30 quality degradation can be prevented. That is, image
processing suitable for extended macroblocks can be
40
SP306420WOOO
performed.
[0107]
[3. Structure of Another Image Encoding Device]
The following is a description of the structure of
5 another image encoding device in which the number of
times control is performed to restrict the number of
motion vectors to a smaller number than the upper limit
value is monitored for each frame, so that encoding
operations can be performed more efficiently.
10 [0108]
Fig. 16 illustrates the structure of another image
encoding device. Like the image encoding device 10, the
image encoding device lOa includes an analog/digital
conversion unit (an A/D conversion unit) 11, a picture
15 rearrangement buffer 12, a subtraction unit 13, an
orthogonal transform unit 14, a quantization unit 15, a
lossless encoding unit 16, an accumulation buffer 17, and
a rate control unit 18. Like the image encoding device
10, the image encoding device lOa further includes an
20 inverse quantization unit 21, an inverse orthogonal
transform unit 22, an addition unit 23, a deblocking
filter .24, a frame memory 25, an intra prediction unit 31,
a motion prediction/compensation unit 32a, a predicted
image/optimum mode select unit 33, and a motion vector
25 number limit setting unit 35. Further, the image
encoding device lOa includes a scene change detection
unit 29. In the following description, only the aspects
that differ from those of the image encoding device 10
will be described.
30 [0109]
The scene change detection unit 29 detects scene
41
• SP306420WOOO
changes by using input image data output from the picture
rearrangement buffer 12, and outputs the scene change
detection results to the motion prediction/compensation
unit 32a.
5 [0110]
10
15
Fig. 17 shows the structure of the other motion
prediction/compensation unit and the motion vector number
limit setting unit.
[0111]
The motion prediction/compensation unit 32a
includes a motion search unit 321a, a mode determination
unit 322a, a motion compensation processing unit 323, and
a motion vector buffer 324.
[0112]
Rearranged image data supplied from the picture
rearrangement buffer 12, and reference image data read
from the frame memory 25 are supplied to the motion
search unit 321a. Also, the scene change detection
results are supplied from the scene change detection unit
20 29 to the motion search unit 321a, and the number of
times control is performed to restrict the number of
motion vectors to a smaller number than the upper limit
value is supplied from the mode determination unit 322a
to the motion search unit 321a.
25 [0113]
The motion search unit 321a performs motion search
operations in all candidate inter prediction modes, and
outputs motion vector information indicating the motion
vectors detected through the motion search operations to
30 the mode determination unit 322a. Based on the detected
motion vectors, the motion search unit 321a performs
42
• SP306420WOOO
motion compensation on the reference image data, to
generate and output predicted image data to the mode
determination unit 322a. The motion search unit 321a
further outputs size identification information for
5 identifying the block size of a block on which an upper
limit value is set, to the motion vector number limit
setting unit 35. In a case where the number of times
control is performed to restrict the number of motion
vectors to a smaller number than the upper limit value is
10 larger than a predetermined number of times, the motion
search unit 321a makes the macroblocks larger in the next
frame. When the scene change detection results indicate
that a scene change has been detected, the motion search
unit 321a performs an operation to return the macroblock
15 size to a predetermined size.
[0114]
The motion vector information and predicted image
data about all the candidate inter prediction modes are
supplied from the motion search unit 321a to the mode
20 determination unit 322a. Motion vector information that
is generated at an encoded adjacent block is also
supplied from the motion vector buffer 324 to the mode
determination unit 322a. Further, rearranged image data
supplied from the picture rearrangement buffer 12 is also
25 supplied to the mode determination unit 322a to calculate
cost functions.
[0115]
The mode determination unit 322a calculates a cost
function value in each of the candidates for the optimum
30 inter prediction mode by using the image data supplied
from the picture rearrangement buffer 12 and the
43
• SP306420WOOO
predicted image data supplied from the motion search unit
321a. The mode determination unit 322a selects the mode
with the smallest cost function value as the optimum
inter prediction mode for the macroblock being encoded.
5 [0116]
Using the motion vector information about the
adjacent macroblock supplied from the motion vector
buffer 324, the mode determination unit 322a calculates
the total number of motion vectors of the two successive
10 macroblocks. In a case where the total number of motion
vectors in the mode with the smallest cost function value
is larger than the upper limit value, the mode
determination unit 322a performs control to determine the
optimum inter prediction mode to be the mode having the
15 smallest cost function value with the total value not
exceeding the upper limit value, such as control to
exclude, from the candidates for the optimum inter
prediction mode, each prediction mode having a larger
total number of motion vectors of two successive
20 macroblocks than the upper limit value. The number of
times control is performed (the number of macroblocks) in
each frame is then output to the motion search unit 321a
for each frame.
25
[0117]
Further, the mode determination unit 322a outputs
the prediction mode information, the motion vector
information, and the cost function value of the mode
determined to be the optimum mode, to the motion
compensation processing unit 323. Based on the motion
30 vector information supplied from the motion vector buffer
324, the mode determination unit 322a also calculates
44
• SP306420WOOO
predicted motion vectors. The mode determination unit
322a calculates difference motion vectors that are the
differences between the calculated predicted motion
vectors and the motion vectors detected by the motion
5 search unit 321a, and outputs difference motion vector
information indicating the difference motion vectors
contained in the motion vector information to the motion
compensation processing unit 323.
10
[0118]
As described above, in a case where the number of
times control is performed to restri~t the number of
motion vectors to a smaller number than the upper limit
value is monitored in each frame, and the number of times
control is performed becomes larger than a predetermined
15 number of times, the macroblocks are made larger in the
next frame. By doing so, the upper limit value becomes
larger, and the limit on the number of motion vectors
becomes less strict. Accordingly, the number of times
control is performed to restrict the number of motion
20 vectors to a smaller number than the upper limit value
can be made smaller, and efficient encoding operations
can be performed. When a scene change is detected, the
macroblocks are returned to the predetermined size.
Accordingly, when image continuity is lost due to a scene
25 change, encoding operations can be performed with the
predetermined macroblock size.
[0119]
[4. Other Operations of an Image Encoding Device]
The following is a description of other operations
30 of an image encoding device that has an operation mode to
allow a larger number of motion vectors than the upper
45
.. SP306420WOOO
5
limit value. It should be noted that the structure of
the image encoding device in this case is the same as
that illustrated in Fig. 1 or 10.
[0120]
The limit of the number of motion vectors is set so
as to guarantee real-time operations in view of memory
bandwidth as described above. Therefore, an operation
mode not required to guarantee real-time operations may
be set to allow a larger number of motion vectors than
10 the upper limit value. In that case, the number of
motion vectors is not limited to the upper limit value,
and accordingly, more suitably predicted images can be
generated.
[0121]
15 In a case where such an operation mode is used,
information indicating that real-time decoding operations
are not guaranteed, or determination information (such as
a flag) indicating that the number of motion vectors is
larger than the upper limit value, is provided in a
20 sequence parameter set or the like in the compressed
image information. As such a flag is provided, the image
decoding device that decodes the compressed image
information can easily determine that the number of
motion vectors is larger than the upper limit value by
25 detecting the flag. Accordingly, it is possible to
readily notify users or the like that real-time decoding
operations are not guaranteed.
30
[0122]
[5. Structure of an Image Decoding Device]
compressed image information generated by encoding
input images is supplied to an image decoding device via
46
• SP306420WOOO
a predetermined transmission path or a recording medium
or the like, and is decoded therein.
[0123]
Fig. 18 shows the structure of an image decoding
5 device that is an image processing device decoding the
compressed image information. The image decoding device
50 includes an accumulation buffer 51, a lossless
decoding unit 52, an inverse quantization unit 53, an
inverse orthogonal transform unit 54, an addition unit 55,
10 a deblocking filter 56, a picture rearrangement buffer 57,
and a digital/analog conversion unit (a D/A conversion
unit) 58. The image decoding device 50 further includes
a frame memory 61, an intra prediction unit 62, a motion
compensation unit 63, a selector 64, and a motion vector
15 number limit setting unit 65.
[0124]
The accumulation buffer 51 stores transmitted
compressed image information. The lossless decoding unit
52 decodes the compressed image information supplied from
20 the accumulation buffer 51 by a technique compatible with
the encoding technique used by the lossless encoding unit
16 of Fig. 1.
[0125]
The lossless decoding unit 52 outputs the
25 prediction mode information obtained by decoding the
compressed image information to the intra prediction unit
62 and the motion compensation unit 63.
[0126]
The inverse quantization unit 53 inversely
30 quantizes the quantized data decoded by the lossless
decoding unit 52, using a technique compatible with the
47
• SP306420WOOO
quantization technique used by the quantization unit 15
of Fig. 1. The inverse orthogonal transform unit 54
performs an inverse orthogonal transform on the output
from the inverse quantization unit 53 by a technique
5 compatible with the orthogonal transform technique used
by the orthogonal transform unit 14 of Fig. 1, and
outputs the result to the addition unit 55.
[0127]
The addition unit 55 generates decoded image data
10 by adding the data subjected to the inverse orthogonal
transform to predicted image data supplied from the
selector 64, and outputs the decoded image data to the
deblocking filter 56 and the intra prediction unit 62.
[0128]
15 The deblocking filter 56 performs deblocking
filtering on the decoded image data supplied from the
addition unit 55, and removes block distortions. The
resultant data is supplied to and stored in the frame
memory 61, and is also output to the picture
20 rearrangement buffer 57.
[0129]
The picture rearrangement buffer 57 performs image
rearrangement. Specifically, the frame order rearranged
in the order of encoding performed by the picture
25 rearrangement buffer 12 of Fig. 1 is rearranged in the
original display order, and is output to the D/A
conversion unit 58.
[0130]
The D/A conversion unit 58 performs a D/A
30 conversion on the image data supplied from the picture
rearrangement buffer 57, and outputs the converted image
48
• SP306420WOOO
data to a display (not shown) to display the images.
[0131]
The frame memory 61 stores the decoded image data
that has been subjected to the filtering operation and is
5 supplied from the deblocking filter 24.
[0132]
Based on the prediction mode information supplied
from the lossless decoding unit 52, the intra prediction
unit 62 generates predicted images from the decoded image
10 data supplied from the addition unit 55, and outputs the
generated predicted image data to the selector 64.
[0133]
Based on the prediction mode information and motion
vectors supplied from the lossless decoding unit 52, the
15 motion compensation unit 63 performs motion compensation
by reading the image data of reference images from the
frame memory 61, and generates and outputs predicted
image data to the selector 64. The motion compensation
unit 63 outputs the macroblock size indicated by the
20 prediction mode information to the motion vector number
limit setting unit 65. Further, the motion compensation
unit 63 determines whether the total number of motion
vectors of two successive macroblocks is larger than the
upper limit value sent from the motion vector number
25 limit setting unit 65, and outputs the determination
result to a system controller, for example.
[0134]
The selector 64 outputs the predicted image data
generated by the intra prediction unit 62 to the addition
30 unit 55. The selector 64 also outputs the predicted
image data generated by the motion compensation unit 63
49
•
to the addition unit 55.
[0135]
SP306420WOOO
The motion vector number limit setting unit 65 sets
the upper limit-value of the number of motion vectors in
5 accordance with the macroblock size sent from the motion
compensation unit 63, and notifies the motion
compensation unit 63 of the set upper limit value. If
the macroblock is larger than a 16 x 16 pixel macroblock,
the motion vector number limit setting unit 65 sets a
10 larger upper limit value than the upper limit value in a
16 x 16 pixel macroblock, to allow a check to be made to
determine whether real-time operations can be guaranteed
in view of memory bandwidth, with the number of motion
vectors not being restricted to a smaller number than
15 necessary. For example, in a case where the upper limit
value of the number of motion vectors is p when the
macroblock size is 16 x 16 pixels, and the macroblock
size is (16 x m) x (16 x n) pixels, the motion vector
number limit setting unit 65 sets the upper limit value
20 at (m x n x p), which is in predetermined ratio to the
macroblock size. Accordingly, a check can be made to
determine whether real-time operations can be guaranteed
in view of memory bandwidth, with the number of motion
vectors not being restricted to a smaller number than
25 necessary.
[0136]
Fig. 19 is a diagram showing the structure of the
motion compensation unit and the motion vector number
limit setting unit.
30 [0137]
The motion compensation unit 63 includes a
50
•
5
SP306420WOOO
macroblock size buffer 631, a motion vector generation
unit 632, a mode buffer 633, and a motion compensation
processing unit 634.
[0138]
The macroblock size buffer 631 stores size
identification information supplied from the lossless
decoding unit 52. The macroblock size buffer 631 also
outputs the stored size identification information to the
motion vector number limit setting unit 65.
10 [0139]
The motion vector number limit setting unit 65
determines the upper limit value of the number of motion
vectors in accordance with the macroblock size, and
notifies the motion vector generation unit 632 of the
15 upper limit value.
[0140]
To reconfigure motion vector information, the
motion vector generation unit 632 includes a buffer that
stores the motion vector information about adjacent
20 blocks in time and space. Using the motion vector
information about the adjacent blocks and the difference
motion vector information indicated by the prediction
mode information supplied from the lossless decoding unit,
the motion vector generation unit 632 regenerates the
25 motion vector information about the block. The motion
vector generation unit 632 also determines whether the
number of reconfigured motion vectors is larger than the
upper limit value determined by the motion vector number
limit setting unit 65, and outputs the determination
30 result to the system controller (not shown) or the like.
[0141]
51
• SP306420WOOO
The mode buffer 633 stores the prediction mode
information that is supplied from the lossless decoding
unit 52 and indicates the prediction mode. The mode
buffer 633 also outputs the stored prediction mode
5 information to the motion compensation processing unit
634.
[0142]
Based on the prediction mode information, the
motion compensation processing unit 634 reads the image
10 data of reference images from the frame memory 61, and
performs motion compensation in accordance with the
motion vector information generated by the motion vector
generation unit 632, to generate predicted image data.
The motion compensation processing unit 634 outputs the
15 generated predicted image data to the selector 64.
[0143]
[6. Operations of the Image Decoding Apparatus]
Referring now to the flowchart in Fig. 20, an image
decoding operation to be performed by the image decoding
20 device 50 is described.
[0144]
In step ST81, the accumulation buffer 51 stores
transmitted compressed image information. In step ST82,
the lossless decoding unit 52 performs a lossless
25 decoding operation. The lossless decoding unit 52
decodes the compressed image information supplied from
the accumulation buffer 51. Specifically, the quantized
data of each picture encoded by the lossless encoding
unit 16 of Fig. 1 is obtained. The lossless decoding
30 unit 52 also performs lossless decoding on the prediction
mode information contained in the compressed image
52
• SP306420WOOO
information. In a case where the obtained prediction
mode information is information about an intra prediction
mode, the prediction mode information is output to the
intra prediction unit 62. In a case where the prediction
5 mode information is information about an inter prediction
mode, on the other hand, the lossless decoding unit 52
outputs the prediction mode information to the motion
compensation unit 63.
10
[0145]
In step ST83, the inverse quantization unit 53
performs an inverse quantization operation. The inverse
quantization unit 53 inversely quantizes the quantized
data decoded by the lossless decoding unit 52, having
characteristics compatible with the characteristics of
15 the quantization unit 15 of Fig. 1.
[0146]
In step ST84, the inverse orthogonal transform unit
54 performs an inverse orthogonal transform operation.
The inverse orthogonal transform unit 54 performs an
20 inverse orthogonal transform on the transform coefficient
data inversely quantized by the inverse quantization unit
53, having the characteristics compatible with the
characteristics of the orthogonal transform unit 14 of
Fig. 1.
25 [0147]
In step ST85, the addition unit 55 generates
decoded image data. The addition unit 55 adds the data
obtained through the inverse orthogonal transform
operation to predicted image data selected in step ST89,
30 which will be described later, and generates the decoded
image data. In this manner, the original images are
53
•
decoded.
[0148]
SP306420WOOO
In step ST86, the deblocking filter 56 performs a
filtering operation. The deblocking filter 56 performs
5 deblocking filtering on the decoded image data output
from the addition unit 55, and removes block distortions
contained in the decoded images.
[0149]
In step ST87, the frame memory 61 performs a
10 decoded image data storing operation.
[0150]
In step ST88, the intra prediction unit 62 and the
motion compensation unit 63 perform predicted image
generating operations. The intra prediction unit 62 and
15 the motion compensation unit 63 each perform a predicted
image generating operation in accordance with the
prediction mode information supplied from the lossless
decoding unit 52.
20
[0151]
Specifically, in a case where prediction mode
information about an intra prediction is supplied from
the lossless decoding unit 52, the intra prediction unit
62 generates predicted image data based on the prediction
mode information. In a case where prediction mode
25 information about an inter prediction is supplied from
the lossless decoding unit 52, on the other hand, the
motion compensation unit 63 performs motion compensation
based on the prediction mode information, to generate
predicted image data.
30 [0152]
In step ST89, the selector 64 selects predicted
54
• SP306420WOOO
image data. Specifically, the selector 64 selects the
predicted image supplied from the intra prediction unit
62 and the predicted image data generated by the motion
compensation unit 63, and supplies the selected predicted
5 image data to the addition unit 55, which adds the
selected predicted image data to the output from the
inverse orthogonal transform unit 54 in step ST85, as
described above.
10
[0153]
In step ST90, the picture rearrangement buffer 57
performs image rearrangement. Specifically, the order of
frames rearranged for encoding by the picture
rearrangement buffer 12 of the image encoding device 10
of Fig. 1 is rearranged in the original display order by
15 the picture rearrangement buffer 57.
[0154]
In step ST91, the 0/A conversion unit 58 performs a
0/A conversion on the image data supplied from the
picture rearrangement buffer 57. The images are output
20 to the display (not shown), and are displayed.
[0155]
Referring now to the flowchart in Fig. 21, the
predicted image generating operation in step ST88 of Fig.
20 is described.
25 [ 0156]
In step ST101, the lossless decoding unit 52
determines whether the current block is intra-encoded.
If the prediction mode information obtained by performing
lossless decoding is prediction mode information about an
30 intra prediction, the lossless decoding unit 52 supplies
the prediction mode information to the intra prediction
55
• SP306420WOOO
unit 62, and the operation moves on to step ST102. In a
case where the prediction mode information is prediction
mode information about an inter prediction mode, on the
other hand, the lossless decoding unit 52 supplies the
5 prediction mode information to the motion compensation
unit 63, and the operation moves on to step ST103.
[0157]
In step ST102, the intra prediction unit 62
performs a predicted image generating operation. Using
10 decoded image data supplied from the addition .unit 55 and
the prediction mode information, the intra prediction
unit 62 performs an intra prediction, to generate
predicted image data.
15
[0158]
In step ST103, the motion compensation unit 63
performs a predicted image generating operation. Based
on the prediction mode information and motion vectors
supplied from the lossless decoding unit 52, the motion
compensation unit 63 performs motion compensation on a
20 reference image read from the frame memory 61, and
generates predicted image data.
[0159]
Fig. 22 is a flowchart showing the inter-predicted
image generating operation of step ST103. In step ST111,
25 the motion vector number limit setting unit 65 performs
the same operation as that shown in Fig. 15, to set the
upper limit value of the number of motion vectors. The
motion vector number limit setting unit 65 sets the upper
limit value of the number of motion vectors in accordance
30 with the macroblock size. If the macroblock is larger
than a 16 x 16 pixel macroblock, the motion vector number
56
• SP306420WOOO
limit setting unit 65 sets a larger upper limit value
than the upper limit value in a 16 x 16 pixel macroblock,
to allow a check to be made to determine whether realtime
operations can be guaranteed in view of memory
5 bandwidth, with the number of motion vectors not being
restricted to a smaller number than necessary. For
example, in a case where the upper limit value of the
number of motion vectors is p when the macroblock size is
16 x 16 pixels, and the macroblock size is (16 x m) x (16
10 x n) pixels, the motion vector number limit setting unit
65 sets the upper limit value at (m x n x p). The
operation then moves on to step ST112.
[0160]
In step ST112, the motion compensation unit 63
15 obtains prediction mode information. To generate
predicted image data, the motion compensation unit 63
obtains the prediction mode information from the lossless
decoding unit 52, and moves on to step ST113.
20
[0161]
In step ST113, the motion compensation unit 63
reconfigures motion vector information. The motion
compensation unit 63 generates predicted motion vector
information from the motion vectors of a decoded adjacent
block, for example, and reconfigures motion vector
25 information based on the predicted motion vector
30
information and the difference motion vector information
indicated by the prediction mode information. The
operation then moves on to step ST114.
[0162]
In step ST114, the motion compensation unit 63
monitors the number of motion vectors. The motion
57
• SP306420WOOO
compensation unit 63 monitors whether the total number of
motion vectors of two adjacent macroblocks is larger than
the upper limit value determined in step ST111. In a
case where the number of motion vectors is larger than
5 the upper limit value, the motion compensation unit 63
notifies the system controller or the like to that effect,
and moves on to step ST115.
[0163]
In step ST115, the motion compensation unit 63
10 generates predicted image data. Based on the prediction
mode information obtained in step ST112 and the motion
vector information reconfigured in step ST113, the motion
compensation unit 63 performs motion compensation by
reading the reference image data from the frame memory 61,
15 and generates and outputs predicted image data to the
selector 64.
[0164]
As described above, with the image processing
device illustrated in Fig. 18 and the image processing
20 method shown in Fig. 20, the upper limit value of the
number of motion vectors is set in accordance with
macroblock size, and monitoring is performed to determine
whether the number of motion vectors is larger than the
upper limit value. The result of the monitoring is then
25 output. Accordingly, based on the monitoring result, a
check can be easily made to determine whether real-time
operations can be guaranteed. In a case where the number
of motion vectors is larger than the upper limit value,
and real-time operations are not guaranteed, the user can
30 be notified to that effect through display or sound or
the like.
58
• SP306420WOOO
[0165]
In a case where the compressed image information
contains determination information indicating that the
number of motion vectors is larger than the upper limit
5 value; a determination result indicating that the number
of motion vectors is larger than the upper limit value is
output based on the determination information.
Accordingly, when real-time operations are not guaranteed,
the user can be notified to that effect through display
10 or sound or the like.
[0166]
The macroblock size is not limited to that
described in the above embodiments, and may be some other
size. At present, to achieve higher encoding efficiency
15 than that of H.264/AVC, an image encoding technique
called HEVC(High Efficiency Video Coding) is being
developed as a standard by JCTVC (Joint Collaboration
Team- Video Coding), which is a joint standardization
organization of ITU-T and ISO/IEC.
20 [0167]
In HEVC, the size of a coding unit (CU) can be
dynamically designated in each sequence. The coding
units in HEVC are also called a coding tree block in
which the coding unit having the largest size is the
25 largest coding unit (LCU), and the coding unit having the
smallest size is the smallest coding unit (SCU). The
sizes of the LCU and the SCU are designated in a sequence
parameter set that is part of compressed image
information. In this manner, a range of usable coding
30 unit sizes can be defined. Further, the value of
split_flag is designated, to specify the size of each

CLAIMS
1. An image processing device comprising:
a setting unit configured to set an upper limit
5 value of the number of motion vectors in accordance with
a block size of a block that is a unit for encoding to be
performed when image data is encoded;
a motion prediction/compensation unit configured to
detect a prediction mode when the number of motion
10 vectors of the block is equal to or smaller than the
upper limit value set by the setting unit, and performs
motion compensation on a reference image by using the
motion vectors of the detected prediction mode, to
generate a predicted image; and
15 an encoding unit configured to encode the image
data to generate an encoded stream by using the predicted
image generated by the motion prediction/compensation
unit.
20 2. The image processing device according to claim 1,
wherein, when the block size of the block is larger than
a predetermined size, the setting unit sets the upper
limit value at a larger value than an upper limit value
in a block of the predetermined size.
25
3. The image processing device according to claim 2,
wherein, when the upper limit value of the number of
motion vectors in a block of a predetermined size (N) is
p, and a larger block size than the predetermined size is
30 (N X m) X (N X n) pixels, the setting unit sets the upper
limit value at (p x m x n ) .
85
^ SP306420WO00
4. The image processing device according to claim 3,
wherein the predetermined size is 16 x 16 pixels.
5 5. The image processing device according to claim 2,
wherein a block having a larger size than the
predetermined size is the largest unit for encoding with
the largest size among units for encoding in a
hierarchical structure.
10
6. The image processing device according to claim 2,
wherein, when detecting the prediction mode, the motion
prediction/compensation unit excludes each prediction
mode having a larger number of motion vectors than the
15 upper limit value from candidates for the prediction mode
to be detected.
7. The image processing device according to claim 2,
wherein the motion prediction/compensation unit monitors
20 the number of times control is performed to restrict the
number of motion vectors to a smaller number than the
upper limit value in each frame, and, when the number of
times control is performed exceeds a predetermined number
of times, increases a block size of each block in the
25 next frame.
8. The image processing device according to claim 7,
further comprising:
a scene change detection unit configured to detect
30 a scene change in the image data,
wherein, when a scene change is detected by the
86
^ SP306420WO00
scene change detection unit, the motion
prediction/compensation unit sets the block size of each
block at a predetermined size set in advance.
5 9. The image processing device according to claim 1,
wherein, in an operation mode to generate a predicted
image by detecting an optimum prediction mode, regardless
of the number of motion vectors, and performing motion
compensation on a reference image with motion vectors of
10 the detected optimum prediction mode, when the nurnber of
motion vectors in a prediction mode exceeds the upper
limit value, the encoding unit transmits the encoded
stream and determination information indicating that the
number of motion vectors is larger than the upper limit
15 value.
10. The image processing device according to claim 1,
further comprising:
a generation unit configured to generate size
20 identification information for identifying a block size
of each block, having an upper limit value set thereon by
the setting unit,
wherein the encoding unit transmits the encoded
stream and the size identification information.
25
11. An image processing method comprising:
setting an upper limit value of the number of
motion vectors in accordance with a block size of a block
that is a unit for encoding to be performed when image
30 data is encoded;
generating a predicted image by detecting a
87
^ SP306420WO00
prediction mode when the number of motion vectors of the
block is equal to or smaller than the set upper limit
value, and performing motion compensation on a reference
image with motion vectors of the detected prediction
5 mode; and
generating an encoded stream by encoding the image
data with the generated predicted image.
12. An image processing device comprising:
10 a setting unit configured to set an upper limit
value of the number of motion vectors in accordance with
a block size of a block that is a unit for decoding to be
performed when an encoded stream generated by encoding
image data is decoded;
15 a motion compensation unit configured to detect a
prediction mode when the number of motion vectors of the
block is equal to or smaller than the upper limit value
set by the setting unit, and perform motion compensation
on a reference image by using motion vectors of the
20 detected prediction mode, to generate a predicted image;
and
a decoding unit configured to decode the encoded
stream by using the predicted image generated by the
motion compensation unit.
25
13. The image processing device according to claim 12,
wherein, when the block size of the block is larger than
a predetermined size, the setting unit sets the upper
limit value at a larger value than an upper limit value
30 in a block of the predetermined size.
88
^ SP306420WO00
14. The image processing device according to claim 13,
wherein, when the upper limit value of the number of
motion vectors in a block of a predetermined size (N) is
p, and a larger block size than the predetermined size is
5 (N X m) X (N X n) pixels, the setting unit sets the upper
limit value at (p x m x n ).
15. The image processing device according to claim 14,
wherein the predetermined size is 16 x 16 pixels.
10
16. The image processing device according to claim 13,
wherein a block having a larger size than the
predetermined size is the largest unit for encoding with
the largest size among units for encoding in a
15 hierarchical structure.
17. The image processing device according to claim 13,
wherein, when detecting the prediction mode, the motion
compensation unit excludes each prediction mode having a
20 larger nijmber of motion vectors than the upper limit
value from candidates for the prediction mode to be
detected.
18. The image processing device according to claim 13,
25 wherein the motion compensation unit monitors the number
of times control is performed to restrict the number of
motion vectors to a smaller number than the upper limit
value in each frame, and, when the number of times
control is performed exceeds a predetermined nijmber of
30 times, increases a block size of each block in the next
frame.
89
^ SP306420WO00
19. The image processing device according to claim 18,.
further comprising:
a scene change detection unit configured to detect
5 a scene change in the image data,
wherein, when a scene change is detected by the
scene change detection unit, the motion compensation unit
sets the block size of each block at a predetermined size
set in advance.
10
20. The image processing device according to claim 12,
wherein, in an operation mode to .generate a predicted
image by detecting an optimum prediction mode, regardless
of the number of motion vectors, and perforxaing motion
15 compensation on a reference image with motion vectors of
the detected optimum prediction mode, the decoding unit
obtains-determination information from the encoded stream,
the determination information indicating that the number
of motion vectors is larger than the upper limit value.
20 ' .
21. The image processing device according to claim 12, ^
wherein the decoding unit obtains size identification
information from the encoded stream, the size
identification information identifying a block size of
25 each block having an upper limit value set thereon.
22. An image processing method comprising the steps of:
setting an upper limit value of the number of
motion vectors in accordance with a block size of a block
30 that is a unit for decoding to be performed when an
encoded stream generated by encoding image data is
" •" :.....;.^llSiMMiM' • - 9 0 • .
fj^ SP306420WO00 "
decoded;
generating a predicted image by detecting a
prediction mode when the number of motion vectors of the
block is equal to or smaller than the upper limit value
5 set by the setting unit, and performing motion
compensation on a reference image with motion vectors of
the detected prediction mode; and
decoding the encoded stream with the generated
predicted image.