DESCRIPTION
IMAGE PROCESSING DEVICE AND IMAGE PROCESSING METHOD
TECHNICAL FIELD
5 [0001]
The present invention relates to an image
processing device and an image processing method.
Particularly, the present invention aims to improve the
encoding efficiency in multi-view image encoding.
10
BACKGROUND ART
[0002]
In recent years, apparatuses that handle image
information as digital information and achieve high-
15 efficiency information transmission and accumulation in
doing so, or apparatuses compliant with a standard such
as MPEG for compression through orthogonal transforms
like discrete cosine transforms and motion compensations,
have been spreading among broadcast stations and general
20 households.
[0003]
Particularly, MPEG2 (ISO/IEC 13818-2) is defined as
a general-purpose image encoding technique, and is
currently used for a wide range of applications for
25 professionals and general consumers. Further, H.264 and
MPEG-4 Part 10 (hereinafter referred to as "H.264/AVC
(Advanced Video Coding)"), which can achieve a higher
encoding efficiency than encoding techniques such as
MPEG2, have become standards, though requiring a larger
30 amount of calculation for encoding and decoding.
[0004]
1
SP301579WO00
By such image encoding techniques, the amount of
information is made smaller by reducing redundancy in the
temporal and spatial directions. For an I-picture on
which in-picture predictive encoding is performed so as
5 to reduce spatial redundancy, a predicted image is
generated by using the correlation between pixels, for
example. For a P-picture on which in-picture predictive
encoding is performed so as to reduce temporal redundancy,
motion vectors are detected on a block basis by referring
10 to a previous image, and a predicted image is generated
by using the detected motion vectors. As for a B-picture,
motion vectors are detected on a block basis by referring
to previous and subsequent pictures, and a predicted
image is generated by using the detected motion vectors.
15 In the B-picture, the first reference picture is called a
reference picture of an LO prediction, and the second
reference picture is called a reference picture of an LI
prediction.
[0005]
20 According to H.264/AVC, reference pictures can be
selected from already encoded pictures. The selected
reference pictures are managed by reference indexes. A
reference index is used as the information indicating to
which picture is referred when motion vectors are
25 detected, and the reference index is encoded together
with the information indicating the detected motion
vectors.
[0006]
A reference index is set to a value of 0 or greater.
30 The smaller the value of the reference index, the smaller
the amount of information after encoding (the coding
2
SP301579WO00
amount). Further, reference indexes can be arbitrarily
assigned to reference pictures. Therefore, a reference
index with a smaller number is assigned to a reference
picture with a large number of motion vectors to be
5 referred to. By doing so, the coding amount at the time
of reference index encoding is reduced, and the encoding
efficiency can be increased.
[0007]
Patent Document 1 discloses a technique by which a
10 reference index with a small value is assigned to a
reference picture that is close to the picture being
encoded in terms of time, when field encoding is
performed on a 2D image of an interlaced scan type.
15 CITATION LIST
PATENT DOCUMENT
[0008]
Patent Document 1: Japanese Patent Application Laid-Open
No. 2010-63092
20
SUMMARY OF THE INVENTION
PROBLEMS TO BE SOLVED BY THE INVENTION
[0009]
Meanwhile, in frame sequential (FS)-AVC or
25 multiview video coding (MVC), not only a temporal
prediction that uses the correlation between images in
the temporal direction, but also a parallax prediction
that uses the correlation between images of different
viewpoints is performed.
30 [0010]
Fig. 1 shows a conventional reference index
3
4
SP301579WO00
assignment method, or a method of assigning reference
indexes when moving image data of two viewpoints are
encoded by MVC, for example. It should be noted that
CamO represents the image data of a left-eye image, and
5 Caml represents the image data of a right-eye image. The
image data of Caml is the image data of a dependent view
to be encoded by using the image data of CamO as the
image data of a reference picture. The image data of
CamO to be referred to when the image data of the
10 dependent view is encoded is the image data of a base
view.
[0011]
The P-pictures of Caml to be referred to in a
temporal prediction as indicated by the solid arrows, and
15 the I-picture and P-pictures of CamO to be referred to in
a parallax prediction as indicated by dotted arrows serve
as the reference pictures for the P-pictures in the image
data of Caml. The reference pictures are managed by the
numbers each shown as reference index ref_idx, and a
20 value of 0 or greater is assigned to each of the
reference pictures. For example, a reference index
ref_idx = 0 is assigned to a temporal prediction
reference picture, and a reference index ref_idx = 1 is
assigned to a parallax prediction reference picture.
25 [0012]
In a case where the correlation between images of
different viewpoints becomes dramatically lower, only the
same effects as those in a case where a single image is
referred to are substantially achieved, even if reference
30 indexes are assigned to reference pictures as shown in
Fig. 1 so that more than one image can be referred to.
4
SP301579WO00
That is, when the correlation between a left-eye image
and a right-eye image is extremely low, a prediction
using images of different viewpoints is inefficient.
Therefore, parallax predictions using the reference index
5 ref_idx = 1 are ineffective. As a result, only the
temporal predictions using the reference index ref_idx =
0 are performed, and substantially the same effects as
those in a case where a single image is referred to are
obtained.
10 [0013]
In view of this, the present invention aims to
provide an image processing device and an image
processing method that can increase the encoding
efficiency in multi-view image encoding.
15
SOLUTIONS TO PROBLEMS
[0014]
A first aspect of the present invention is an image
processing device that includes: a feature amount
20 generation unit that generates a feature amount
indicating the correlation between images of different
viewpoints; and a reference index assignment unit that
re-assigns a reference index assigned to a reference
picture of a parallax prediction using the correlation
25 between the images of different viewpoints, to a
reference picture of a temporal prediction using the
correlation between images in a temporal direction, when
the correlation between the images of different
viewpoints is determined to be lower than a predetermined
30 threshold value based on the feature amount generated by
the feature amount generation unit.
5
SP301579WO00
[0015]
In the present invention, a feature amount
indicating the correlation between images of different
viewpoints is generated by the feature amount generation
5 unit. For example, in an operation to encode the first
picture in a GOP, at least one of the total sum. of the
differences between blocks being encoded and a reference
block in the image when parallax vectors are detected,
the proportion of intra macroblocks in the image, and the
10 image complexity ratio between the picture being encoded
and a reference picture of a different viewpoint, is
calculated as the feature amount. Based on the feature
amount, reference indexes are assigned to a reference
picture of a parallax prediction using the correlation
15 between images of different viewpoints, and to a
reference picture of a temporal prediction using the
correlation between images in the temporal direction. In
a case where the correlation is determined to be lower
than a predetermined threshold value, for example, the
20 reference index assignment is changed, and the reference
index assigned to the reference picture of the parallax
prediction is re-assigned to the reference picture of the
temporal prediction. Also, in a case where the
correlation is determined to be lower than the
25 predetermined threshold value, a change is made to the
GOP structure, to turn a non-reference picture that is
closer in the temporal direction, into a reference
picture.
[0016]
30 A second aspect of the present invention is an
image processing method that includes: a feature amount
6
SP301579WO00
generation step of generating a feature amount indicating
the correlation between images of different viewpoints;
and a reference index assignment step of re-assigning a
reference index assigned to a reference picture of a
5 parallax prediction using the correlation between the
images of different viewpoints, to a reference picture of
a temporal prediction using the correlation between
images in a temporal direction, when the correlation
between the images of different viewpoints is determined
10 to be lower than a predetermined threshold value based on
the feature amount generated in the feature amount
generation step.
EFFECTS OF THE INVENTION
15 [0017]
According to the present invention, a feature
amount indicating the correlation between images of
different viewpoints is generated. When the correlation
is determined to be lower than a predetermined threshold
20 value based on the feature amount, the reference index
assigned to a reference picture of a parallax prediction
using the correlation between the images of different
viewpoints is re-assigned to a reference picture of a
temporal prediction using the correlation between images
2 5 in the temporal direction. Accordingly, the encoding
efficiency in a case where the correlation between images
of different viewpoints is low can be increased in multiview
image encoding.
30 BRIEF DESCRIPTION OF DRAWINGS
[0018]
7
SP301579WO00
Fig. 1 is a diagram for explaining a conventional
reference index assignment method;
Fig. 2 is a diagram showing an example structure of
an encoding system;
5 Fig. 3 is a diagram showing the structure of an
image processing device;
Fig. 4 is a flowchart showing an operation of an
image processing device;
Fig. 5 is a diagram illustrating a reference index
10 assignment method implemented where the correlation is
low;
Figs. 6(A) and 6(B) are diagrams illustrating a
reference index assignment method implemented in a case
where B-pictures are contained;
15 Figs. 7(A), 7(B), and 7(C) are diagrams for
explaining an operation performed in a case where a
change is made to a GOP structure; and
Fig. 8 is a diagram showing the structure of a
computer device.
20
MODES FOR CARRYING OUT THE INVENTION
[0019]
The following is a description of embodiments for
carrying out the invention. Explanation will be made in
25 the following order.
1. Example structure of an encoding system
2. Example structure of an image processing device
3. Reference index assigning operation
4. Structure used in a case where image encoding is
30 performed through a software operation
[0020]
8
SP301579WO00
<1. Example structure of an encoding system>
Fig. 2 is a diagram showing an example structure of
an encoding system to which the present invention is
applied. The encoding system 10 includes a left-
5 viewpoint image generating device llL, a right-viewpoint
image generating device IIR, and a multi-view encoding
device 20.
[0021]
The left-viewpoint image generating device IIL is
10 an imaging device or an image data generating device that
generates image data of a left-eye image. The rightviewpoint
image generating device llR is an imaging
device or an image data generating device that generates
image data of a right-eye image. The left-viewpoint
15 image generating device IIL and the right-viewpoint image
generating device IIR operate in synchronization with
each other.
[0022]
The image data of the left-eye image generated by
20 the left-viewpoint image generating device IIL and the
image data of the right-eye image generated by the rightviewpoint
image generating device llR are input to the
multi-view encoding device 20. The multi-view encoding
device 20 encodes the image data of the left-eye image
25 and encodes the image data of the right-eye image,
multiplexes the resultant encoded data, and outputs the
resultant data as a bit stream.
[0023]
The multi-view encoding device 20 includes an image
30 processing device that encodes the image data of the
left-eye image input from the left-viewpoint image
9
SP301579WO00
generating device IIL as image data of a base view, for
example. The multi-view encoding device 20 also includes
an image processing device of the present invention that
encodes the image data of the right-eye image input from
5 the right-viewpoint image generating device IIR as image
data of a dependent view, for example. The image data of
the base view is used in temporal predictions that do not
use images of other viewpoints as reference pictures, and
the image data of the dependent view is used in temporal
10 predictions and parallax predictions that use the image
of the base view as a reference picture.
[0024]
<2. Example structure of an image processing
device>
15 Next, the image processing device of the present
invention is described. In the image processing device
of the present invention, the image data of a left-eye
image and the image data of a right-eye image are
independent of each other. The image processing device
20 that encodes the image data of a dependent view obtains
the image data of a reference picture to be used in a
parallax prediction and the like, from the image
processing device that encodes the image data of a base
view.
25 [0025]
The image processing device that encodes the image
data of a dependent view generates a feature amount that
depends on the correlation between pictures of different
viewpoints, or between an image of a dependent view and
30 an image of a base view to be used as a reference picture.
Further, based on the generated feature amount, reference
10
SP301579WO00
indexes are assigned to the reference picture of a
parallax prediction that uses the correlation between
images of different viewpoints and to the reference
picture of a temporal prediction that uses the
5 correlation between images in the temporal direction.
[0026]
Fig. 3 shows the structure of an image encoding
device 20dv that is an image processing device that
encodes image data of a dependent view. The image
10 encoding device 20dv includes an analog/digital
conversion unit (A/D conversion unit) 21, a picture
rearrangement buffer 22, a subtraction unit 23, an
orthogonal transform unit 24, a quantization unit 25, a
lossless encoding unit 26, an accumulation buffer 27, and
15 a rate control unit 28. The image encoding device 20dv
also includes an inverse quantization unit 31, an inverse
orthogonal transform unit 32, an addition unit 33, a
deblocking filter 34, and a frame memory 35. Further,
the image encoding device 20dv includes a reference index
20 assignment unit 45, an intra prediction unit 51, a
motion/parallax prediction/compensation unit 52, and a
predicted image/optimum mode select unit 53.
[0027]
The A/D conversion unit 21 converts analog image
25 signals into digital image data, and outputs the image
data to the picture rearrangement buffer 22.
[0028]
The picture rearrangement buffer 22 rearranges the
frames of the image data output from the A/D conversion
30 unit 21. The picture rearrangement buffer 22 rearranges
the frames in accordance with the GOP (Group of Pictures)
11
SP301579WO00
structure related to the encoding operation, and outputs
the rearranged image data to the subtraction unit 23, the
intra prediction unit 51, and the motion/parallax
prediction/compensation unit 52.
5 [0029]
The subtraction unit 23 receives the image data
output from the picture rearrangement buffer 22 and
predicted image data selected by the later described
predicted image/optimum mode select unit 53. The
10 subtraction unit 23 calculates prediction error data that
is the difference between the image data output from the
picture rearrangement buffer 22 and the predicted image
data supplied from the predicted image/optimum mode
select unit 53, and outputs the prediction error data to
15 the orthogonal transform unit 24.
[0030]
The orthogonal transform unit 24 performs an
orthogonal transform operation, such as a discrete cosine
transform (DCT) or a Karhunen-Loeve transform, on the
20 prediction error data output from the subtraction unit 23.
The orthogonal transform unit 24 outputs coefficient data
obtained by performing the orthogonal transform operation
to the quantization unit 25.
[0031]
25 The quantization unit 25 receives the coefficient
data output from the orthogonal transform unit 24 and a
rate control signal supplied from the later described
rate control unit 28. The quantization unit 25 quantizes
the coefficient data, and outputs the quantized data to
30 the lossless encoding unit 26 and the inverse
quantization unit 31. Based on the rate control signal
12
t
SP301579WO00
supplied from the rate control unit 28, the quantization
unit 25 switches quantization parameters (quantization
scales), to change the bit rate of the quantized data.
[0032]
5 The lossless encoding unit 26 receives the
quantized data output from the quantization unit 25, and
prediction mode information supplied from the later
described intra prediction unit 51, the motion/parallax
prediction/compensation unit 52, and the predicted
10 image/optimum mode select unit 53. The prediction mode
information contains a macroblock type indicating the
block size of the picture being encoded, a prediction
mode, a reference index, and the like. The lossless
encoding unit 26 performs an encoding operation on the
15 quantized data through variable-length coding or
arithmetic coding or the like, to generate and output an
encoded stream to the accumulation buffer 27. The
lossless encoding unit 26 also performs lossless coding
on the prediction mode information, and adds the
20 resultant information to the header information in the
encoded stream, for example.
[0033]
The accumulation buffer 27 stores the encoded
stream supplied from the lossless encoding unit 26. The
25 accumulation buffer 27 also outputs the stored encoded
stream at a transmission rate in accordance with the
transmission path.
[0034]
The rate control unit 28 monitors the free space in
30 the accumulation buffer 27, generates a rate control
signal in accordance with the free space, and outputs the
13
SP301579WO00
rate control signal to the quantization unit 25. The
rate control unit 2 8 obtains information about the free
space from the accumulation buffer 27, for example. When
the remaining free space is small, the rate control unit
5 28 lowers the bit rate of the quantized data through the
rate control signal. When the remaining free space in
the accumulation buffer 27 is sufficiently large, the
rate control unit 28 increases the bit rate of the
quantized data through the rate control signal.
10 [0035]
The inverse quantization unit 31 inversely
quantizes the quantized data supplied from the
quantization unit 25. The inverse quantization unit 31
outputs the coefficient data obtained by performing the
15 inverse quantization operation to the inverse orthogonal
transform unit 32.
[0036]
The inverse orthogonal transform unit 32 performs
an inverse orthogonal transform operation on the
20 coefficient data supplied from the inverse quantization
unit 31, and outputs the resultant data to the addition
unit 33.
[0037]
The addition unit 33 adds the data supplied from
25 the inverse orthogonal transform unit 32 to the predicted
image data supplied from predicted image/optimum mode
select unit 53, to generate image data of a reference
picture. The addition unit 33 outputs the image data to
the deblocking filter 34 and the intra prediction unit 51.
30 [0038]
The deblocking filter 34 performs a filtering
14
SP301579WO00
operation to reduce block distortions that occur at the
time of image encoding. The deblocking filter 34
performs a filtering operation to remove block
distortions from the image data supplied from the
5 addition unit 33, and outputs the filtered image data to
the frame memory 35.
[0039]
The frame memory 35 stores the filtered image data
supplied from the deblocking filter 34, and the reference
10 picture image data supplied from an image encoding device
20bv that encodes a base view.
[0040]
A feature amount generation unit 41 generates a
feature amount. In a case where the image data of a
15 dependent view is encoded, the feature amount is the ;
information for determining whether the correlation
between images of different viewpoints is low. The
feature amount generation unit 41 generates the feature
amount from the information obtained in the operation to
20 encode the first picture in the GOP, for example. The
feature amount generation unit 41 uses, as the feature
amount, the total sum (such as SAD: Sum of Absolute
Differences) of differences between the blocks of
pictures being encoded (blocks being encoded) and the
25 block of the reference picture (the reference block) in
the image when parallax vectors are detected, for example.
The feature amount generation unit 41 may also use, as
the feature amount, the proportion of intra macroblocks
in the image, or the complexity ratio between the picture
30 being encoded and an image of a reference picture of a
different viewpoint, or the like.
15
SP301579WO00
[0041]
In a case where the differences between the blocks
being encoded and the reference block are used as the
feature amount, the feature amount generation unit 41
5 calculates the total sum of the differences calculated by
the later described motion/parallax
prediction/compensation unit 52 in the image, and sets
the total sum as the feature amount.
[0042]
10 In a case where the proportion of the intra
macroblocks is used as the feature amount, the feature
amount generation unit 41 calculates the proportion of
macroblocks determined to have an intra prediction as an
optimum mode in a parallax prediction by the later
15 described predicted image/optimum mode select unit 53 in
an image, and sets the calculated proportion as the
feature amount.
[0043]
In a case where the complexity ratio of an encoded
20 picture is used as the feature amount, the feature amount
generation unit 41 calculates the complexity of the
encoded first picture in the GOP, and sets the calculated
complexity ratio as the feature amount. That is, the
feature amount generation unit 41 calculates the
25 complexities Xi and Xp of an I-picture (Ibvl) and a Ppicture
(Pdvl) according to the equations (1) and (2),
and sets the ratio between the calculated complexities
(Xi/Xp) as the feature amount:
Xi = SiQi ... (1)
30 Xp = SpQp ... (2)
[0044]
16
SP301579WO00
In the equation (1), Xi represents the complexity
of the I-picture, Si represents the generated coding
amount of the I-picture, and Qi represents the mean
quantization scale code (a quantization parameter) used
5 at the time of I-picture encoding. Likewise, in the
equation (2), Xp represents the complexity of the Ppicture,
Sp represents the generated coding amount of the
P-picture, and Qp represents the mean quantization scale
code (a quantization parameter) used at the time of P-
10 picture encoding.
[0045]
As described above, the feature amount generation
unit 41 outputs a feature amount to the reference index
assignment unit 45, the feature amount being at least one
15 of the total sum of the differences between the blocks
being encoded and the reference block in the image, the
proportion of intra macroblocks in the image, and the
complexity ratio between the picture being encoded and
the image of a reference picture of a different viewpoint.
20 [0046]
Based on the feature amount generated by the
feature amount generation unit 41, the reference index
assignment unit 45 deteirmines a reference index
assignment method for the reference picture of the
25 parallax prediction and the reference picture of the
temporal prediction. For example, in a case where the
feature amount is generated from the information obtained
in the operation to encode the first picture in the GOP,
the reference index assignment unit 45 determines the
30 reference index assignment method for the subsequent
pictures (the pictures other than the first picture) in
17
SP301579WO00
the GOP. By the determined assignment method, the
reference index assignment unit 45 assigns reference
indexes to the reference pictures stored in the frame
memory 35.
5 [0047]
In a case where the total sum of the differences
between the blocks being encoded and the reference block
is generated as the feature amount, the reference index
assignment unit 45 determines the correlation to be low
10 when the total sum is larger than a predetermined
threshold value. In a case where the proportion of intra
macroblocks is generated as the feature amount, the
reference index assignment unit 45 determines the
correlation to be low when the proportion is larger than
15 a predetermined threshold value. In a case where the
complexity ratio is generated as the feature amount, the
reference index assignment unit 45 determines the
correlation to be low when the complexity ratio (Xi/Xp)
is lower than a predetermined threshold value.
20 [0048]
When determining the correlation to be lower than a
predetermined threshold value, the reference index
assignment unit 45 changes the reference index assignment,
and re-assigns the reference index assigned to the
25 parallax prediction reference picture, to the temporal
prediction reference picture.
[0049]
The intra prediction unit 51 performs intra
prediction operations in all candidate intra prediction
30 modes, using the image data of the picture being encoded
output from the picture rearrangement buffer 22 and the
18
SP301579WO00
image data supplied from the addition unit 33. The intra
prediction unit 51 further calculates a cost function
value in each of the intra prediction modes, and selects
an optimum intra prediction mode that is the intra
5 prediction mode with the smallest cost function value
calculated or the intra prediction mode with the highest
encoding efficiency. The intra prediction unit 51
outputs the predicted image data generated in the optimum
intra prediction mode, the prediction mode information
10 about the optimum intra prediction mode, and the cost
function value in the optimum intra prediction mode, to
the predicted image/optimiam mode select unit 53. The
intra prediction unit 51 also outputs the prediction mode
information about the intra prediction mode in the intra
15 prediction operation in each intra prediction mode to the
lossless encoding unit 26, so as to obtain the generated
coding amount used in the calculation of the cost
function values. The cost function values can be
calculated by a method called JM (Joint Model) installed
20 in H.264/AVC reference software, for example.
[0050]
The motion/parallax prediction/compensation unit 52 i
performs a motion/parallax prediction/compensation
operation for each block size of blocks being encoded.
25 From each image of each block being encoded among images
read out from the picture rearrangement buffer 22, the
motion/parallax prediction/compensation unit 52 detects
motion vectors by using image data that is read out from
the frame memory 35 and has been subjected to a
30 deblocking filtering operation, and detects parallax
vectors by using the image data of a base view. Based on
19
SP301579WO00
the detected vectors, the motion/parallax
prediction/compensation unit 52 further performs a
compensation operation on the reference picture, to
generate a predicted image.
5 [0051]
Also, the motion/parallax prediction/compensation
unit 52 calculates a cost function value for each block
size of pictures being encoded and each reference picture,
and selects an optimum inter prediction mode that is the
10 block size and the reference picture having the smallest
cost function value. The motion/parallax
prediction/compensation unit 52 outputs the predicted
image data generated in the optimum inter prediction mode,
the prediction mode information about the optimum inter
15 prediction mode, and the cost function value in the .
optimum inter prediction mode, to the predicted
image/optimum mode select unit 53. To obtain the
generated coding amount used in the calculation of the
cost function values, the motion/parallax
20 prediction/compensation unit 52 also outputs the
prediction mode information about the inter prediction
mode to the lossless encoding unit 26 in the inter
prediction operation with each block size. Further, in a
case where the total sum of the differences between the
25 blocks being encoded and the references block is
generated as the feature amount by the feature amount
generation unit 41, the motion/parallax
prediction/compensation unit 52 calculates the difference
between each block being encoded and the reference block
30 when parallax vectors are detected, and outputs the
difference to the feature amount generation unit 41.
20
SP301579WO00
[0052]
The predicted image/optimum mode select unit 53
compares the cost function value supplied from the intra
prediction unit 51 with the cost function value supplied
5 from the motion/parallax prediction/compensation unit 52,
and selects the mode with the smaller cost function value
as the optimum mode with the highest encoding efficiency.
The predicted image/optimum mode select unit 53 also
outputs the predicted image data generated in the optimum
10 mode to the subtraction unit 23 and the addition unit 33.
The predicted image/optimum mode select unit 53 further
outputs the prediction mode information (such as the
macroblock type, the prediction mode, and the reference
index) about the optimum mode to the lossless encoding
15 unit 26. Further, in a case where the proportion of
intra macroblocks is generated as the feature amount by
the feature amount generation unit 41, the predicted
image/optimum mode select unit 53 outputs the information
about the macroblocks for which an intra prediction mode
20 has been selected among the pictures being encoded, to
the feature amount generation unit 41.
[0053]
<3. Operation of the image processing device>
Fig. 4 is a flowchart showing an operation of the
25 image encoding device 20dv. In step STl, the image
encoding device 20dv determines whether the picture being
encoded is a picture of a dependent view. If the picture
being encoded is a picture of a dependent view, the image
encoding device 20dv moves on to step ST2. If the
30 picture being encoded is a picture of a base view, the
image encoding device 20dv moves on to step ST9.
21
SP301579WO00
[0054]
In step ST2, the image encoding device 20dv
determines whether the picture being encoded is the first
picture in the GOP. If the picture being encoded is the
5 first picture, the image encoding device 20dv moves on to
step ST3. If the picture being encoded is a subsequent
picture in the GOP, the image encoding device 20dv moves
on to step ST6.
[0055]
10 In step ST3, the image encoding device 20dv
performs an encoding operation on the picture being
encoded, and moves on to step ST4. In the encoding
operation, the reference index assignment unit 45 sets
reference indexes by a predetermined assignment method.
15 [0056]
In step ST4, the image encoding device 20dv
generates a feature amount. The feature amount
generation unit 41 of the image encoding device 20dv
generates the feature amount from the information
20 obtained in the operation to encode the first picture,
and moves on to step ST5. The feature amount generation
unit 41 generates the feature amount that is the total
sum of the differences between the blocks being encoded
and the reference block in the image when parallax
25 vectors are detected, the proportion of the intra
macroblocks in the image, the complexity ratio of the
image, or the like.
[0057]
In step ST5, the image encoding device 20dv
30 determines a reference index assignment method. Based on
the feature amount generated in step ST4, the reference
22
SP301579WO00
index assignment unit 45 of the image encoding device
20dv determines the reference index assignment method to
be implemented in the operation to encode the subsequent
pictures. When determining that the correlation between
5 the pictures of the dependent view and the base view is
low based on the feature amount, the reference index
assignment unit 45 uses the assignment method to reassign
the reference index assigned to a parallax
prediction, to another reference picture of a temporal
10 prediction. In a case where the total s\am of the
differences between the blocks being encoded and the
reference block in the image is generated as the feature
amount, for example, the reference index assignment unit
45 determines the correlation to be low when the total
15 sum is larger than a predetermined threshold value. In a
case where the proportion of intra macroblocks in the
image is generated as the feature amount, for example,
the reference index assignment unit 45 determines the
correlation to be low when the proportion is larger than
20 a predetermined threshold value. In a case where the
proportion of intra macroblocks in the image is generated
as the feature amount, for example, the reference index
assignment unit 45 determines the correlation to be low
when the proportion is larger than a predetermined
25 threshold value. Further, in a case where the complexity
ratio is used as the feature amount, the reference index
assignment unit 45 determines the correlation to be low
when the complexity ratio is lower than a predetermined
threshold value. When determining the correlation to be
30 low, the reference index assignment unit 45 uses, for the
subsequent pictures, the assignment method to re-assign
23
SP301579WO00
the reference index assigned to a parallax prediction, to
another reference picture of a temporal prediction.
[0058]
When moving on to step ST6 after the picture being
5 encoded is determined not to be the first picture in the
GOP in step ST2, the image encoding device 20dv
determines whether the assignment method needs to be
changed. If the assignment method determined beforehand
for the first picture in the GOP differs from the
10 assignment method determined for the subsequent pictures
in step ST5, the image encoding device 20dv moves on to
step ST7. If those methods are the same, the image
encoding device 20dv moves on to step ST8.
[0059]
15 In step ST7, the image encoding device 20dv issues
a RPLR (Reference Picture List Reordering) command. The
reference index assignment unit 4 5 of the image encoding
device 20dv issues the RPLR command so that correct
reference pictures can be used in an image decoding
20 device based on the reference indexes even if the
reference index assignment to the subsequent pictures is
changed. That is, the reference index assignment unit 45
supplies the RLPR, which is a syntax element, to the
lossless encoding unit 26, and incorporates the RLPR into
25 the header of the encoded stream of image data, for
example. The operation then moves on to step ST8.
[0060]
In step ST8, the image encoding device 20dv
performs an encoding operation on the picture being
30 encoded. In the encoding operation, the reference index
assignment unit 45 also sets reference indexes by the
24
SP301579WO00
assignment method determined for the subsequent pictures
in step ST5.
[0061]
When moving on to step ST9 after the picture being
5 encoded is determined not to be a picture of a dependent
view in step STl, the image encoding device 20dv assigns
reference indexes by the assignment method determined
beforehand, and performs an encoding operation.
[0062]
10 As this operation is performed for each picture
being encoded, the reference index assigned to a
reference picture of a parallax prediction is re-assigned
to another reference picture of a temporal prediction
when the correlation between the images of a dependent
15 view and a base view is determined to be low based on the
first picture in the GOP.
[0063]
Fig. 5 illustrates a reference index assignment
method to be implemented in a case where the correlation
20 between images of a dependent view and a base view is low.
In a case where the correlation between images of a
dependent view and a base view is determined to be low
based on the first picture, the reference index
assignment method for the subsequent pictures in the GOP
25 is changed. For example, a reference index ref_idx = 0
is assigned to the reference picture one frame before,
and a reference index ref_idx = 1 is assigned to the
reference picture two frames before. Therefore, in the
case illustrated in Fig. 5, in an operation to encode a
30 P-picture (Pdv3) in the image data of Caml (a dependent
view), the reference index ref_idx = 1 assigned to a P-
25
SP301579WO00
picture (Pbv3) in the image data of CamO (a base view) as
a parallax prediction reference picture is re-assigned to
a P-picture (Pdvl) of the dependent view.
[0064]
5 As reference indexes are assigned to the parallax
prediction reference picture and the temporal prediction
reference picture based on the feature amount, the
reference index assignment method is changed when the
correlation between images is determined to be low, so
10 that a base picture of a different viewpoint with a low
degree of correlation is not used as a reference picture.
Further, encoding can be performed by selecting a
reference picture with high encoding efficiency from
reference pictures in a temporal prediction. Accordingly,
15 the encoding efficiency in multi-view image encoding can
be increased.
[0065]
In Fig. 5, the GOP of the dependent view is formed
with I-pictures and P-pictures. However, in a GOP
20 structure containing B-pictures, reference index
assignment is also changed when the correlation is
determined to be low.
[0066]
Figs. 6 illustrate a reference index assignment
25 method to be implemented in a case where the GOP contains
B-pictures. Fig. 6(A) illustrates a situation prior to
assignment, and Fig. 6(B) illustrates a situation after
the assignment.
[0067]
30 A B-picture in the image data of Caml has a
reference picture that is a P-picture of Caml, which is
26
SP301579WO00
referred to in an anterior prediction, or a Bs-picture in
the image data of CamO, which is referred to in a
parallax prediction, in the LO prediction (LIST_0). The
B-picture has another reference picture that is a P-
5 picture of Caml, which is referred to in a posterior
prediction in the Ll prediction (LIST_1), for example.
Further, the pictures that can be used in LIST_X (X being
0 or 1) are managed by reference indexes ref_idx, as
described above. Fig. 6(A) illustrates an example case
10 where a reference index ref_idx = 0 is assigned to
temporal prediction reference pictures, and a reference
index ref_idx = 1 is assigned to a parallax prediction
reference picture.
[0068]
15 When the correlation between images of a dependent
view and a base view is determined to be low in the first
picture in the GOP, the reference index assignment unit
45 re-assigns a reference index as shown in Fig. 6(B).
In the operation to encode a B-picture (Bdv4), for
20 example, the reference index assignment unit 45 reassigns
the reference index ref_idx = 1 assigned to the
Bs-picture (Bsbv4), which is the parallax prediction
reference picture, to a P-picture (Pdvl) of the dependent
view.
25 [0069]
As described above, when the correlation between
the images of a dependent view and a base view is
determined to be low in the first picture in the GOP, the
reference index assignment method is changed, so that a
30 base picture of a different viewpoint with a low degree
of correlation is not used as a reference picture in the
27
SP301579WO00
operation to encode a B-picture. Also, encoding can be
performed by selecting a reference picture with a high
encoding efficiency from reference pictures in a temporal
prediction. Accordingly, the encoding efficiency in
5 multi-view image encoding can be increased.
[0070]
Also, in the above described embodiment, the
correlation between the images of a dependent view and a
base view is determined by using the first picture in the
10 GOP. In a case where the total sum of the differences
between the blocks being encoded and the reference block
is used as the feature amount, however, a check can be
made to determine whether the correlation between images
is low even in the middle of the GOP. Accordingly, when
15 the correlation between images is determined to be low
based on the feature amount in the middle of the GOP, the
reference index assignment method can be changed.
[0071]
In the above described embodiment, the reference
2 0 index assignment method is changed when the correlation
between images is determined to be low. However, the GOP
structure can also be changed, to increase the encoding
efficiency in multi-view image encoding.
[0072]
25 In a case where a reference index is re-assigned as
shown in Fig. 6(B), the P-picture (Pdvl) of the dependent
view is further away from the B-picture (Bdv4) in terms
of time. Therefore, in a case where the correlation
between images of a dependent view and a base view is
30 determined to be low in the first picture in the GOP, the
GOP structure is changed so that a reference index can be
28 [
I
[
SP301579WO00
assigned to a non-reference picture that is closer to the
picture being encoded in terms of time.
[0073]
Figs. 7 illustrates a case where the GOP structure
5 is changed. Fig. 7(A) illustrates a situation prior to
assignment. Figs. 7(B) and 7(C) illustrate situations
where changes are made to the assignment and the GOP
structure.
[0074]
10 Fig. 7(A) illustrates an example case where a
reference index ref_idx = 0 is assigned to temporal
prediction reference pictures, and a reference index
ref_idx = 1 is assigned to a parallax prediction
reference picture.
15 [0075]
When the correlation between images of a dependent
view and a base view is determined to be low in the first
picture in the GOP, a change is made to the GOP picture
to turn a B-picture (Bdv2), which is a non-reference
20 picture, into a P-picture (Pdv2) , as shown in Fig. 7(B).
In the operation to encode a B-picture (Bdv4) in this
case, the reference index ref_idx = 1 assigned to the Bspicture
(Bsbv4), which is a parallax prediction reference i
picture, can be re-assigned to a P-picture (Pdv2), which
25 is closer than in the case of Fig. 6(B) in terms of time.
Although a B-picture (Bdv2) is changed to a P-picture
(Pdv2) in Fig. 7(B), all the B-pictures in the GOP may be
changed to P-pictures.
[0076]
30 When the correlation between images of a dependent
view and a base view is determined to be low in the first
29
SP301579WO00
picture in the GOP, a change can also be made to the GOP
picture to turn a B-picture (Bdv2), which is a nonreference
picture, into a Bs-picture (Bsdv2), as shown in
Fig. 7(C). In the operation to encode the B-picture
5 (Bdv4) in this case, the reference index ref_idx = 1
assigned to the Bs-picture (Bsbv4), which is a parallax
prediction reference picture, can be re-assigned to the
Bs-picture (Bsdv2), which is closer than in the case of
Fig. 6(B) in terms of time.
10 [0077]
In this manner, when the correlation between images
of a dependent view and a base view is determined to be
low in the first picture of the GOP, a change is made to
the GOP structure, and a reference index is assigned to a
15 picture that is closer in the temporal direction. That
is, a B-picture that is closer in the temporal direction
is turned into a P-picture or a Bs-picture, so that the
P-picture or the Bs-picture can be referred to from the
picture being encoded. As a change is made to the GOP
2 0 structure in this manner, encoding can be performed by
using a reference picture that is a picture closer in the
temporal direction. Accordingly, the prediction accuracy
can be made higher than in a case where the GOP structure
is not changed. Thus, the encoding efficiency in multi-
25 view image encoding can be increased.
[0078]
In a case where image data of FS-AVC, by which
images of different viewpoints are switched by the frame,
for example, the feature amount generation unit 41
30 generates the feature amount by using image data of
another viewpoint extracted from input image data. The
30
SP301579WO00
image data of another viewpoint extracted from the input
image data, and the image data of a reference picture
generated by encoding the image data of another viewpoint
are also stored in the frame memory 35. Through such an
5 operation, FS-AVC image data can also be encoded.
[0079]
<4. Structure used in a case where image encoding
is performed through a software operation>
Further, the image processing device may be a
10 computer device that performs the above described series
of operations in accordance with a program.
[0080]
Fig. 8 is a diagram showing an example structure of
a computer device that performs the above described
15 series of operations in accordance with a program. A CPU
(Central Processing Unit) 61 of a computer device 60
performs various kinds of operations in accordance with a
computer program recorded on a ROM (Read Only Memory) 62
or a recording unit 68.
20 [0081]
Computer programs to be executed by the CPU 61 and
data are stored in a RAM (Random Access Memory) 63 as
appropriate. The CPU 61, the ROM 62, and the RAM 63 are
connected to one another by a bus 64.
25 [0082]
An input/output interface 65 is also connected to
the CPU 61 via the bus 64. An input unit 66 such as a
touch panel, a keyboard, a mouse, or a microphone, and an
output unit 67 formed with a display or the like are
30 connected to the input/output interface 65. The CPU 61
performs various kinds of operations in accordance with
31
SP301579WO00
instructions input through the input unit 66. The CPU 61
also outputs operation results to the output unit 67.
[0083]
The recording unit 68 connected to the input/output
5 interface 65 is formed with a hard disk or a SSD (Solid
State Drive), and records computer programs to be
executed by the CPU 61 and various kinds of data. A
communication unit 69 communicates with an external
device via a wired or wireless communication medium such
10 as a network like the Internet or a local area network,
or digital broadcasting. The computer device 60 may also
obtain a computer program via the communication unit 69,
and record the computer program on the ROM 62 or the
recording unit 68.
15 [0084]
When a removable medium 72 that is a magnetic disk,
an optical disk, a magnetooptical disk, a semiconductor
memory, or the like is mounted, a drive 70 drives the
removable medium 72, to obtain a recorded computer
20 program and recorded data. The obtained computer program
and data are transferred to the ROM 62, the RAM 63, or
the recording unit 68, where necessary.
[0085]
The CPU 61 reads and executes the computer program
25 for performing the above described series of operations,
and performs an encoding operation on the image data of
multi-view images recorded on the recording unit 68 or
the removable medium 72, or on the image data of multiview
images supplied via the communication unit 69.
30 [0086]
It should be noted that the present invention
32
SP301579WO00
should not be interpreted to be limited to the above
described embodiments of the invention. For example, a
multi-view image is not necessarily formed with the two
images of a left-eye image and a right-eye image, but may
5 be formed with images of three or more viewpoints. The
embodiments of the invention disclose the present i
I
invention through examples, and it should be obvious that !
those skilled in the art can modify or replace those
embodiments with other embodiments without departing from
10 the scope of the invention. That is, the claims should
be taken into account in understanding the subject matter
of the invention.
i
INDUSTRIAL APPLICABILITY
I
15 [0087]
In the image processing device and the image
processing method of the present invention, a feature
amount indicating the correlation between images of
different viewpoints is generated. When the correlation
20 is determined to be lower than a predetermined threshold
value from the feature amount, the reference index
assigned to a reference picture of a parallax prediction
using the correlation between images of different
viewpoints is re-assigned to a reference picture of a
25 temporal prediction using the correlation between images
xn the temporal direction. Accordingly, the encoding
efficiency in a case where the correlation between images
of different viewpoints is low can be increased in multiview
image encoding. Thus, the present invention can be
30 applied to imaging devices that generate and encode
multi-view images, editing devices that edit and encode [
33 I
r
I
SP301579WO00
multi-view images, recording devices that encode multiview
images and record the encoded images on recording
media, and the like.
5 REFERENCE SIGNS LIST
[0088]
10 Encoding system, IIL Left-viewpoint image
generating device, llR Right-viewpoint image generating
device, 20 Multi-view encoding device, 20bv, 20dv Image
10 encoding device, 21 A/D conversion unit, 22 Picture
rearrangement buffer, 23 Subtraction unit, 24 Orthogonal
transform unit, 25 Quantization unit, 26 Lossless
encoding unit, 27 Acciomulation buffer, 28 Rate control
unit, 31 Inverse quantization unit, 32 Inverse orthogonal
15 transform unit, 33 Addition unit, 34 Deblocking filter,
35 Frame memory, 41 Feature amount generation unit, 45
Reference index assignment unit, 51 Intra prediction unit,
52 Motion/parallax prediction/compensation unit, 53
Predicted image/optimum mode select unit, 60 Computer
20 device, 61 CPU (Central Processing Unit), 62 ROM (Read
Only Memory), 63 RAM (Random Access Memory), 64 Bus, 65
Input/output interface, 66 Input unit, 67 Output unit, 68
Recording unit, 69 Communication unit, 70 Drive, 72
Removable medium
25
34

CLAIMS
1. An image processing device comprising:
a feature amount generation unit configured to
5 generate a feature amount indicating a correlation
between images of different viewpoints; and
a reference index assignment unit configured to reassign
a reference index assigned to a reference picture
of a parallax prediction using the correlation between
10 the images of different viewpoints, to a reference
picture of a temporal prediction using a correlation
between images in a temporal direction, when the
correlation between the images of different viewpoints is
determined to be lower than a predetermined threshold
15 value based on the feature amount generated by the
feature amount generation unit.
2. The image processing device according to claim 1,
wherein the feature amount generation unit generates the
20 feature amount, based on an operation to encode a first
picture in a GOP, and
the reference index assignment unit performs the
reference index assignment on a subsequent picture
following the first picture in the GOP, based on the
25 feature amount.
3. The image processing device according to claim 2,
wherein, in the operation to encode the first picture in
the GOP, the feature amount generation unit calculates
30 the feature amount that is at least one of a total sum of
differences between a block being encoded and a reference
35
SP301579WO00
block in the image when a parallax vector is detected, a
proportion of intra macroblocks in the image, and an
image complexity ratio between the picture being encoded
and a reference picture of a different viewpoint.
5
4. The image processing device according to claim 2,
wherein, when the correlation is determined to be lower
than the predetermined threshold value, the reference
index assignment unit turns a non-reference picture into
10 a reference picture in the GOP structure, the nonreference
picture being closer in a temporal direction.
5. The image processing device according to claim 1,
wherein the feature amount generation unit calculates the
15 feature amount that is a total sum of differences between
a block being encoded and a reference block in an image
when a motion vector is detected through motion detection,
and
based on the feature amount, the reference index
20 assignment unit performs the reference index assignment
on a subsequent picture following the picture used in the
calculation of the feature amount.
6. An image processing method comprising the steps of:
25 generating a feature amount indicating a
correlation between images of different viewpoints; and
re-assigning a reference index assigned to a
reference picture of a parallax prediction using the correlation between the images of different viewpoints, to a reference picture of a temporal prediction using a correlation between images in a temporal direction, when the correlation between the images, of different viewpoints is determined to be lower than a predetermined threshold value based on the generated feature amount.