DESCRIPTION
Title of Invention: IMAGE PROCESSING APPARATUS AND IMAGE
PROCESSING METHOD
Technical Field
[0001]
The present invention relates to an image processing
apparatus and image processing method, and more particularly,
to an image processing apparatus and image processing method
configured to be able to generate high-quality syntheticview
images.
Background Art
[0002]
There exists view synthesis technology that generates
images with arbitrary views. This view synthesis technology
is a technology that generates 2D images for M views (>N)
from 2D images for N views and depth images (depth
information).
[0003]
An overview of view synthesis technology will be
described with reference to Fig. 1.
[0004]
As illustrated in Fig. 1, with view synthesis
technology, input images 1 consisting of a 2D image and a
- 2 -
SP302819
depth image are input. Then, view synthesis is conducted
using the input images 1, and synthetic-view images 2 are
generated, being 2D images of views farther inward and
outward from the camera positions for the input images 1.
[ 0005]
In the example in Fig. 1, synthetic-view images 2 for
six views are generated from input images 1 for two views.
The input images 1 for two views and the synthetic-view
images 2 for six views yield images for a total of eight
views.
[0006]
In actual applications, view synthesis technology is
used in conjunction with compression technology. An
exemplary configuration of a system combining view synthesis
technology and compression technology is illustrated in Fig.
2.
[0007]
In the system in Fig. 2, 2D images 11 for N views and
depth images 12 for N views are input into a multiview video
encoding apparatus 13.
[0008]
The multiview video encoding apparatus 13 encodes the
2D images 11 for N views and the depth images 12 for N views
in an Advanced Video Coding (AVC) format or Multiview Video
Coding (MVC) format, and supplies them to a multiview video
- 3 -
SP302819
decoding apparatus 14.
[0009]
The multiview video decoding apparatus 14 takes the
encoded 2D images 11 for N views and depth images 12 for N
views supplied from the multiview video encoding apparatus
13, decodes them in a format corresponding to the AVC format
or MVC format, and supplies them to a view synthesizing
apparatus 15.
[0010]
The view synthesizing apparatus 15 uses the 2D images
11 and depth images 12 for N views obtained as a result of
the decoding by the multiview video decoding apparatus 14 to
generate synthetic-view images for (M-N) views. The view
synthesizing apparatus 15 outputs 2D images for M views,
which consist of the 2D images 11 for N views and the
synthetic-view images for (M-N) views, as reconstructed 2D
images 16 for M views.
[0011]
Meanwhile, a method of encoding and decoding image data
for multiple views is described in PTL 1, for example.
Citation List
Patent Literature
[0012]
PTL 1: Japanese Unexamined Patent Application
Publication No. 2008-182669
- 4 -
SP302819
Summary of Invention
Technical Problem
[0013]
With the system in Fig. 2, since images for many views
are generated from images for a few views, large amounts of
missing information occur in the synthetic-view images.
Particularly, information on occluded areas is completely
missing in the case of generating images for views farther
outward from the camera positions of the input images. In
the related art, surrounding in-picture information is used
to interpolate such occluded areas. However, there are
limits to in-picture interpolation capability, and the
quality of synthetic-view images is not adequate.
[0014]
The present invention has been devised in light of such
circumstances, and is configured to enable the generation of
high-quality synthetic-view images.
Solution to Problem
[0015]
An image processing apparatus according to a first
aspect of the present invention is an image processing
apparatus provided with receiving means that receives
residual information, which is the error between syntheticview
images generated using reference 2D images and depth
information, and 2D images at the view synthesis positions
- 5 -
SP302819
of the synthetic-view images, encoding means that encodes
the reference 2D images to generate an encoded stream, and
transmitting means that transmits the residual information
received by the receiving means, the depth information, and
the encoded stream generated by the encoding means.
[0016]
An image processing method according to the first
aspect of the present invention corresponds to an image
processing apparatus according to the first aspect of the
present invention.
[0017]
In the first aspect of the present invention, residual
information is received, residual information being the
error between synthetic-view images, which are generated
using reference 2D images and depth information, and 2D
images at the view synthesis positions of the synthetic-view
images. The reference 2D images are encoded to generate an
encoded stream. The residual information, the depth
information, and the encoded stream are transmitted.
[0018]
An image processing apparatus according to a second
aspect of the present invention is an image processing
apparatus provided with receiving means that receives
residual information and depth information, the residual
information being the error between synthetic-view images
- 6 -
SP302819
generated using reference 2D images and the depth
information, and 2D images at the view synthesis positions
of the synthetic-view images, decoding means that decodes an
encoded stream obtained as a result of encoding the
reference 2D images, generating means that generates the
synthetic-view images using the reference 2D images decoded
by the decoding means and the depth information received by
the receiving means, and residual information compensating
means that adds the residual information received by the
receiving means into the synthetic-view images generated by
the generating means.
[0019]
An image processing method according to the second
aspect of the present invention corresponds to an image
processing method according to the second aspect of the
present invention.
[0020]
In the second aspect of the present invention, residual
information and depth information are received, the residual
information being the error between synthetic-view images
generated using reference 2D images and the depth
information, and 2D images at the view synthesis positions
of the synthetic-view images. An encoded stream obtained as
a result of encoding the reference 2D images is decoded, and
the synthetic-view images are generated using the decoded
- 7 -
SP302819
reference 2D images and the received depth information. The
received residual information is added into the generated
synthetic-view images.
Advantageous Effects of Invention
[0021]
According to the first aspect of the present invention,
it is possible to transmit information for generating highquality
synthetic-view images.
[0022]
According to the second aspect of the present invention,
it is possible to generate high-quality synthetic-view
images.
Brief Description of Drawings
[0023]
[Fig. 1] Fig. 1 is a diagram illustrating an overview
of view synthesis technology.
[Fig. 2] Fig. 2 is a block diagram illustrating an
exemplary configuration of a system of the related art.
[Fig. 3] Fig. 3 is a diagram illustrating an overview
of the present invention.
[Fig. 4] Fig. 4 is a diagram illustrating an overview
of the present invention.
[Fig. 5] Fig. 5 is a block diagram illustrating an
exemplary configuration of an embodiment of a system to
which the present invention has been applied.
- 8 -
SP302819
[Fig. 6] Fig. 6 is a flowchart illustrating processing
by the system in Fig. 5.
Description of Embodiments
[0024]
[Overview of invention]
Figs. 3 and 4 are diagrams illustrating an overview of
the present invention.
[0025]
Herein, Figs. 3 and 4 are diagrams illustrating the
case of generating images for eight views from images for
two views.
[0026]
As illustrated in Fig. 3, in the present invention not
only are input images 1 encoded, but also residual
information, which is information on the error between the
synthetic-view images 2 and 2D images 31 acquired by cameras
at the view synthesis positions of the synthetic-view images
2. This residual information is useful information that
includes missing information missing from the synthetic
images.
[0027]
Then, when conducting view synthesis, the input images
1 are used to generate synthetic-view images 2, and those
synthetic-view images 2 are compensated by the residual
information to generate final synthetic-view images 41, as
- 9 -
SP302819
illustrated in Fig. 4.
[0028]
In this way, in the present invention, residual
information is added into synthetic-view images 2, thereby
making it possible to compensate for missing information and
generate high-quality synthetic-view images 41.
[0029]
[Exemplary configuration of embodiment of system]
Fig. 5 is a block diagram illustrating an exemplary
configuration of an embodiment of a system to which the
present invention has been applied.
[0030]
Note that in Fig. 5, like signs are given to elements
like those in Fig. 2, and description thereof is omitted or
reduced as appropriate because such description would be
repetitive.
[0031]
In the system in Fig. 5, 2D images 11 for N views
(reference 2D images), depth images 12 for N views, and 2D
images 101 for (M-N) views at the view synthesis positions
are input. The 2D images 101 for (M-N) views are imaged by
cameras at the view synthesis positions and are used in
order to acquire residual information.
[0032]
In order to acquire residual information with a
- 10 -
SP302819
residual information acquiring apparatus 103, a view
synthesizing apparatus 102 uses the 2D images 11 for N views
and the depth images 12 for N views to generate synthetic-.
view images for (M-N) views similarly to the view
synthesizing apparatus 15 in Fig. 1, and supplies them to
the residual information acquiring apparatus 103.
[0033]
The residual information acquiring apparatus 103
calculates the error between the synthetic-view images for
(M-N) views supplied from the view synthesizing apparatus
102 and the 2D images 101 for (M-N) views at the view
synthesis positions, and takes the result as residual
information. The residual information acquiring apparatus
103 supplies the residual information to a multiview video
encoding apparatus 104.
[0034]
The multiview video encoding apparatus 104 encodes the
2D images 11 for N views, the depth images 12 for N views,
and the residual information supplied from the residual
information acquiring apparatus 103 in an AVC format or MVC
format. Then, the multiview video encoding apparatus 104
supplies the encoded stream obtained as a result of the
encoding to a multiview video decoding apparatus 105.
[0035]
The multiview video decoding apparatus 105 decodes the
- 11 -
SP302819
encoded stream supplied from the multiview video encoding
apparatus 104 in a format corresponding to the AVC format or
MVC format, and obtains 2D images 11 for N views, depth
images 12 for N views, and residual information. The
multiview video decoding apparatus 105 supplies the 2D
images 11 for N views and depth images 12 for N views to the
view synthesizing apparatus 15, and supplies the residual
information to a residual information compensating apparatus
106.
[0036]
The residual information compensating apparatus 106
adds the residual information supplied from the multiview
video decoding apparatus 105 into synthetic-view images for
(M-N) views generated by the view synthesizing apparatus 15,
and compensates for the missing information in the
synthetic-view images for (M-N) views. The residual
information compensating apparatus 106 outputs the
compensated synthetic-view images for (M-N) views and the 2D
images 11 for N views supplied from the view synthesizing
apparatus 15 as reconstructed 2D images 107 for M views.
The reconstructed 2D images 107 for M views are used to
display a stereoscopic image, for example, and a user is
able to view the stereoscopic image without using glasses.
[0037]
[Description of processing by system]
- 12 -
SP302819
Fig. 6 is a flowchart illustrating processing by the
system in Fig. 5.
[0038]
In step Sil of Fig. 6, the view synthesizing apparatus
102 uses 2D images 11 for N views and depth images 12 for N
views to conduct view synthesis for (M-N) views and generate
synthetic-view images for (M-N) views. The view
synthesizing apparatus 102 then supplies the synthetic-view
images for (M-N) views to the residual information acquiring
apparatus 103.
[0039]
In step S12, the residual information acquiring
apparatus 103 calculates residual information between the
synthetic-view images for (M-N) views supplied from the view
synthesizing apparatus 102 and 2D images 101 for (M-N) views
at the view synthesis positions. The residual information
acquiring apparatus 103 supplies the residual information to
a multiview video encoding apparatus 104.
[0040]
In step S13, the multiview video encoding apparatus 104
encodes the 2D images 11 for N views, the depth images 12
for N views, and the residual information supplied from the
residual information acquiring apparatus 103 in an AVC
format or MVC format. Then, the multiview video encoding
apparatus 104 supplies the encoded stream obtained as a
- 13 -
SP302819
result to the multiview video decoding apparatus 105.
[0041]
In step S14, the multiview video decoding apparatus 105
decodes the encoded stream in a format corresponding to the
AVC format or MVC format, the encoded stream being the
encoded 2D images 11 for N views, depth images 12 for N
views, and residual information supplied from the multiview
video encoding apparatus 104. The multiview video decoding
apparatus 105 then supplies the 2D images 11 for N views,
depth images 12 for N views, and residual information
obtained as a result to the view synthesizing apparatus 15,
and supplies the residual information to the residual
information compensating apparatus 106.
[0042]
In step S15, the view synthesizing apparatus 15 uses
the 2D images 11 for N views and depth images 12 for N views
supplied from the multiview video decoding apparatus 105 to
conduct view synthesis for (M-N) views and generate
synthetic-view images for (M-N) views. The view
synthesizing apparatus 102 then supplies the synthetic-view
images for (M-N) views and the 2D images 11 for N views to
the residual information acquiring apparatus 103.
[0043]
In step S16, the residual information compensating
apparatus 106 adds the residual information supplied from
- 14 -
SP302819
the multiview video decoding apparatus 105 to the syntheticview
images for (M-N) views generated by the view
synthesizing apparatus 15, and compensates for the missing
information in the synthetic-view images for (M-N) views.
[0044]
In step S16, the residual information compensating
apparatus 106 outputs the compensated synthetic-view images
for (M-N) views and the 2D images 11 for N views supplied
from the view synthesizing apparatus 105 as reconstructed 2D
images 107 for M views. The process then ends.
[0045]
Although the 2D images 11 for N views, the depth images
12 for N views, and the residual information are all encoded
in the foregoing description, information other than the 2D
images 11 for N views may also not be encoded.
[0046]
Additionally, it may be configured such that the
multiview video encoding apparatus 104 also includes
residual presence information indicating whether or not
residual information exists for each synthetic-view image,
and transmits this information to the multiview video
decoding apparatus 105 together with the 2D images 11 for N
views, the depth images 12 for N views, and the residual
information.
[0047]
- 15 -
SP302819
Furthermore, it may also be configured such that the
residual information transmitted to the multiview video
decoding apparatus 105 together with the 2D images 11 for N
views and the depth images 12 for N views is only residual
information with respect to synthetic-view images at view
synthesis positions farther outward than the views of the 2D
images 11 for N views (in the example in Fig. 3, the
synthetic-view images 2 for the two views left of the input
image 1 for the view on the left side, and the syntheticview
images 2 for the two views right of the input image 1
for the view on the right side). Similarly, the transmitted
residual information may only be residual information with
respect to synthetic-view images at view synthesis positions
farther inward than the views of the 2D images 11 for N
views (in the example in Fig. 3, the synthetic-view images 2
for the two views in between the views of the input images 1
for two views).
[0048]
Note that in this specification, the term "system"
represents the totality of an apparatus composed of a
plurality of apparatus.
[0049]
In addition, embodiments of the present invention are
not limited to the foregoing embodiments, and various
modifications are possible within a scope that does not
- 16 -
SP302819
depart from the principal matter of the present invention.
Reference Signs List
[0050]
15 view synthesizing apparatus
104 multiview video encoding apparatus
105 multiview video decoding apparatus
106 residual information compensating apparatus
- 17 -
SP302819

CLAIMS
[Claim 1] (amended)
An image processing apparatus comprising:
receiving means that receives residual information,
which is the error between synthetic-view images generated
using reference 2D images and depth information, and 2D
images at the view synthesis positions of the synthetic-view
images;
encoding means that generates an encoded stream by
encoding the reference 2D images, and generates a residual
stream by encoding the residual information received by the
receiving means; and
transmitting means that transmits the residual stream
generated by the encoding means, the depth information, and
the encoded stream generated by the encoding means.
Claim )] ( amended)
The image processing apparatus according to Claim 1,
wherein
the encoding means generates a depth stream by encoding
the depth information, and
the transmitting means transmits the residual stream,
the depth stream generated by the encoding means , and the
- 18 -
encoded stream.
SP302819
[Claim _2 ]
The image processing apparatus according to Claim 1,
further comprising:
computing means that computes the residual information
by calculating the error between the synthetic-view images
and 2D images at the view synthesis positions of the
synthetic-view images.
[Claim i ]
The image processing apparatus according to Claim 1,
wherein
the number of views in the reference 2D images is N,
and
the number of views in the synthetic-view images is the
value obtained by subtracting N from M, where M is greater
than N.
[Claim 5]
The image processing apparatus according to Claim4,,
wherein
the number of views in the reference 2D images is 2,
and
the number of views in the synthetic-view images is 6.
- 19 -
SP302819
[Claim 6-]
The image processing apparatus according to Claim 1,
wherein
the receiving means also receives residual presence
information indicating whether or not residual information
exists for the synthetic-view images, and
the transmitting means also transmits the residual
presence information received by the receiving means.
[Claim '7]
The image processing apparatus according to Claim 1,
wherein
the residual information is the error between outer
synthetic-view images and 2D images at the view synthesis
positions of the outer synthetic-view images, the outer
synthetic-view images being the synthetic-view images at
view synthesis positions farther outward than the views of
the reference 2D images.
[Claim 8]
The image processing apparatus according to Claim 7y,
wherein
the receiving means receives outer residual presence
information indicating whether or not error exists between
- 20 -
SP302819
the outer synthetic-view images and 2D images at the view
synthesis positions of the outer synthetic-view images, and
the transmitting means also transmits the outer
residual presence information received by the receiving
means.
[Claim q ]
The image processing apparatus according to Claim 1,
wherein
the residual information is the error between inner
synthetic-view images and 2D images at the view synthesis
positions of the inner synthetic-view images, the inner
synthetic-view images being the synthetic-view images at
view synthesis positions farther inward than the views of
the reference 2D images.
[Claim 10] (amended)
An image processing method comprising:
a receiving step in which an image processing apparatus
receives residual information, which is the error between
synthetic-view images generated using reference 2D images
and depth information, and 2D images at the view synthesis
positions of the synthetic-view images;
an encoding step in which the image processing
apparatus generates an encoded stream by encoding the
- 21 -
SP302819
reference 2D images, and generates a residual stream by
encoding the residual information received in the receiving
step; and
a transmitting step in which the image processing
apparatus transmits the residual stream generated in the
encoding step, the depth information, and the encoded stream
generated in the encoding step.
[Claim 1[] (amended)
An image processing apparatus comprising:
receiving means that receives a residual stream and
depth information, the residual stream being an encoded
stream of residual information which is the error between
synthetic-view images generated using reference 2D images
and the depth information, and 2D images at the view
synthesis positions of the synthetic-view images;
decoding means that decodes the residual stream and an
encoded stream obtained as a result of encoding the
reference 2D images;
generating means that generates the synthetic-view
images using the reference 2D images decoded by the decoding
means and the depth information received by the receiving
means; and
residual information compensating means that adds the
residual information decoded by the decoding means into the
- 22 -
SP302819
synthetic-view images generated by the generating means.
[Claim l ] (amended)
An image processing meLi,od comprising:
a receiving step in which an image processing apparatus
receives a. residual stream and depth information, the
residual stream being an encoded stream of residual
information which is the error between synthetic-view images
generated using reference 2D images and the depth
information, and 2D images at the view synthesis positions
of the synthetic-view images;
a decoding step in which the image processing apparatus
decodes the residual stream and an encoded stream obtained
as a result of encoding the reference 2D images;
a generating step in which the image processing
apparatus generates the synthetic-view images using the
reference 2D images decoded in the decoding step and the
depth information received in the receiving step; and
aresidual information compensating step in which the
image processing apparatus adds the residual information
decoded in the decoding step into the synthetic-view images
generated in the generating step.