DATA STRUCTURE, IMAGE PROCESSING APPARATUS, IMAGE PROCESSING
METHOD, AND PROGRAM
5 TECHNICAL FIELD
[0001]
The present invention relates to data structures, image
processing apparatuses, image processing methods, and
programs. Particularly, the present invention relates to a
10 data structure, an image processing apparatus, an image
processing method, and a program for enabling appropriate
processing of video data of 3D content.
BACKGROUND ART
15 [0002]
Although 2D images are the mainstream of content such
as movies, attention is recently being drawn to 3D images,
and various techniques have been suggested for 3D image display
apparatuses and 3D image encoding and decoding methods (see
20 Patent Documents 1 and 2, for example).
[0003]
A 3D image is formed with an image for the left eye to
be viewed with the left eye and an image for the right eye
to be viewed with the right eye. As predetermined parallaxes
25 are set for the image for the left eye and the image for the
right eye, viewers see the image in three dimensions.
CITATION LIST
PATENT DOCUMENTS
30 [0004]
Patent Document la Japanese Patent Application Laid-Open No.
SP262995WO00
10-327430
Patent Document 2: Japanese Patent Application Laid-Open No,
2008-182669
SUMMARY OF THE INVENTION
PROBLEMS TO BE SOLVED BY THE INVENTION
[0005]
The information (parallax information) about the
maximum value and the minimum value of parallaxes set for the
10 image for the left eye and the image for the right eye of a
3D image can be detected from the provided images for the left
eye and the right eye, but it is also possible to provide the
information as metadata (additional information) for the 3D
content. In that case, the information about the maximum and
15 minimum values of parallaxes varies with image size.
Therefore, appropriate processing cannot be performed, unless
to what image size the parallax information corresponds is
accurately recognized by the 3D content receiving side.
[0006]
20 The present invention has been made in view of the above
circumstances, and the object thereof is to enable appropriate
processing of video data of 3D content.
SOLUTIONS TO PROBLEMS
25 [0007]
A data structure according to a first aspect of the
present invention includes: image data of a 3D image; and a
reference image size that is a predetermined image size to
be a reference for the 3D image, and maximum and minimum
30 parallaxes for the reference image size.
[0008]
2
SP262995W000
In the first aspect of the present invention, a data
structure includes image data of a 3D image, a reference image
size that is apredetermined image size serving as the reference
for the 3D image, and the maximum and minimum parallaxes for
the reference image size.
[0009]
An image processing apparatus according to a second
aspect of the present invention includes: an obtaining unit
that obtains image data of a 3D image, and content data
10 containing a reference image size that is a predetermined image
size to be a reference for the 3D image and maximum and minimum
parallaxes for the reference image size; a detecting unit that
detects an image size corresponding to the image data of the
3D image obtained by the obtaining unit; a parallax calculating
15 unit that calculates maximum and minimum parallaxes
corresponding to the image size of the obtained image data,
when the detected image size of the image data is not the same
as the reference image size; and a processing unit that performs
predetermined image processing on the image data obtained by
20 the obtaining unit, based on the calculated maximum and minimum
parallaxes.
[0010]
An image processingmethod according to the second aspect
of the present invention includes the steps of: obtaining image
25 data of a 3D image, and content data containing a reference
image size that is a predetermined image size to be a reference
for the 3D image and maximum and minimum parallaxes for the
reference image size; detecting an image size corresponding
to the obtained image data of the 3D image; calculating maximum
30 and minimum parallaxes corresponding to the image size of the
obtained image data, when the detected image size of the image
3
SP262995WO00
data is not the same as the reference image size; and performing
predetermined image processing on the obtained image data,
based on the calculated maximum and minimum parallaxes.
[0011]
5 A program according to the second aspect of the present
invention causes a computer to perform an operation including
the steps of obtaining image data of a 3D image, and content
data containing a reference image size that is a predetermined
image size serving as the reference for the 3D image and the
10 maximum and minimum parallaxes for the reference image size;
detecting the image size corresponding to the obtained image
data of the 3D image; calculating the maximum and minimum
parallaxes corresponding to the image size of the obtained
image data, when the detected image size of the image data
15 is not the same as the reference image size; and performing
predetermined image processing on the obtained image data,
based on the calculated maximum and minimum parallaxes.
[0012]
In the second aspect of the present invention, image
20 data of a 3D image, and content data containing a reference
image size that is a predetermined image size serving as. the
reference for the 3D image and the maximum and minimum
parallaxes for the reference image size are obtained. The
image size corresponding to the obtained image data of the
25 3D image is then detected. If the detected image size of the
image data is not the same as the reference image size, the
maximum and minimum parallaxes corresponding to the image size
of the obtained image data are calculated. Based on the
calculated maximum and minimum parallaxes, predetermined
30 image processing is performed on the obtained image data.
[0013]
4
SP262995WO00
A data structure according to a. third aspect of the
present invention includes: image data of a 3D image; and
maximum and minimum parallaxes for an image size corresponding
to the image data of the 3D image.
[0014]
In the third aspect of the present invention, a data
structure includes image data of a 3D image, and the maximum
and minimum parallaxes for the image size corresponding to
the image data of the 3D image.
10 [0015]
An image processing apparatus according to a fourth
aspect of the present invention includes: an obtaining unit
that obtains image data of a 3D image, and content data
containing maximum and minimum parallaxes for an image size
15 corresponding to the image data of the 3D image; an
enlarging/reducing unit that enlarges or reduces the image
size corresponding to image data of the 3D image obtained by
the obtaining unit, the image size being enlarged or reduced
at a predetermined enlargement or reduction ratio; a
20 calculating unit that calculates maximum and minimum
parallaxes for the image data of the enlarged or reduced image
size; and an output unit that outputs the maximum and minimum
parallaxes updated to the calculation results, together with
the image data subjected to the enlargement or reduction.
25 [0016]
An image processing method according to the fourth aspect
of the present invention includes the steps of: obtaining image
data of a 3D image, and content data containing the maximum
and minimum parallaxes for the image size corresponding to
30 the image data of the 3D image; enlarging or reducing the image
size corresponding to the obtained image data of the 3D image
S
SP262995W000
at a predetermined enlargement or reduction ratio; calculating
the maximum and minimum parallaxes for the image data of the
enlarged or reduced image size; and outputting the maximum
and minimum parallaxes updated to the calculation results,
5 together with the image data subjected to the enlargement or
reduction.
[0017]
A program according to the fourth aspect of the present
invention causes a computer to perform an operation including
10 the steps of: obtaining image data of a 3D image, and content
data containing the maximum and minimum parallaxes for the
image size corresponding to the image data of the 3D image;
enlarging or reducing the image size corresponding to the
obtained image data of the 3D image at a predetermined
15 enlargement or reduction ratio; calculating the maximum and
minimumparallaxes for the image data of the enlarged or reduced
image size; and outputting the maximum and minimum parallaxes
updated to the calculation results, together with the image
data subjected to the enlargement or reduction.
20 [0018]
In the fourth aspect of the present invention, image
data of a 3D image, and content data containing the maximum
and minimum parallaxes for the image size corresponding to
the image data of the 3D image are obtained. The image size
25 corresponding to the obtained image data of the 3D image is
then enlarged or reduced at a predetermined enlargement or
reduction ratio. The maximum and minimum parallaxes for the
image data of the enlarged or reduced image size are calculated.
The maximum and minimum parallaxes updated to the calculation
30 results, together with the image data subjected to the
enlargement or reduction, are then output.
6
SP262995WO00
[0019]
Each of the above programs to be provided may be
transmitted via a transmission medium, or may be recorded on
a recording medium.
[0020]
Each of the above image processing apparatuses may be
an independent apparatus, or may be an internal block of an
apparatus.
10 EFFECTS OF THE INVENTION
[0021]
According to the first through fourth aspects of the
present invention, video data of 3D content can be
appropriately processed.
15
BRIEF DESCRIPTION OF DRAWINGS
[0022]
Fige 1 is a block diagram showing an example structure
of an embodiment of a recording apparatus to which the present
20 invention is applied.
Fig. 2 is a flowchart for explaining a recording
operation.performed by the recording apparatus of Fig. 1e
Fig. 3 is a diagram showing the hierarchical structure
of data recorded on a recording mediums
25 Fig. 4 is a diagram showing an example of an extended
region of a MPEG4 box.
Fig. 5 is a diagram showing another example of an extended
region of a MPEG4 box.
Fig, 6 is a diagram showing yet another example of an
30 extended region of a MPEG4 box.
Fig. 7 is a block diagram showing an example structure
7
SP262995WO00
of an embodiment of a reproducing apparatus to which the present
invention is applied.
Fig. 8 is a block diagram showing an example structure
of the 3D image processing unit in detail.
5 Fig. 9 is a flowchart for explaining a reproducing
operation performed by the reproducing apparatus of Fig. 7.
Fig. 10 is a block diagram showing an example structure
of an embodiment of a data conversion apparatus to which the
present invention is applied.
10 Fig. 11 is a block diagram showing an example structure
of an embodiment of a computer to which the present invention
is applied.
MODE FOR CARRYING OUT THE INVENTION
15 [0023]

[Example Structure of Recording Apparatus]
Fig. 1 is a block diagram of an example structure of
an embodiment of a recording apparatus to which the present
20 invention is applied.
[0024]
The recording apparatus 10 of Fig. 1 includes a video
encoder 11, an audio encoder 12, a multiplexing unit 13, and
a recording control unit 14.
25 [0025]
The recording apparatus 10 encodes content data of 3D
content, and records the encoded data on a recording medium
20 such as a BDROM (Blu-Ray (a registered trade name) Disc
Read Only Memory) The content data contains image data of
30 3D images (hereinafter referred to as 3D video data) and audio
data corresponding to the image data. The 3D video data is
9
SP262995WO00
formed with image data of images for the left eye and image
data of images for the right' eye. The content data also
contains additional information that is the information about
parallaxes set in the images for the left eye and the images
for the right eye.
[0026]
The video encoder 11 of the recording apparatus 10
encodes 3D video data input from the outside by an encoding
method such as MPEG2 (Moving Picture Experts Group phase 2),
10 MPEG4, or AVC (Advanced Video Coding) The video encoder 11
supplies a video stream that is the ES (Elementary Stream)
obtained as the result of the encoding, to the multiplexing
unit 13.
[0027]
15 The audio encoder 12 encodes the audio data corresponding
to the 3D video data input from the outside by an encoding
method such as MPEG. The audio encoder 12 then supplies an
audio atreamthat is the ES obtained as the result of the encoding
to the multiplexing unit 13.
20 [0028]
The multiplexing unit 13 multiplexes the video stream
supplied from the video encoder 11 and the audio stream supplied
from the audio encoder 12, and supplies the multiplexed stream
obtained as the result of the multiplexing to the recording
25 control unit 14.
[0029]
The recording control unit 14 records the multiplexed
stream supplied from the multiplexing unit 13 on the recording
medium 20. The recording control unit 14 also records a
30 definition file on the recording medium 20. The definition
file contains a predetermined image size to be a reference
9
SP262995W000
for. 3D images to be recorded on the recording medium 20
(hereinafter referred to as the reference image size), and
the maximum parallax value (the maximum parallax) and the
minimum parallax value (the minimum parallax) of an image
5 having the above image size. In the following, the maximum
parallax value (the maximum parallax) and the minimum parallax
value (the minimum parallax) will also be referred as the
maximum/minimum parallaxes.
[0030]
10 Here, the image size of 3D video data to be recorded
on the recording medium 20 and the reference image size are
substantially the same, but may not be exactly the same. That
is, the maximum/minimum parallaxes in the additional
information are the maximum parallax value and the minimum
15 parallax value of an image having the reference image size.
Therefore, when the image size of 3D video data to be recorded
on the recording medium 20 is not the same as the reference
image size, the maximum/minimum parallaxes of the 3D video
data differs from the maximum/minimum parallaxes recorded as
20 the definition file,
[0031]
In a case where a reference image size of "720 x 480,"
a maximum parallax of "+72, " and a minimum parallax of "-48, "
are recorded as the definition file, for example, the maximum
25 parallax and the minimum parallax are "+72" and "-48,"
respectively, when the reference image size is "720 x 480."
In this case, the image size of the 3D video data to be recorded
on the recording medium 20 may be "1920 x 1080."
[0032]
30 The maximum/minimum parallaxes and the reference image
size are input from an operation input unit (not shown), and
10
SP262995WO00
are supplied to the recording control unit 14.
[0033]
In the recording apparatus 10 having the above described
structure, the "reference image size and maximum/minimum
5 parallaxes" are recorded as additional information (metadata)
on the recording medium 20, and, accordingly, proper processing
using the "maximum/minimum parallaxes" can be performed at
the time of reproduction.
[0034]
10 [Description of Operation of the Recording Apparatus]
Fig. 2 is a flowchart for explaining a recording
operation to be performed by the recording apparatus 10 of
Fig. 1. This recording operation is started when additional
information, 3D video data, and audio data are input, for
15 example.
[0035]
In step 510, the recording control unit 14 records the
"reference image size and maximum/minimum parallaxes" that
are the additional information input from the outside are
20 recorded as the definition file on the recording medium 20.
[0036]
In step 511, the video encoder 11 encodes the 3D video
data input from the outside by an encoding method such as MPEG2,
MPEG4, or AVC. The video encoder 11 then supplies the video
25 stream obtained as the result of the encoding to the
multiplexing unit 13.
[0037]
In step S12, the audio encoder 12 encodes the audio data
corresponding to the 3D video data input from the outside by
30 an encoding method such as MPEG, and supplies the audio stream
obtained as the result of the encoding to the multiplexing
11
SP262995W000
unit 13.
[0038]
In step 513, the multiplexing unit 13 multiplexes the
video stream from the video encoder 11 and the audio stream
from the audio encoder 12, and supplies the multiplexed stream
obtained as the result of the multiplexing to the recording
control unit 14.
[0039]
In step 5.14, the recording control unit 14 records the
10 multiplexed stream supplied from the multiplexing unit 13 on
the recording medium 20, and ends the operation.
[0040]
[Data Structure of Additional Information]
Fig. 3 is a diagram showing the hierarchical structure
15 of data to be recorded on the recording medium 20.
[0041]
As shown in Fig. 3, the hierarchy of data to be recorded
on the recording medium 20 is formed with a layer C of ESs
such as the audio stream and the video stream, a layer B as
20 the system layer of the multiplexed stream, a layer A of
information unique to the recording medium 20, and the like.
[0042]
In the above described example, the "reference image
size and maximum/minimum parallaxes" as the additional
25 information are recorded as the definition file unique to the
recording medium 20, or are recorded in the layer A. However,
the "reference image size and maximum/minimum parallaxes" can
be recorded in the layer B or the layer C.
[0043]
30 In a case where the additional information is recorded
in the layer C, for example, the additional information is
12
SP262995WO00
recorded as SEI (Supplemental Enhancement Information), or
part of a SPS (Sequence Parameter Set) or a PPS (Picture
Parameter Set), if the encoding method is AVC. If the encoding
method is MPEG2, the additional information is recorded as
a video sequence or extension-and-user-data.
[00441
In this case, the additional information can be made
variable in one video stream. The "reference image size and
maximum/minimum parallaxes" can be changed for each video
10 stream.
[0045]
In a case where the additional information is recorded
in the layer B, the additional information is recorded in a
private packet of a TS ( Transport Stream ) , a private pack of
15 a PS ( Program Stream), an extended region of a box contained
in MPEG4 configuration ( Config ) information , or the like.
[0046]
The extended region of the MPEG4 box in which the
additional information is recorded is located in the Private
20 Extension box (uuid in Fig. 4 ) immediately behind the ftyp
box located at the top of the file as shown in Fig. A, for
example. In this case , the reproducing apparatus that
reproduces the 3D video data recorded on the recording medium
20 can obtain the "reference image size and maximum/minimum
25 parallaxes " prior to a decoding operation. The "reference
image size and maximum/minimum parallaxes" are invariable in
the file.
[0047]
Other than the "reference image size andmaximum/minimum
30 parallaxes," information indicating the type of codec, the
bit rate, the frame size, the aspect ratio, and whether the
13
52262995WO00
images are 2D images or 3D images is written in the Private
Extension box.
[0048]
The extension region of the MPEG4 box in which the
5 additional information is recorded may be provided in a region
(stsd in Fig. 5) in track information (trek) in a moov box,
as shown in Fig. 5. In this case, the video stream contains
information indicating the region of the additional
information to be referred to, and, based on the information,
10 the reproducing apparatus obtains the "reference image size
and maximum/minimum parallaxes." Accordingly, the
"reference image size and maximum/minimum parallaxes" can be
changed in the video stream. However, the accessibility is
poorer than in the case of Fig. 4.
15 [0049]
Further, the extended region of the MPEG4 box in which
the additional information is recorded may be provided in a
mdat box, as shown in Fig. 6. That is, the additional
information may be recorded as a media stream (side
20 info.stream). In this case, the video stream and the
additional information are in synchronization with each other
through time information. Accordingly, the "reference image
size and maximum/minimum parallaxes" can be changed by the
second.
25 [0050]
In the examples illustrated in Figs. 4 through 6, the
moov box and the mdat box are located in order behind the ftyp
box; but the positions of the moov box and the mdat box are
not limited to them.
30 [0051]

[Example Structure of Reproducing Apparatus]
Fig. 7 is a block diagram showing an example structure
of an embodiment of a reproducing apparatus to which the present
invention is applied.
[0052]
The reproducing apparatus 50 of Fig. 7 includes a reading
unit 51, a dividing unit 52, a video decoder 53, a 3D image
processing unit 54, and an audio decoder 55. The reproducing
10 apparatus 50 reproduces 3D video data recorded on a recording
medium 20 and the audio data corresponding to the 3D video
data, as well as additional information. Based on the
additional information, the reproducing apparatus 50 properly
displays 3D images.
15 [0053]
Specifically, the reading unit 51 of the reproducing
apparatus 50 reads the additional information containing the
"reference image size and maximum/minimum parallaxes"
recorded on the recording medium 20, and supplies the
20 additional information to the 3D image processing unit 54.
The reading unit 51 reads a multiplexed stream recorded on
the recording medium 20, and supplies the multiplexed stream
to the dividing unit 52.
[0054]
25 The dividing unit 52 divides the multiplexed stream
supplied from the reading unit 51 into a video stream and an
audio stream. The dividing unit 52 then supplies the video
stream to the video decoder 53, and supplies the audio stream
to the audio decoder 55.
30 [0055]
The video decoder 53 decodes the video stream supplied
15
SP262995WO00
from the dividing unit 52 by the method corresponding to the
encoding method used by the video encoder 11 of Fig. 1, and
supplies the 3D video data obtained as the result of the decoding
to the 3D image processing unit 54.
5 [0056]
The 3D image processing unit 54 performs predetermined
image processing on the 3D video data supplied from the video
decoder 53, where necessary. In this embodiment, the 3D image
processing unit 54 performs processing to adjust the depth
10 positions of captions to be displayed and superimposed on 3D
images. The 3D image processing unit 54 outputs the processed
3D video data to a display unit 61.
[0057]
The image area in which the captions are to be displayed
15 may be supplied from the outside, or may be independently
detected in the 3D image processing unit 54. To detect a
caption display area, the processing disclosed in JP
2008-166988 A (an operation to detect an area that does not
vary over a predetermined period of time as a caption area),
20 which was suggested by the applicant, can be used, for example.
[0058]
The audio decoder 55 decodes the audio stream supplied
from the dividing unit 52 by the method corresponding to the
encoding method used by the audio encoder 12 of Fig. 1, and
25 supplies the audio data obtained as the result of the decoding
to a speaker 62.
[0059]
The display unit 61 displays, in a time-divisional manner,
for example, the images for the left eye and the images for
30 the right eye corresponding to the video data supplied from
the 3D image processing unit 54. At this point, the viewer
16
SP262995W000
wears glasses with a shutter that is synchronized with the
switching between the images for the left eye and the images
for the right eye, to see the images for the left eye only
with the left eye, and see the images for the right eye only
with the right eye. By doing so, the viewer can see the 3D
images in three dimensions.
[0060]
The speaker 62 outputs sound corresponding to the audio
data supplied from the audio decoder 55.
10 [0061]
[Details of an Example Structure of the 3D Image Processing
Unit 54]
Fig. 8 is a block diagram showing an example structure
of the 3D image processing unit 54 of Fig. 7 in detail.
15 [0062]
The 3D image processing unit 54 includes an image size
detecting unit 71, amaximum/minimumparallax calculating unit
72, and a caption adjusting unit 73.
[0063]
20 The image size detecting unit 71 detects the image size
from the 3D video data supplied from the reading unit 51 The
image size detected here is the image size to be displayed
on the display unit 61, and therefore, will be hereinafter
referred to as the displayed image size. The displayed image
25 size can be recognized by counting signals representing the
validity period of the images, for example. The image size
detecting unit 71 supplies the displayed image size as the
detection result to the maximum/minimum parallax calculating
unit 72.
30 [0064]
The maximum/minimum parallax calculating unit 72
17
SP262995W000
obtains the "reference image size and maximum/minimum
parallaxes" supplied from the reading unit 51, and obtains
the displayed image size supplied from the image size detecting
unit 71.
[0065]
The maximum/minimum parallax calculating unit 72
compares the supplied displayed image size with the "reference
image size" in the additional information. If the displayed
image size and the reference image size are different sizes,
10 the maximum/minimum parallaxes for the displayed image size
are calculated.
[0066]
In a case where the "reference image size, maximum
parallax, and minimum parallax" are "720 x 480, +'72, and -48, "
15 respectively, and the "displayed image size" is "1920x1080,
for example, the maximum/minimum parallax calculating unit
72 calculates the maximum/minimum parallaxes for the displayed
image size in the following manner.
The maximum parallax = +72 x (7:920/720) = +192
20 The minimum parallax = -48 x (1920/720) = -128
[0067]
The maximum/minimum parallax calculating unit 72 then
supplies the maximum/minimum parallaxes for the displayed
image size as the calculation result to the caption adjusting
25 unit 73. In a case where the displayed image size and the
reference image size are the same, the "maximum/minimum
parallaxes" in the obtained additional information is supplied
directly as the maximum/minimum parallaxes for the displayed
image size to the caption adjusting unit 73.
30 [0068]
The caption adjus Ling unit 73 adjusts the depth positions
18
SP262995WO00
of the captions to be displayed and superimposed on the 3D
images, in accordance with the maximum/minimum parallaxesfor
the displayed image size. That is, the caption adjusting unit
73 adjusts the captions to be located slightly forward from
the depth position determined by the maximum parallax for the
displayed image size (or to be the closest to the viewer).
[0069]
[Description of Operation of the Reproducing Apparatus]
Fig. 9 is a flowchart for explaining a reproducing
10 operation to be performed by the reproducing apparatus 50.
This reproducing operation is started when a viewer requests
reproduction of 3D content recorded on the recording medium
20, for example.
[0070]
15 In step 531, the reading unit 51 reads the "reference
image size and maximum/minimum parallaxes" recorded as the
additional information on the recording medium 2 0, andsupplies
the "reference image size and maximum/minimum parallaxes" to
the 3D image processing unit 54.
20 [0071]
In step 232, the reproducing apparatus 50 readr^ and
decodes the multiplexed stream of the 3D content recorded on
the recording medium 20. That is, the reading unit 51 reads
the multiplexed stream of the 3D content from the recording
25 medium 20, and supplies the multiplexed stream to the dividing
unit 52. The dividing unit 52 divides the multiplexed stream
into a video stream and an audio stream. The video decoder
53 decodes the video stream by the method corresponding to
the encoding method used by the recording apparatus 10, and
30 supplies the 3D video data obtained as the result of the decoding
to the 3D image processing unit 54. The audio decoder 55
19
SP262995W000
decodes the audio stream by the method corresponding to the
encoding method used by the recording apparatus 10, and
supplies the audio data obtained as the result of the decoding
to the speaker 62.
[0072]
In step S33, the image size detecting unit 71 detects
the image size (the displayed image size) from the 3D video
data supplied from the reading unit 51, and supplies the image
size to the maximum/minimum parallax calculating unit 72. in
10 step S34, the maximum/minimum parallax calculating unit 72
determines whether the "displayed image size" detected by the
image size detecting unit 71 and the "reference image size"
supplied from the reading unit 51 are the same.
[0073]
15 If the "displayed image size" and the "reference image
size" are determinednot tobe the same instep 534, the operation
moves on to step S35. The maximum/minimum parallax
calculating unit 72 then calculates the maximum/minimum
parallaxes for the displayed image size, and supplies the
20 maximum/minimum parallaxes to the caption adjusting unit 73.
[0074]
If the "displayed image size" and the "reference image
size" are determined to be the same in step S34, on the other
hand, the operation moves on to step S36. In step S36, the
25 maximum/minimum parallax calculating unit 72 supplies the
"reference image size" in the additional information supplied
from the reading unit 51 directly as the maximum/minimum
parallaxes for the displayed image size to the caption
adjusting unit 73.
30 [0075]
In step 537, the caption adjusting unit 73 adjusts the
20
S2262995W000
depth positions of the captions to be displayed and
superimposed on the 3D images, in accordance with the
maximum/minimum parallaxes for the displayed image size. The
adjusted 3D video data is output to the display unit 61. In
step S37, the audio decoder 55 also outputs the audio data
corresponding to the 3D video data. The operation then comes
to an end.
[0076]
As described above, inaddition to the multiplexed stream
10 of 3D content, the "reference image size and maximum/minimum
parallaxes" of the 3D content is recorded as the additional
information on the recording medium 20. The reproducing
apparatus 50 reads the additional information, and compares
the additional information with the image size of the 3D video
15 data obtained by decoding, to readily recognize the precise
maximum/minimum parallaxes of the read 3D video data. Based
on the precise maximum/minimum parallaxes of the 3D video data,
predetermined signal processing can be properly performed.
[0077]
20 In this embodiment, the depth positions of captions are
adjusted in the predetermined signal processing. However,
the processing based on the maximum/minimum parallaxes is not
limited to that.
[0078]
25 In the above described example, the multiplexed stream
and additional information (metadata) of 3D content are
provided from the content provider side to the content viewer
side via the recording medium 20. However, there are cases
where 3D content is provided by transmission via a network
30 such as a satellite broadcasting network, a cable television
network, or the Internet. Also, there is a. possibility that
21
SP262995WO00
the image size of 3D content transmitted from the content
provider side is enlarged or reduced in the transmission path
before the content viewer receives the 3D content. In that
case, the image size of the transmitted 3D video data differs
from the "reference image size" transmitted as the additional
information. Even in such a case, the maximum/minimum
parallaxes for the image size of the received 3D video data
can be promptly and precisely recognized from the "reference
image size and maximum/minimum parallaxes" in the additional
10 information, and image processing based on the precise
maximum/minimum parallaxes can be properly performed.
[0079]

In the above described example, the "reference image
15 size and maximum/minimum parallaxes" to be recorded or
transmitted as the additional information is fixed even in
a case where the image size of 3D content is enlarged or reduced
However, the "maximum /minimum parallaxes" in the additional
information may be updated as the image size is enlarged or
20 reduced. In such a case, the "reference image size" is always
the same as the image size of 3D video data to be recorded
or transmitted, and therefore, can be omitted.
[0080]
[Example Structure of Data Conversion Apparatus]
25 Fig a 10 is a functional block diagram of a data conversion
apparatus (an image processing apparatus) that converts the
image size of 3D video data, and updates the information about
the maximum/minimum parallaxes. In Fig. 10, explanation of
audio data will be omitted.
30 [0081]
The data conversion apparatus 80 of Fig. 10 includes
22
SP262995WO00
an obtaining unit 81, an image enlarging/reducing unit 82,
a maximum/minimum parallax updating unit 83, and an output
unit 84.
[0082]
5 The obtaining unit 81 obtains 3D video data input from
the outside and the "maximum/minimum parallaxes" as the
additional information. The obtaining unit 81 supplies the
3D video data to the image enlarging/reducing unit 82, and
supplies the "maximum/minimum parallaxes" to the
10 maximum/minimum parallax updating unit 83.
[0083]
The image enlarging/reducing unit 82 performs
processing to enlarge or reduce the image size of the supplied
3D video data at an enlargement or reduction ratio that is
15 input and supplied from an operation input unit or the like
(not shown) . The image enlarging/reducing unit 82 supplies
the processed 3D video data to the output unit 84.
[0084]
Based on the enlargement or reduction ratio supplied
20 from the operation input unit or the like, the maximum/minimum
parallax updating unit 83 updates the "maximum/minimum
parallaxes" supplied from the obtaining unit 81 to the
"maximum/minimum parallaxes" corresponding to the processed
images subjected to the enlarging or reducing operation by
25 the image enlarging /reducing unit 82. The maximum/minimum
parallax updating unit 83 then supplies the updated
"maximum/minimum parallaxes" to the output unit 84.
[0085]
The output unit 84 outputs the 3D video data supplied
30 from the image enlarging/reducing unit 82 and the
"maximum/minimum parallaxes" as the additional information,
23
SP262995WO00
in a multiplexing or time-divisional manner or the like.
[0086]
The enlargement or reduction ratio may not be input
through the operation input unit or the like, but may be a
5 predetermined value.
[0087]
Operations of the data conversion apparatus 80 are now
described through a specific example.
[0088]
10 First, the obtaining unit 81 obtains 3D video data of
an image size of "720 x 480," and additional information in
which "+72/-48" is written as the "maximum/minimum
parallaxes," for example. The 3D video data obtained by the
obtaining unit 81 is supplied to the image enlarging/reducing
15 unit 82, and the additional information is supplied to the
maximum/minimum parallax updating unit 83.
[0089]
The enlargement or reduction ratio that is input to the
operation input unit or the like by a predetermined user is
20 supplied to the image enlarging/ reducing unit 82. Here,
"1920/720" is supplied, for example.
[0090]
Based on the supplied enlargement or reduction ratio,
the image enlarging/reducing unit 82 performs processing to
25 enlarge the image size of the 3D video data from "720 x 480"
to "1920 x 1080," and supplies the processed 3D video data
to the output unit 84.
[0091]
The maximum/reduction parallax updating unit 83 updates
30 the "maximum/minimum parallaxes" in the additional
information at the enlargement or reduction ratio supplied
24
SP262995WO00
from the operation input unit or the like, or at "1920/720."
Specifically, the maximum/reduction parallax updating unit
83 performs the following calculations:
The maximum parallax = +72 x (1920/720) = +192
5 The minimum parallax = -48 x (1920/720) = -128
In this manner, "+192/-128" is obtained as the updated
"maximum/minimum parallaxes." The maximum/reduction
parallax updating unit 83 then supplies the updated
"maximum/minimum parallaxes" = "+192/-128" to the output unit
10 84.
[0092]
The processing by the image enlarging/reducing unit 82
and the processing by the maximum/reduction parallax updating
unit 83 can be performed in parallel.
15 [0093]
The output unit 84 outputs the 3D video data that is
supplied from the image enlarging/reducing unit 82 and has
the image size of "1920 x 1080, " and the additional information
in which "+192/-128" is written as the "maximum/minimum
20 parallaxes."
[0094]
This data conversion apparatus 80 may be located in a
stage before the output unit of the content provider side,
in the middle of a transmission path, or in a stage after the
25 input unit of the content obtaining side.
[0095]
[Description of Computer to Which the Invention is Applied]
The above described series of operations canbe performed
by either hardware or software. In a case where the series
30 of operations are performed by software, the program forming
the software is installed in a general-purpose computer or
25
SP262995WO00
the like.
[0096]
Fig. 11 shows an example structure of an embodiment of
a computer in which the program for performing the above
5 described series of operations is installed.
[0097]
The program can be recorded beforehand in a storage unit
208 as a recording medium provided in the computer, or in a
ROM (Read Only Memory) 202.
10 [0098]
Alternatively, the program can be stored (recorded) in
a removable medium 211. This removable medium 211 can be
provided as so-called packaged software. Here, the removable
medium 211 may be a CD-ROM (Compact Disc Read Only Memory),
15 a MO (Magneto Optical) disc, a DVD (Digital Versatile Disc),
a magnetic disc, a semiconductor memory, or the like.
[0099]
The program is installed from the above described
removable medium 211 into the computer via a drive 210.
20 Alternatively, the program can be downloaded into the computer
via a communication network or a broadcasting network., and
be installed into the built-in storage unit 208. That is,
the program can be received by a communication unit 209 via
a wired or wireless transmission medium, and be installed into
25 the storage unit 208.
[0100]
The computer includes a CPU (Central Processing Unit)
201, and an input/output interface 205 is connected to the
CPU 201 via a bus 204.
30 [0101]
When a user inputs an instruction by operating an input
26
SP262995W000
unit 206 or the like via the input/output interface 205, the
CPU 201 executes the program stored in the ROM 202 in accordance
with the instruction. Alternatively, the CPU 201 loads the
program stored in the storage unit 208 into a RAM (RandomAccess
5 Memory) 203, and executes the program.
[0102]
In this manner, the CPU 201 performs the operations
according to the above described flowcharts, or performs
operations with the structures illustrated in the above
10 described block diagrams. Where necessary, the CPU 201 then
outputs the operation results from an output unit 207, or
transmits the operation results from the communication unit
209, or further records the operation results into the storage
unit 208, via the input/output interface 205, for example.
15 [0103]
The input unit 206 is formed with a keyboard , a mouse,
a microphone , and the like. The output unit 207 is formed
with a LCD (Liquid Crystal Display ), a speaker , and the like.
[0104]
20 In this specification , the operations to be performed
by the computer according to the program are not neces :..nrily
performed in chronological order according to the sequences
shown in the flowcharts . That is, the operations to be
performed by the computer according to the program include
25 operations to be performed in parallel or independently of
one another ( such as parallel processing or object.-based
processing),
[0105]
The program may be executed by a single computer
30 (processor), or maybe executed through distributed processing
by more than one computer. Further, the program may be
27
S1262995WO00
transferred to a remote computer, and be executed therein.
[0106]
In the above described embodiments, each 3D image is
a two-viewpoint 3D image with two viewpoints. However, those
embodiments can also be applied to multi-viewpoint 3D images
with three or more viewpoints.
[0107]
Embodiments of the present invention are not limited
to the above described embodiments, and various changes may
10 be made to them without departing from the scope of the
invention.
REFERENCE SIGNS LIST
[0108]
15 20 Recording medium
50 Reproducing apparatus
51 Reading unit
54 3D image processing unit
71 Image size detecting unit
20 72 Maximum/minimum parallax calculating unit
73 Caption adjusting unit
80 Data conversion apparatus
81 Obtaining unit
82 Image enlarging/reducing unit
25 83 Maximum/minimum parallax updating unit
84 Output unit
28
SP262995W000
CLAIMS
1. A data structure comprising:
image data of a 3D image; and
a reference image size that is a predetermined image
size to be a reference for the 3D image, and maximum and minimum
parallaxes for the reference image size.
2. An image processing apparatus comprising:
10 an obtaining unit that obtains image data of a 3D image,
and content data containing a reference image size that is
a predetermined image size to be a reference for the 3D image
and maximum and minimum parallaxes for the reference image
size;
15 a detecting unit that detects an image size corresponding
to the image data of the 3D image obtained by the obtaining
unit;
a parallax calculating unit that calculates maximum and
minimum parallaxes corresponding to the image size of the
20 obtained image data, when the detected image size of the image
data is not the same as the reference image size; and
a processing unit that performs predetermined image
processing on the image data obtained by the obtaining unit,
based on the calculated maximum and minimum parallaxes.
25
3. The image processing apparatus according to claim 2,
wherein the parallax calculating unit calculates the maximum
and minimum parallaxes corresponding to the image size of the
image data by multiplying the maximum and minimum parallaxes
30 for the obtained reference image size by a ratio between the
detected image size of the image data and the reference image
29
SP262995W000
size.
4o An image processing method comprising the steps of:
obtaining image data of a 3D image, and content data
5 containing a reference image size that is a predetermined image
size to be a reference for the 3D image and maximum and minimum
parallaxes for the reference image size;
detecting an image size corresponding to the obtained
image data of the 3D image;
10 calculating maximum and minimum parallaxes
corresponding to the image size of the obtained image data,
when the detected image size of the image data is not the same
as the reference image size; and
performing predetermined image processing on the
15 obtained image data, based on the calculated maximum and
minimum parallaxes.
5 e A program for causing a computer toper form an operation
comprising the steps of:
20 obtaining image data of a 3D image, and content data
containing a reference image size that is a predetermined image
size to be a reference for the 3D image and maximum and minimum
parallaxes for the reference image size;
detecting an image size corresponding to the obtained
25 image data of the 3D image;
calculating maximum and minimum parallaxes
corresponding to the image size of the obtained image data,
when the detected image size of the image data is not the same
as the reference image size; and
30 performing predetermined image processing on the
obtained image data, based on the calculated maximum and
30
SP262995W000
minimum parallaxes.
6. A data structure comprising:
image data of a 3D image; and
maximum and minimum parallaxes for an image size
corresponding to the image data of the 3D image.
7. An image processing apparatus comprising:
an obtaining unit that obtains image data of a 3D image,
10 and content data containing maximum and minimum parallaxes
for an image size corresponding to the image data of the 3D
image;
an enlarging/reducing unit that enlarges or reduces the
image size corresponding to image data of the 3D image obtained
15 by the obtaining unit, the image size being enlarged or reduced
at a predetermined enlargement or reduction ratio;
a calculating unit that calculates maximum and minimum
parallaxes for the image data of the enlarged or reduced image
size; and
20 an output unit that outputs the maximum and minimum
parallaxes updated to the calculation results, togethe with
the image data subjected to the enlargement or reduction.
8. An image processing method comprising the steps of:
25 obtaining image data of a 3D image, and content data
containing maximum and minimum parallaxes for an image size
corresponding to the image data of the 3D image;
enlarging or reducing the image size corresponding to
the obtained image data of the 3D image at a predetermined
30 enlargement or reduction ratio;
calculating maximum andminiminnparallaxes for the image
31
SP262995W000
data of the enlarged or reduced image size; and
outputting the maximum and minimum parallaxes updated
to the calculation results, together with the image data
subjected to the enlargement or reduction.
9. A program for causing a computer to perform an operation
comprising the steps of:
obtaining image data of a 3D image, and content data
containing maximum and minimum parallaxes for an image size
10 corresponding to the image data of the 3D image;
enlarging or Teducing the image size corresponding to
the obtained image data of the 3D image at a predetermined
enlargement or reduction ratio;
calculating maximum and minimum parallaxes for the image
15 data of the enlarged or reduced image size; and
outputting the maximum and minimum parallaxes updated
to the calculation results, together with the image data
subjected to the enlargement or reduction.