[0001]The present disclosure, coding apparatus and coding method, a decoding apparatus and decoding method, in particular, obtaining a two-dimensional image data and depth image data of the viewpoint corresponding to the predetermined display image generation method regardless of the viewpoint at the time of imaging encoding apparatus and encoding method capable of, relating to decoding apparatus and decoding method.
Background technique
[0002]
　In multi-view stereo technique, three-dimensional data consisting of three-dimensional position information of a three-dimensional object captured by a plurality of cameras and a two-dimensional image data recording, coding, transmission, decoding, are devised transmission system for displaying It is (for example, see non-Patent Document 1). The encoding method of 3-dimensional data, and the like MPEG (Moving Picture Experts Group phase) method (for example, see Non-Patent Document 2).
[0003]
　The recording obtained by the plurality of cameras, and two-dimensional image data of a plurality of viewpoints, and a depth image data consisting of depth indicating the position in the depth direction of the object for each pixel (the direction perpendicular to the imaging plane) as it is, coding, transmission, decoding, transmission system for displaying has been devised. As a method of encoding a two-dimensional image data and depth image data, MVCD (Multiview and depth video coding) method, AVC (Advanced Video Coding) scheme, HEVC (High Efficiency Video Coding) scheme, and the like.
CITATION
Non-Patent Document
[0004]
非特許文献1 : Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, Steve Sullivan, “High-Quality Streamable Free-Viewpoint Video, Alvaro Collet”, Microsoft Corporation
非特許文献2 : Marius Preda,"MPEG Graphics Compression Model" MPEG document: N9892,May 2008
Summary of the Invention
Problems that the Invention is to Solve
[0005]
　However, when transmitting three-dimensional information of the object as a three-dimensional data, the receiving side, it is necessary to processor high ability to process three-dimensional data.
[0006]
　Furthermore, when transmitting three-dimensional information of the object in the two-dimensional image data and depth image data obtained by a plurality of cameras, the processing of the receiving side is facilitated, the viewpoint of the plurality of cameras is predetermined display image generation method not necessarily a viewpoint suitable for, occurs excess or deficiency of the viewpoint at the receiving side.
[0007]
　The present disclosure has been made in view of such circumstances, it is possible to obtain a two-dimensional image data and depth image data of the viewpoint corresponding to the predetermined display image generation method regardless of the viewpoint at the time of imaging it is intended to way.
Means for Solving the Problems
[0008]
　Encoding apparatus of the first aspect of the present disclosure, the three-dimensional data of an object generated from the 2-dimensional image data of a plurality of first aspect, the plurality corresponding to a predetermined display image generation method a second viewpoint a two-dimensional image data and the two-dimensional data generation unit for generating a depth image data representing the depth direction of the position of the subject of the respective pixels to the third aspect of the of the plurality generated by two-dimensional data generation unit first an encoding unit for encoding a two-dimensional image data of the depth image data of two viewpoints, transmitting the depth image data and the two-dimensional image data of said plurality of second viewpoint is encoded by the encoding unit a coding apparatus and a transmission unit for.
[0009]
　Coding method of the first aspect of the present disclosure, corresponding to the encoding apparatus of the first aspect of the present disclosure.
[0010]
　In the first aspect of the present disclosure, the three-dimensional data of an object generated from the 2-dimensional image data of a plurality of first aspect, a two-dimensional plurality of second viewpoint corresponding to the predetermined display image generation method a depth image data representing the depth direction of the position of the object of each pixel is generated for the image data and the third viewpoint, the depth image data and the two-dimensional image data is coded in the generated plurality of second viewpoint is, the depth image data and the two-dimensional image data of the encoded plurality of second viewpoint is transmitted.
[0011]
　Decoding apparatus of the second embodiment of the present disclosure, the encoded data of the two-dimensional image data of a plurality of first viewpoint corresponding to the predetermined display image generation method, the depth direction of the subject of each pixel for the second viewpoint using a decoding unit for decoding the encoded data of the depth image data indicating the position, and the two-dimensional image data of said plurality of first viewpoint obtained as a result of decoding by the decoder and said depth image data , a three-dimensional data generation unit for generating three-dimensional data of the object, the three-dimensional said generated by the data generating unit based on the three-dimensional data, the display image data to two-dimensional image data in the predetermined display image generation method a decoding device and a two-dimensional data generation unit for generating a.
[0012]
　Decoding method and program of the second aspect of the present disclosure correspond to the decoding apparatus of the second aspect of the present disclosure.
[0013]
　In the second aspect of the present disclosure, the encoded data of the two-dimensional image data of a plurality of first viewpoint corresponding to the predetermined display image generation method, the position in the depth direction of the object for each pixel for the second viewpoint the decoded and the coded data of the depth image data indicating the using the depth image data and the two-dimensional image data of said plurality of first viewpoint obtained as the result of decoding three-dimensional data of the object is generated , based on the generated the three-dimensional data, two-dimensional image data in the predetermined display image generating method is generated as the display image data.
[0014]
　Incidentally, the decoding device of the encoding device and the second side of the first aspect can be realized by causing a computer to execute a program.
[0015]
　Further, in order to realize a decoding device of the encoding device and the second side of the first side, a program to be executed by a computer, by transmitting via a transmission medium or by being recorded on a recording medium, it is possible to provide.
Effect of the invention
[0016]
　According to a first aspect of the present disclosure, it can be encoded. Further, according to the first aspect of the present disclosure, encoded to a decoding device for two-dimensional image data and depth image data of the viewpoint corresponding to the predetermined display image generation method can be acquired without depending on the viewpoint of the time of imaging can do.
[0017]
　According to a second aspect of the present disclosure, it can be decoded. Further, according to the second aspect of the present disclosure, it is possible to obtain a two-dimensional image data and depth image data of the viewpoint corresponding to the predetermined display image generation method regardless of the viewpoint at the time of imaging.
[0018]
　Here, the advantages described in the present invention is not necessarily limited, it may be any of the effects described in the present disclosure.
BRIEF DESCRIPTION OF THE DRAWINGS
[0019]
FIG. 1 is a block diagram showing a configuration example of a first embodiment of a transmission system according to the present disclosure.
It is a block diagram showing a configuration example of a conversion unit of FIG. 1; FIG.
3 is a diagram showing an example of a plurality of viewpoints corresponding to the predetermined display image generation method.
It is a diagram illustrating an example of FIG. 4 camera related information.
Is a flowchart illustrating an encoding process of FIG. 5 the image pickup apparatus and the encoding apparatus of FIG.
6 is a block diagram showing a first configuration example of the conversion unit of FIG.
7 is a block diagram showing a second configuration example of the conversion unit of FIG.
8 is a block diagram showing a third configuration example of the conversion unit of FIG.
Is a flowchart illustrating a decoding process of the decoding device in FIG. 9 Fig.
Is a block diagram showing a configuration example of a second embodiment of the transmission system applied with the [10] The present disclosure.
11 is a block diagram showing a configuration example of an imaging apparatus in FIG. 10.
It is a diagram illustrating an example of FIG. 12 sync information.
Is a block diagram illustrating a configuration example of an encoding device of FIG. 13 FIG. 10.
14 is a flowchart illustrating an encoding process of the image pickup apparatus and the encoding apparatus of FIG. 10.
Is a block diagram showing a configuration example of the decoding unit of FIG. 15 FIG. 10.
16 is a flowchart illustrating a decoding process of the decoding apparatus of FIG. 10.
It is a block diagram showing a configuration example of a third embodiment of a transmission system to which the FIG. 17 the present disclosure.
It is a block diagram showing a configuration example of a synthesis apparatus of FIG. 18 FIG. 17.
Is a flowchart illustrating the synthesizing process of synthesizing apparatus [19] FIG.
FIG. 20 is a block diagram showing a configuration example of a synthesis apparatus according to the fourth embodiment.
21 is a diagram explaining a process for generating a coordinate transformation data.
Is a diagram illustrating an example of FIG. 22 the coordinate transformation information.
Is a flowchart illustrating the synthesizing process of synthesizing apparatus [23] FIG.
FIG. 24 is a block diagram showing a configuration example of a conversion unit in the fourth embodiment.
[25] is a flowchart illustrating a decoding process of the decoding device in the fourth embodiment.
FIG. 26 is a block diagram showing a configuration example of a synthesis apparatus according to the fifth embodiment.
Is a diagram illustrating an example of FIG. 27 the color shift correction information.
It is a flowchart illustrating the synthesizing process of synthesizing apparatus [28] Figure 26.
FIG. 29 is a block diagram showing a configuration example of a conversion unit in the fifth embodiment.
[FIG. 30] is a block diagram showing a configuration example of hardware of a computer.
DESCRIPTION OF THE INVENTION
[0020]
　Hereinafter, embodiments of the present disclosure (hereinafter, referred to as embodiments) will be described. The description will be made in the following order.
　1. First Embodiment: transmission system (FIGS.
　1-9) 2. Second Embodiment: transmission system (FIGS. 10 to
　16) 3. Third Embodiment: transmission system
　(17-19) 4. Fourth Embodiment: transmission system (FIGS. 20 through
　25) 5. Fifth Embodiment: transmission system (FIGS. 26 to
　29) 6. Sixth Embodiment: computer (Fig. 30)
[0021]
　
　(Configuration example of a first embodiment of the transmission system)
　FIG. 1 is a block diagram showing a configuration example of a first embodiment of a transmission system according to the present disclosure.
[0022]
　Transmission system 10 of FIG. 1, the image pickup apparatus 11, the encoding device 12, and the decoding device 13 and the display device 14. Transmission system 10 uses a two-dimensional image data and the like obtained by the imaging device 11 generates display image data in a predetermined display image generation method is displayed.
[0023]
　Specifically, the imaging device 11 of the transmission system 10 is constituted, for example, a multi-camera, distance measuring instrument, and the image processing unit. Multi-camera imaging device 11 has a plurality (e.g., 8) is constituted by a camera captures a 2-dimensional image data of a moving image of the object is at least partially identical with each camera. Range measurements may, for example, provided on each camera, and generates a depth image data of the same viewpoint and the camera.
[0024]
　The image processing unit of the imaging device 11 (three-dimensional data generation unit) a two-dimensional image data and depth image data of the viewpoint of each camera, and by using the internal and external parameters of each camera, the modeling Visual Hull etc. carried out, to create a mesh. The image processing unit includes a geometric information indicating the connection of the three-dimensional position and each point of each of the points constituting the mesh created (Vertex) (Polygon) (Geometry), 3 of the object and a two-dimensional image data of the mesh generated as dimensional data, and supplies to the encoding device 12.
[0025]
　Details of the two-dimensional image data and the method of generating three-dimensional data from the depth image data of a plurality of viewpoints, for example, Saied Moezzi, Li-Cheng Tai, Philippe Gerard, "Virtual View Generation for 3D Digital Video", University of California , San Diego and Takeo Kanade and Peter Rander, PJ Narayanan,: are described in the "Virtualized Reality Constructing Virtual Worlds from Real Scenes".
[0026]
　Coding apparatus 12 is constituted by the conversion unit 21, encoding unit 22, and transmission unit 23.
[0027]
　Conversion unit 21 of the encoding device 12 sets the internal parameters and external parameters of the virtual camera of a plurality of viewpoints corresponding to the predetermined display image generation method as camera parameters. Converter 21, based on the camera parameter from the three-dimensional data supplied from the imaging device 11, and generates a two-dimensional image data and depth image data of a plurality of viewpoints corresponding to the predetermined display image generation method.
[0028]
　For more information from the three-dimensional data of 3DCG technology to generate a two-dimensional image data and depth image data of a plurality of viewpoints, for example, Masayuki Tanimoto, "Toward the ultimate video communication" of Electronics, Information and Communication Engineers Technical report. CS, communication method 110 (323), 73-78, have been described in such 2010-11-25.
[0029]
　In the present specification, the viewpoint of the two-dimensional image data and depth image data is assumed to be the same, the number of two-dimensional image data and depth image data viewpoint and viewpoint may be different. The number of viewpoints and the viewpoint of the two-dimensional image data and depth image data can be the same as the point of view of the camera of the imaging device 11, or may be different.
[0030]
　Converter 21, from the 3-dimensional data supplied from the imaging device 11, the three-dimensional data of the occlusion area not visible from a plurality of viewpoints corresponding to the predetermined display image generation system (hereinafter, referred to as occlusion 3D data) to. Converter 21, two-dimensional image data and depth image data of a plurality of viewpoints corresponding to the predetermined display image generation method, occlusion three-dimensional data, as well as camera-related information is information relating to the virtual camera of the camera parameters and the like of each viewpoint supplied to the encoding unit 22 the metadata including.
[0031]
　Encoding unit 22 encodes the two-dimensional image data supplied, depth image data, and metadata from the converter 21. The coding method can be employed MVCD (Multiview and depth video coding) method, AVC method, the HEVC method or the like.
[0032]
　If the encoding method is MVCD method, two-dimensional image data and depth image data of all viewpoints is encoded together. As a result, one encoded stream containing encoded data and metadata of the two-dimensional image data and depth image data is generated. In this case, the camera parameters of the metadata is arranged in reference Displays information SEI encoded stream. Moreover, information on the depth image data of the metadata is arranged in Depth representation information SEI.
[0033]
　On the other hand, if the coding method is AVC scheme and HEVC method, depth image data and the two-dimensional image data for each viewpoint is encoded separately. As a result, the encoded stream of each viewpoint including a two-dimensional image data and metadata each viewpoint, encoded streams of each viewpoint including the encoded data and metadata depth image data for each viewpoint is generated. In this case, metadata is disposed, for example, User unregistered SEI of the encoded stream. Also, the metadata includes information for associating the coded stream and the camera parameters, and the like.
[0034]
　Incidentally, without including information associating the coded stream and the camera parameters such as the meta data, the encoded stream may be included only the metadata corresponding to the encoded stream.
[0035]
　Encoding unit 22 supplies the transmitting section 23 the encoded stream. Transmission unit 23 transmits the encoded stream supplied from the coding section 22 to the decoder 13. In the present specification, so that the metadata is transmitted is placed in the coded stream, it may be transmitted separately from the encoded stream.
[0036]
　Decoder 13 is composed of a receiving unit 31, decoding unit 32 and the conversion unit 33,.
[0037]
　Receiving unit 31 of the decoding device 13 receives an encoded stream transmitted from the transmission unit 23, and supplies the decoding section 32. Decoding unit 32, the encoded stream supplied from the receiving unit 31 is decoded by method corresponding to the coding method in the encoding unit 22. Decoding unit 32 supplies the two-dimensional image data and depth image data of a plurality of viewpoints obtained as a result as well as the metadata to the converter 33.
[0038]
　Converter 33, based on the display image generation method of the decoding device 13 and the meta data supplied from the decoding unit 32, a two-dimensional image data and depth image data of a plurality of viewpoints, two-dimensional image data of the predetermined viewpoint, or select a two-dimensional image data and depth image data of a predetermined viewpoint. Converter 33, two-dimensional image data of a predetermined viewpoint is selected or, based on the 2-dimensional image data and depth image data of a predetermined viewpoint, it generates display image data, to the display device 14.
[0039]
　Display device 14, two-dimensional head mounted display or a two-dimensional monitor, and the like three-dimensional head mounted display or a three-dimensional monitor. Display device 14, based on the display image data supplied from the conversion unit 33, and displays the two-dimensional display or three-dimensional display image.
[0040]
　(Configuration of the conversion unit 21)
　FIG 2 is a block diagram showing a configuration example of a conversion unit 21 of FIG. 1.
[0041]
　Conversion unit 21 of FIG. 2 is constituted by the object position determination unit 51, the camera determining unit 52,2 dimensional data generation unit 53 and the occlusion determining section 54,.
[0042]
　Object position determination unit 51 of the conversion unit 21 determines the three-dimensional position of the center of the object of the three-dimensional object corresponding to the three-dimensional data supplied from the imaging device 11 of FIG. 1, the three-dimensional position supplies subject position information to the camera determining unit 52 indicating.
[0043]
　The camera determination unit 52, based on object position information supplied from the object position determination unit 51 determines the camera parameters of a plurality of viewpoints corresponding to the predetermined display image generation method, the two-dimensional data generation unit 53 and the occlusion determination supplied to the part 54. The camera determination unit 52 generates a camera-related information from the camera parameters of each viewpoint supplied to the encoding unit 22 of FIG. 1 as metadata.
[0044]
　2-dimensional data generation unit 53, based on the camera determining unit 52 in the camera parameter of a plurality of viewpoints to be supplied, for each viewpoint, performs perspective projection of a three-dimensional object corresponding to the three-dimensional data.
[0045]
　Specifically, the relationship between the matrix M corresponding to the three-dimensional coordinates of the matrix m'the world coordinate system corresponding to the two-dimensional position of each pixel, the internal parameter A and the external parameter R of the camera | with t, the following represented by formula (1).
[0046]
[Number 1]

[0047]
　Equation (1) it is more particularly represented by formula (2).
[0048]
[Number 2]

[0049]
　In the formula (2), (u, v ) is the two-dimensional coordinates on the image, f x , f y is the focal length. Also, C x , C y is the main point, r 11 to r 13 , r 21 through r 23 , r 31 through r 33 , and t 1 to t 3 is a parameter, (X, Y, Z ) is a three-dimensional coordinates of the world coordinate system.
[0050]
　Accordingly, the two-dimensional data generation unit 53, by the above-mentioned formula (1) or (2), using the camera parameters, determining the three-dimensional coordinates corresponding to the two-dimensional coordinates of each pixel.
[0051]
　Then, two-dimensional data generation unit 53, for each viewpoint, by a two-dimensional image data of the three-dimensional coordinates corresponding to the two-dimensional coordinates of each pixel of the three-dimensional data into two-dimensional image data of each pixel, generating a 2-dimensional image data to associate the two-dimensional coordinates and image data of each pixel. Further, the two-dimensional data generation unit 53, for each viewpoint, calculated depth of each pixel based on the three-dimensional coordinates corresponding to the two-dimensional coordinates of each pixel, the depth image data associating two-dimensional coordinates and depth of each pixel generated. Depth may be, for example, the reciprocal 1 / z in the depth direction of the position z of the subject. 2-dimensional data generation unit 53 supplies the two-dimensional image data and depth image data for each viewpoint to the encoding unit 22.
[0052]
　Occlusion determining section 54, based on the camera parameters supplied from the camera determining unit 52 extracts the occlusion 3D data from the 3D data supplied from the imaging device 11, and supplies to the encoding unit 22 as metadata.
[0053]
　(Several examples of viewpoint corresponding to the predetermined display image generation method)
　FIG. 3 is a diagram showing an example of a plurality of viewpoints corresponding to the predetermined display image generation method.
[0054]
　In the example of A of FIG. 3, the predetermined display image generation method, a predetermined viewpoint display image generation method for displaying two-dimensional display image of one predetermined viewpoint, a display image free viewpoint is one arbitrary viewpoint 2 it is a free viewpoint display image generating method of dimensions displayed. In this case, as shown in A of FIG. 3, a plurality of viewpoints corresponding to the predetermined display image generation method, for example, the viewpoint of a plurality (8 in the example of A of FIG. 3) that is present at regular intervals on the same circle 71 to a 78, the orientation of the viewpoint 71 to 78 are oriented towards the center of the circle.
[0055]
　By the above, the decoding device 13, by selecting the two-dimensional image data of one of the viewpoint of the viewpoint 71 to 78 as the display image data is supplied to the display device 14, a display device a display image of the viewpoint 14 it can be displayed two-dimensionally.
[0056]
　Further, the decoding unit 13 generates a 3-dimensional data based on two-dimensional image data and depth image data of the viewpoint 71 to 78, to the free viewpoint, the perspective projection of a three-dimensional object corresponding to the three-dimensional data by performing, it can generate a two-dimensional image data of the free viewpoint as the display image data. Accordingly, the decoding device 13, by supplying the generated display image data to the display device 14 can display two-dimensional display image of free-viewpoint display device 14.
[0057]
　In the example of B of FIG. 3, a predetermined display image generation method, a predetermined viewpoint display image generation method, free viewpoint display image generation method, three-dimensional display to a display image based on the 2-dimensional image data of the two viewpoints are displayed three-dimensionally image generation method, and a display image of the view between two viewpoints are interpolated viewpoint display image generating method for displaying two-dimensional.
[0058]
　In this case, as shown in B of FIG. 3, a plurality of viewpoint corresponding to a predetermined display image generation method, for example, the viewpoint of a plurality (10 in the example of B of FIG. 3) present at regular intervals on the same circle 81 to 90. Viewpoint 88 and the viewpoint 89, respectively, perspective 87, viewpoint 88 and arranged in a substantially horizontal direction. Viewpoint 87 and the viewpoint 89 and, a substantially horizontal spacing of the viewpoint 88 and the viewpoint 90, for example, corresponds to the spacing of the human left and right eyes. Orientation of the viewpoint 81 and 90 are oriented toward the center of the circle where the viewpoint 81 to 90 are arranged.
[0059]
　Thus, the decoding apparatus 13, as in the case of A in FIG. 3, it is possible to two-dimensionally displays a display image of any aspect of the display image or free viewpoint of the viewpoint 81 to 90 on the display device 14. Further, the decoding device 13 selects the two-dimensional image data of the pair of the viewpoint 87 and the viewpoint 89 of the pair or viewpoint 88 and the viewpoint 90 as the display image data, by supplying to the display device 14, viewpoint 87 and the viewpoint 89 display image pair pair or viewpoint 88 and the viewpoint 90 can be three-dimensionally displayed on the display device 14.
[0060]
　Further, the decoding unit 13, based on the two-dimensional image data and depth image data pair pair or viewpoint 88 and the viewpoint 90 of the viewpoint 87 and the viewpoint 89, interpolating the two-dimensional image data of viewpoint between the pair can. Accordingly, the decoding device 13, by supplying to the display device 14 a two-dimensional image data interpolated as the display image data, between the pairs of the pair or view 88 and the viewpoint 90 of the viewpoint 87 and the viewpoint 89 viewpoint of the display image can be displayed device 14 two-dimensionally displays the.
[0061]
　For examples of B in A and 3 in FIG. 3, a plurality of number of predetermined display image generation method. Thus, the decoding apparatus 13, based on the two-dimensional image data and depth image data of a plurality of viewpoints corresponding to the predetermined display image generation method, to generate the display image data in one of a predetermined display image generation method it can. Therefore, the encoding device 12 can be said to be able to generate a coded stream having scalability. The number of predetermined display image generation method may be one.
[0062]
　The camera determination unit 52 determines the number of cameras of the imaging device 11, when more than a plurality of viewpoints corresponding to the predetermined display image generation method, the camera parameters of a plurality of viewpoints corresponding to the predetermined display image generation method by it can eliminate redundancy point of view.
[0063]
　For example, although the point of view of the camera of the imaging device 11 is an 8-view perspective 71 through 78, a plurality of viewpoint viewpoint 71 corresponding to a plurality of display image generation system, the viewpoint 73, in four viewpoints of the viewpoint 75, and the viewpoint 77 some cases, the camera determination unit 52 determines the camera parameters of the four viewpoints. Therefore, as compared with the case where all the two-dimensional image data and depth image data of the viewpoint 71 to 78 are encoded, it is possible to eliminate the redundancy of the viewpoint in the coded stream.
[0064]
　Positions of the plurality of viewpoints corresponding to the plurality of display image generation method is not limited to the example of B of A and 3 in FIG. 3, it is possible to an arbitrary position. The camera parameters of each viewpoint, for example, be a camera parameter corresponding to a relatively narrow angle of view. In this case, it is possible to improve the resolution of zooming of the displayed image.
[0065]
　Further, in this specification, so that the number of predetermined display image generation method even when a plurality of the plurality of viewpoints of the encoded stream corresponding to all the predetermined display image generating method is generated but for each predetermined display image generation method, may be encoded stream of a corresponding plurality of viewpoints are generated. In this case, the decoding device 13 generates display image data by using the coded stream corresponding to their display image generation method.
[0066]
　Furthermore, the camera determining unit 52, in response to a request from the decoder 13, may be determined a plurality of viewpoints corresponding to the predetermined display image generation method.
[0067]
　(Camera Examples of relevant information)
　FIG. 4 is a diagram showing an example of a camera-related information.
[0068]
　In the example of FIG. 4, a plurality of viewpoints corresponding to the predetermined display image generation method, a viewpoint 81 to 90 in B of FIG.
[0069]
　As shown in FIG. 4, the camera-related information, the camera ID of each viewpoint, the extrinsic parameters R of the viewpoint | configured T, internal parameter A, IsStereoFlag, and by the stereo camera ID is associated.
[0070]
　The camera ID is an ID unique to the viewpoint. In the example of FIG. 4, the camera ID is assigned in the order from 0 for viewpoint 81 to 90. Further, in FIG. 4, the external parameter R camera ID is i | describes Ti, the internal parameter A and Ai | a T Ri.
[0071]
　isStereoFlag is a flag indicating whether other viewpoint arranged in a substantially horizontal direction exists, is set to 1 when the other viewpoint arranged in a substantially horizontal direction exists. In the example of FIG. 4, the viewpoint 87 or 90 camera ID is 6 to 9, since the other viewpoint arranged in a substantially horizontal direction exists, as shown in FIG. 4, the correspondence to the camera ID is 6-9 It was isStereoFlag is 1.
[0072]
　Stereo camera ID is a camera ID of another viewpoint arranged in a substantially horizontal direction, a pair information for identifying the two viewpoints arranged in a substantially horizontal direction. In the example of FIG. 4, the camera ID of another viewpoint 89 arranged in a substantially horizontal direction of the viewpoint 87 camera ID is 6 is 8, the stereo camera ID associated with the camera ID is a 6 becomes 8 .
[0073]
　Similarly, the camera ID of another viewpoint 90 arranged in a substantially horizontal direction of the viewpoint 88 camera ID is 7 is a 9, a stereo camera ID associated with the camera ID is 7 is nine. Also, next to the stereo camera ID is 6 associated with the camera ID is 8, a stereo camera ID associated with the camera ID is a 9 becomes 7.
[0074]
　(Description of the processing of the image pickup apparatus and an encoding apparatus)
　FIG. 5 is a flowchart illustrating an encoding process of the image pickup device 11 and the encoding device 12 of FIG. 1.
[0075]
　In step S11 in FIG. 5, the imaging device 11 captures a two-dimensional image data of a moving image in a multi-camera, and generates a depth image data ranging measuring instrument. In step S12, the imaging device 11 generates a 3-dimensional data using a two-dimensional image data and depth image data is supplied to the encoding device 12.
[0076]
　In step S13, the object position determination unit 51 (FIG. 2) of the coding apparatus 12 determines the three-dimensional position of the center of the object of the three-dimensional object corresponding to the three-dimensional data supplied from the imaging device 11 supplies subject position information indicating the three-dimensional position in the camera determining unit 52.
[0077]
　In step S14, the camera determining unit 52, based on object position information supplied from the object position determination unit 51 determines the camera parameters of a plurality of viewpoints corresponding to the predetermined display image generation method, the two-dimensional data generation unit 53 and supplies the occlusion determining section 54. The camera determination unit 52 generates a camera-related information of FIG. 4 from the camera parameters and the like, and supplies to the encoding unit 22 of FIG. 1 as metadata.
[0078]
　In step S15, 2-dimensional data generation unit 53, based on the camera determining unit 52 in the camera parameter of a plurality of viewpoints to be supplied, for each viewpoint, it performs perspective projection of a three-dimensional object corresponding to the three-dimensional data, each obtaining three-dimensional coordinates corresponding to the two-dimensional coordinates of the pixel.
[0079]
　In step S16, the two-dimensional data generation unit 53, for each viewpoint, by a two-dimensional image data of the three-dimensional coordinates of each pixel of the three-dimensional data into two-dimensional image data of each pixel, the two-dimensional image data to generate. Further, 2-dimensional data generation unit 53, for each viewpoint, calculated depth of each pixel based on the three-dimensional coordinates of each pixel, to generate a depth image data. 2-dimensional data generation unit 53 supplies the two-dimensional image data and depth image data for each viewpoint to the encoding unit 22.
[0080]
　In step S17, the occlusion determining section 54, based on the camera parameters supplied from the camera determining unit 52 extracts the occlusion 3D data from the 3D data supplied from the imaging device 11, the coding unit 22 as metadata supplied to.
[0081]
　In step S18, the encoding section 22, two-dimensional image data supplied from the conversion unit 21, and encoded depth image data, and metadata, and supplies the encoded stream obtained as the result to the transmission unit 23.
[0082]
　In step S19, the transmission section 23 transmits the encoded stream supplied from the coding section 22 to the decoder 13. Then, the process is terminated.
[0083]
　(First configuration example of the conversion unit 33)
　FIG 6 is a block diagram showing a first exemplary configuration of the conversion unit 33 FIG.
[0084]
　Conversion unit 33 of FIG. 6 is a conversion unit 33 when the display image generation method of the decoding device 13 of FIG. 1 is a predetermined viewpoint display image generation method, and the display control unit 112 and the selector 111.
[0085]
　Selector 111 of the converter 33, based on the camera-related information supplied from the decoding unit 32 of FIG. 1 recognizes the camera ID of the desired one viewpoint. Selection unit 111, two-dimensional image data supplied from the decoding unit 32, a depth image data, and of the occlusion 3D data, selects the 2-dimensional image data for one viewpoint corresponding to the recognized camera ID. Selecting section 111 supplies the two-dimensional image data of one viewpoint selected by the display control unit 112.
[0086]
　The display control unit 112, by supplying to the display device 14 a two-dimensional image data of one viewpoint supplied from the selection unit 111 as the display image data to display the two-dimensional display image on the display device 14.
[0087]
　Although not shown, the configuration of the conversion unit 33 when the display image generation method of the decoding device 13 is a three-dimensional display image generation method, a selection unit 111, arranged in a substantially horizontal direction on the basis of the camera-related information except for selecting two-dimensional image data of two viewpoints, the same configuration as in FIG.
[0088]
　That is, in this case, selection section 111 selects the point of view of the camera ID is a isStereoFlag of the cameras related information 1, the 2-dimensional image data of the viewpoint of the stereo camera ID corresponding to the camera ID. As a result, the display device 14, the two-dimensional image data of two viewpoints selected is supplied as the display image data. Display device 14 displays the display image for the left eye on the basis of the two-dimensional image data for one viewpoint in the display image data, and displays the display image for the right eye based on the two-dimensional image data of the other one viewpoint it allows to view three-dimensional display image.
[0089]
　(Second configuration example of the conversion unit 33)
　FIG 7 is a block diagram showing a second exemplary configuration of the conversion unit 33 FIG.
[0090]
　Conversion unit 33 of FIG. 7 is a conversion unit 33 when the display image generation method of the decoding device 13 is interpolated viewpoint display image generation method, and by the selection unit 131, the interpolation unit 132 and the display control unit 133, .
[0091]
　Selecting unit 131 of the converter 33, based on the camera-related information supplied from the decoding unit 32 of FIG. 1, one of the camera ID corresponding to a 1 IsStereoFlag bract, a stereo camera ID corresponding to one of them It recognizes the door. Selecting unit 131, two-dimensional image data supplied from the decoding unit 32, a depth image data, and of the occlusion 3D data, 2D image data and depth images of two viewpoints corresponding to the recognized two cameras ID was select data, and supplies the interpolation section 132.
[0092]
　Interpolation unit 132, based on the 2-dimensional image data and depth two viewpoints of image data supplied from the selection unit 131, interpolates the two-dimensional image data to viewpoint position between the two viewpoints. Such interpolation techniques are called Depth Image Based Rendering, for example, are described in International Publication WO2014083752 pamphlet. Interpolation unit 132 supplies the display control unit 133 the two-dimensional image data of one viewpoint interpolated.
[0093]
　The display control unit 133, by supplying to the display device 14 a two-dimensional image data of one viewpoint supplied from the interpolation unit 132 as the display image data to display the two-dimensional display image on the display device 14.
[0094]
　(Third configuration example of the conversion unit 33)
　FIG. 8 is a block diagram showing a third exemplary configuration of the conversion unit 33 FIG.
[0095]
　Conversion unit 33 of FIG. 8 is a conversion unit 33 when the display image generation method of the decoding device 13 is a free-viewpoint display image generation method. Conversion unit 33 of FIG. 8 is composed of three-dimensional data generation unit 151, the object position determination unit 152, a camera position determination unit 153,2 dimensional data generation unit 154 and the display control unit 155.
[0096]
　3-dimensional data generation unit 151 of the converter 33, two-dimensional image data supplied from the decoding unit 32, using depth image data, and a camera parameter included in the camera-related information, like the imaging device 11, the subject to generate a three-dimensional data. 3-dimensional data generation unit 151, the generated three-dimensional data, and corrected using the occlusion 3D data supplied from the decoding unit 32. Thus, it is possible to generate three-dimensional data of all areas of the subject including the occlusion region which can not be the only two-dimensional image data and depth image data generating. 3-dimensional data generation unit 151 supplies the corrected three-dimensional data to object position determination unit 152 and the 2-dimensional data generation unit 154.
[0097]
　Object position determination unit 152 determines the three-dimensional position of the center of the object of the three-dimensional object corresponding to the three-dimensional data supplied from the three-dimensional data generation unit 151, object position information indicating the three-dimensional position and it supplies the camera position determination unit 153.
[0098]
　The camera position determination unit 153, based on the object position information supplied from the object position determination unit 152 determines the camera parameters of the free viewpoint, and supplies the camera parameter to the two-dimensional data generation unit 154.
[0099]
　2-dimensional data generation unit 154, the three-dimensional data supplied from the three-dimensional data generation unit 151, based on the camera parameters of the free viewpoint supplied from the camera position determination unit 153, 2 in free-viewpoint display image generation method generating a dimensional image data as display image data.
[0100]
　Specifically, two-dimensional data generation unit 154, based on the camera parameters of the free viewpoint, as with two-dimensional data generation unit 53 of FIG. 2 performs a perspective projection of a three-dimensional object corresponding to the three-dimensional data, obtaining three-dimensional coordinates corresponding to the two-dimensional coordinates of each pixel. Then, two-dimensional data generation unit 154, a two-dimensional image data of the three-dimensional coordinates corresponding to the two-dimensional coordinates of each pixel of the three-dimensional data, generated as the display image data of each pixel.
[0101]
　Further, 2-dimensional data generation unit 154 obtains the depth of each pixel based on the three-dimensional coordinates corresponding to the two-dimensional coordinates of each pixel, to generate a depth image data. 2-dimensional data generation unit 154 supplies the display image data and the depth image data to the display control unit 155.
[0102]
　The display control unit 155 to the display device 14 display image data and the depth image data supplied from the two-dimensional data generation unit 154. Display device 14 is, for example, the display image data and the depth image data as Point Cloud, displaying the two-dimensional display image.
[0103]
　As described above, the conversion unit 33 of FIG. 8 generates three-dimensional data from two-dimensional image data and depth image data of a plurality of viewpoints, based on the camera parameters of the free viewpoint newly determined, the three-dimensional from the data, and generates a two-dimensional image data and depth image data. Thus, converter 33, and you can change the zoom magnification of 2-dimensional image data and depth image data of a plurality of viewpoints that is transmitted from the encoding device 12 to change the viewpoint.
[0104]
　(Process description of the decoding apparatus)
　FIG. 9 is a flowchart illustrating a decoding process of the decoding device 13 of the case structure of the converting unit 33 Fig. 1 is the configuration of FIG. The decoding process is, for example, coded streams are started when that has been transmitted from the encoding device 12.
[0105]
　In step S31 in FIG. 9, receiving portion 31 of the decoding device 13 receives an encoded stream transmitted from the transmission unit 23, and supplies the decoding section 32.
[0106]
　In step S32, the decoding unit 32 decodes the encoded stream supplied from the receiving unit 31. Decoding unit 32 supplies the two-dimensional image data and depth image data of a plurality of viewpoints obtained as a result as well as the metadata to the converter 33.
[0107]
　In Step S33, 3-dimensional data generation unit 151 of the conversion unit 33 (FIG. 8), using the camera parameters included two-dimensional image data supplied from the decoding unit 32, a depth image data, and metadata, the imaging apparatus similar to 11, to generate a three-dimensional data of the object.
[0108]
　In step S34, the three-dimensional data generation unit 151, by using the occlusion 3D data included in the meta data supplied from the decoding unit 32, corrects the three-dimensional data generated in step S33. 3-dimensional data generation unit 151 supplies the corrected three-dimensional data to object position determination unit 152.
[0109]
　In step S35, the object position determination unit 152 determines the three-dimensional position of the center of the object of the three-dimensional object corresponding to the three-dimensional data supplied from the three-dimensional data generation unit 151, the three-dimensional position supplies subject position information to the camera position determination unit 153 to represent.
[0110]
　In step S36, the camera position determination unit 153, based on the object position information supplied from the object position determination unit 152 determines the camera parameters of the free viewpoint, and supplies the camera parameter to the two-dimensional data generation unit 154.
[0111]
　In step S37, the two-dimensional data generation unit 154, based on the camera parameters of the three-dimensional data and the free viewpoint, to generate two-dimensional image data as display image data in a free-viewpoint display image generation method.
[0112]
　In step S38, the two-dimensional data generation unit 154 obtains the depth of each pixel based on the three-dimensional coordinates corresponding to the two-dimensional coordinates of each pixel, to generate a depth image data. 2-dimensional data generation unit 154 supplies the display image data and the depth image data to the display control unit 155.
[0113]
　In step S39, the display control unit 155 to the display device 14 display image data and the depth image data supplied from the two-dimensional data generation unit 154.
[0114]
　As described above, in the transmission system 10, the imaging device 11, to generate three-dimensional data from two-dimensional image data of a plurality of viewpoints taken, the encoding device 12, a predetermined display image generated from the 3-dimensional data generating and coding 2-dimensional image data and depth image data of a plurality of viewpoints corresponding to the method. Thus, the coding device 12 may be encoded such decoding device 13 the two-dimensional image data and depth image data of the viewpoint corresponding to the predetermined display image generation method regardless of the viewpoint at the time of imaging can be acquired .
[0115]
　Further, the decoding unit 13 receives the encoded streams of two-dimensional image data and depth image data of a plurality of viewpoints corresponding to the predetermined display image generation method that is transmitted from the encoding device 12 and decodes. Thus, the decoding apparatus 13 can obtain the two-dimensional image data and depth image data of the viewpoint corresponding to the predetermined display image generation method regardless of the viewpoint at the time of imaging.
[0116]
　Furthermore, when the number of predetermined display image generation method is more, in the decoding device of the plurality of display image generation method can reproduce the coded stream generated by the encoding device 12. For example, the code display image generation method is in both the decoding device of a high function which is decoding device and, free viewpoint image generation method is the display image generation method of low function with a predetermined viewpoint display image generation method, in accordance with their abilities it is possible to reproduce the stream.
[0117]
　
　(Configuration example of a second embodiment of the transmission system)
　FIG. 10 is a block diagram showing a configuration example of a second embodiment of the transmission system applied with the present disclosure.
[0118]
　In the configuration shown in FIG. 10 are denoted by the same reference numerals are the same as the structural elements of FIG. Redundant description will be omitted as appropriate.
[0119]
　Arrangement of Figure 10 the transmission system 170 includes an imaging device 11, the encoding device 12 and decoding device 13, the image pickup apparatus 171, the point that replaces the encoding apparatus 172 and decoding apparatus 173, the configuration of the transmission system 10 of FIG. 1 and different. In the transmission system 170, the two-dimensional image data and depth image data acquired by the imaging device 171 is transmitted are encoded directly.
[0120]
　Specifically, the imaging device 171 of the transmission system 170 is constituted by a multi-camera 181 and the image processing unit 182. A plurality of cameras constituting the multi-camera 181 of the imaging apparatus 171, respectively, to image the two-dimensional image data. Multi-camera 181 supplies the two-dimensional image data captured by the cameras to the image processing unit 182.
[0121]
　The image processing unit 182, one camera among the plurality of cameras constituting the multi-camera 181 as a reference camera, and the reference camera to other cameras. The image processing unit 182, based on the 2-dimensional image data captured by the two-dimensional image data and the reference camera that captured by the reference camera multi-camera 181, the two-dimensional image data of each reference camera to the reference camera synchronization detecting a shift. The image processing unit 182 detects the synchronization deviation of the reference camera as 0. The image processing unit 182 supplies the synchronization shift information and the camera-related information including information indicating a sync of each camera viewpoint detected in the encoder 172.
[0122]
　The image processing unit 182, by performing a stereo matching on the two-dimensional image data captured by the cameras, generates a depth image data of each camera, supplied to the encoder 172.
[0123]
　Encoder 172, a synchronization shift information and the camera-related information supplied from the imaging device 171 and metadata. Encoding apparatus 172, similar to the encoding unit 22 of FIG. 1, two-dimensional image data and depth image data of each camera to be supplied from the imaging device 171, and the metadata is encoded to generate an encoded stream. Encoder 172 (transmission section) transmits the generated encoded stream to the decoding device 173.
[0124]
　Configuration of the decoding apparatus 173, the decoding unit 32 and the conversion unit 33, the point in place of the decoding unit 191 conversion unit 192 is different from the configuration of the decoding device 13 of FIG. 1.
[0125]
　Decoding unit 191 of the decoding device 173, the coded stream supplied from the receiving unit 31 is decoded by method corresponding to the coding method in the encoder 172. Decoding unit 191, based on the synchronization shift information of the metadata obtained as a result, to synchronize the reference camera and a two-dimensional image data and depth image data of each reference camera, and supplies to the converter 192. Further, the decoding unit 191 supplies the camera-related information of the metadata to the conversion unit 192.
[0126]
　Configuration of the conversion unit 192, except that 3-dimensional data is not corrected using the occlusion 3D data, since it is same as the configuration of the conversion unit 33 1, description is omitted.
[0127]
　(Configuration Example of Imaging Apparatus)
　FIG. 11 is a block diagram showing a configuration example of an imaging apparatus 171 in FIG. 10.
[0128]
　Multi-camera 181 of the imaging device 171 in FIG. 11, N (N is plural) formed by the camera 211-1 211-N of the.
[0129]
　Camera 211-1 211-N, for example, the point of view of the camera 211-1 211-N are arranged such that the viewpoint corresponding to the predetermined display image generation method. Camera 211-1 211-N, respectively captures an image, and supplies the two-dimensional image data of a moving image obtained as a result (captured image data) to the image processing unit 182. Hereinafter, when it is not necessary to distinguish the camera 211-1 211-N, it referred to as a camera 211 them together.
[0130]
　The image processing unit 182, calibration unit 221, and the detecting unit 222 and the depth generation unit 223, synchronous shift.
[0131]
　Calibration section 221 of the image processing unit 182 for each camera 211, for the two-dimensional image data supplied from the multi-camera 181, calibrated using the camera parameters. The calibration unit 221, and supplies the shift detection unit 222 synchronizes the two-dimensional image data of each camera 211 after calibration. Moreover, the calibration unit 221 supplies the camera-related information to the encoding device 172 of Figure 10.
[0132]
　Synchronization shift detector 222, one of the camera 211-1 211-N as a reference camera, and the reference camera to rest. Synchronization shift detector 222, based on the 2-dimensional image data of the two-dimensional image data and the reference camera reference camera supplied from the calibration unit 221, for each reference camera, the two-dimensional image data of the reference camera to the reference camera to detect the synchronization shift in msec order.
[0133]
　Specifically, synchronization shift detector 222 detects the flash light from the two-dimensional image data of the reference camera and two-dimensional image data of the reference camera. Synchronization shift detector 222, a flash light is a difference between the imaging time of 2 dimensional image data of the reference camera and two-dimensional image data of the reference camera is detected, detects a synchronization shift of the two-dimensional image data of the reference camera.
[0134]
　Sync of the two-dimensional image data of the reference camera, by a method other than a method for detecting the flash light, may be performed using a two-dimensional image data. Also, if the camera 211 acquires speech data together with the 2-dimensional image data by detecting the sync of the audio data of the reference camera with respect to the reference camera, to detect a synchronization deviation of the two-dimensional image data of the reference camera it may be.
[0135]
　Synchronization shift detector 222 detects the synchronization deviation of the two-dimensional image data of the reference camera as 0. Synchronization shift detector 222 supplies a synchronization shift information of the detected synchronization error in encoder 172. Also, synchronization shift detector 222 supplies the two-dimensional image data of each camera 211 supplied from the calibration unit 221 to the depth generation unit 223.
[0136]
　Depth generator 223, by performing a stereo matching on the two-dimensional image data of each camera 211 supplied from the detection unit 222 synchronization shift, and generates a depth image data of each camera 211. Depth generator 223 supplies the two-dimensional image data and depth image data of each camera 211 to the encoding device 172.
[0137]
　(Example of synchronization deviation information)
　FIG. 12 is a diagram showing an example of a shift synchronization information.
[0138]
　In the example of FIG. 12, the point of view of the camera 211 is a viewpoint 81 to 90 in B of FIG.
[0139]
　As shown in FIG. 12, shift synchronization information includes the camera ID of the viewpoint of each camera 211, num_units_in_tick of the camera 211, time_scale, Delta_num_units_in_tick, and Delta_time_scale by the associated.
[0140]
　num_units_in_tick the frame interval is information indicating how many pieces of the time_unit defined by time_scale. time_scale is the number of one-second time_unit. Therefore, it is possible by num_units_in_tick and time_scale, represents the frame rate.
[0141]
　In the example of FIG. 12, the frame rate of the camera 211 camera ID is 0, 1, 4, and 5 is 60 Hz (59.94 Hz). Thus, for example, num_units_in_tick camera 211 camera ID is 0, 1, 4, and 5 is 1001, time_scale is 60000. The frame rate of the camera 211 camera ID is 2, 3, and 6 to 9 is 30 Hz (29.97 Hz). Thus, for example, num_units_in_tick camera 211 camera ID is 2, 3, and 6 to 9 is 2002, time_scale is 60000.
[0142]
　Delta_num_units_in_tick is information indicating how many pieces of the time_unit a synchronization shift is defined by Delta_time_scale. Delta_time_scale is the number of one second time_unit. Therefore, it is possible by Delta_num_units_in_tick and Delta_time_scale, represents a synchronization shift.
[0143]
　In the example of FIG. 12, the camera 211 camera ID is 0 is the reference camera. Therefore, Delta_num_units_in_tick corresponding to the camera ID is 0 is 0. Further, in the example of FIG. 12, the synchronization deviation of the camera 211 camera ID is 3,4,7, and 9 is 0. Thus, 3,4,7, Delta_num_units_in_tick corresponding to the camera ID and a 9 is also zero.
[0144]
　Further, in the example of FIG. 12, the synchronization deviation of the camera 211 camera ID is 1 and 5 are 1/30 (1 / 29.97) seconds. Thus, for example, Delta_num_units_in_tick corresponding to the camera ID is 1 and 5 are 2002, Delta_time_scale is 60000. Also, synchronization deviation of the camera 211 camera ID is 2, 6, and 8 is 1/15 (1 / 14.985) seconds. Thus, for example, Delta_num_units_in_tick corresponding to the camera ID is 2, 6, and 8 are 4004, Delta_time_scale is 60000.
[0145]
　Note that the synchronization shift information may be synchronized displacement of all the cameras 211 a synchronization shift occurs is included sync common flag indicating whether the same.
[0146]
　In this case, when synchronization shift of all the cameras 211 a synchronization shift occurs is the same, displacement synchronization information, synchronization indicating that sync of all cameras 211 a synchronization shift occurs is the same shift common flag, sync sync flag indicating whether or not each camera 211 (synchronization shift presence information), and composed of information indicating the common sync to all cameras 211 a synchronization shift occurs.
[0147]
　On the other hand, when the synchronization deviation of all cameras 211 a synchronization shift occurs is not identical, the deviation synchronization information, the synchronization deviation of all the cameras 211 and synchronization shift information of FIG. 12, the synchronization deviation has occurred identical constituted by the common flag synchronization shift indicating that it is not.
[0148]
　(Configuration example of an encoding apparatus)
　FIG. 13 is a block diagram illustrating a configuration example of an encoding apparatus 172 of FIG. 10.
[0149]
　Encoding apparatus 172 of FIG. 13 is a coding device 172 for performing coding in AVC method or HEVC method comprises the 2N coding unit 241-1 through 241-2N and the transmission unit 242.
[0150]
　Encoding section 241-1 through 241-2N of the encoder 172, respectively, the two-dimensional image data or depth image data of each camera 211 supplied from the image pickup device 171, encoded in AVC format or HEVC scheme, code generating a stream.
[0151]
　Encoding section 241-1 through 241-2N, the camera-related information and the synchronization shift information supplied from the imaging device 171, as well as information for associating the camera-related information and the synchronization shift information and the coded stream and metadata. Encoding section 241-1 through 241-2N places the metadata User unregistered SEI of the generated coded stream.
[0152]
　Incidentally, the encoding unit 241-1 to 241-N is the User unregistered SEI encoded stream, it is also possible to place only the metadata corresponding to the encoded stream. In this case, information for associating the camera-related information and the synchronization shift information and the coded stream is not included in the metadata. Encoding section 241-1 through 241-2N supplies to the transmission unit 242 the encoded stream metadata is arranged.
[0153]
　Transmission unit 242 transmits the encoded stream supplied from the coding section 241-1 to 241-2N to the decoding device 173 of Figure 10.
[0154]
　(Description of the processing of the image pickup apparatus and the encoding apparatus)
　FIG. 14 is a flowchart illustrating an encoding process of the image pickup device 171 and the encoding device 172 of Figure 10.
[0155]
　In step S51 in FIG. 14, the camera 211 of the imaging device 171 (FIG. 11) captures an image, and supplies the two-dimensional image data of a moving image obtained as a result to the image processing unit 182.
[0156]
　In step S52, the calibration unit 221 of the image processing unit 182 for each camera 211, for the two-dimensional image data supplied from the multi-camera 181, calibrated using the camera parameters. The calibration unit 221, and supplies the shift detection unit 222 synchronizes the two-dimensional image data of each camera 211 after calibration.
[0157]
　In step S53, the calibration unit 221, and supplies to the encoding device 172 of the camera-related information as metadata.
[0158]
　In step S54, the synchronization shift detector 222, for each reference camera, based on the 2-dimensional image data of the base camera and the reference camera which is supplied from the calibration unit 221, of the two-dimensional image data of the reference camera with respect to a reference camera synchronization detecting a shift. Also, synchronization shift detector 222 detects 0 as the synchronization deviation of the two-dimensional image data of the reference camera.
[0159]
　In step S55, the synchronization shift detector 222 supplies a synchronization shift information of the detected synchronization error in the encoding device 172 as metadata. Also, synchronization shift detector 222 supplies the two-dimensional image data of each camera 211 supplied from the calibration unit 221 to the depth generation unit 223.
[0160]
　In step S56, the depth generation unit 223, by performing a stereo matching on the two-dimensional image data of each camera 211 supplied from the detection unit 222 synchronization shift, and generates a depth image data of each camera 211.
[0161]
　In step S57, the depth generation unit 223 supplies the two-dimensional image data and depth image data of each camera 211 to the encoding device 172.
[0162]
　In step S58, the encoder 172, two-dimensional image data and depth image data of each camera 211 supplied from the image pickup apparatus 171, and the metadata is encoded to generate an encoded stream.
[0163]
　In step S59, the encoding device 172 transmits the generated encoded stream to the decoding device 173. Then, the process is terminated.
[0164]
　(Configuration example of the decoding unit)
　15 is a block diagram showing a configuration example of the decoding unit 191 of FIG. 10.
[0165]
　In the example of FIG. 15, the configuration of the encoding device 172 is a structure of FIG. 13, the coding scheme of the coded stream transmitted from the encoding device 172 is to AVC or HEVC scheme. Decoding unit 191 of FIG. 15 is composed of the 2N decoding unit 261-1 through 261-2N and the output unit 262.
[0166]
　Decoding processing unit 261-1 through 261-2N of the decoding unit 191, an encoded stream of the two-dimensional image data and depth image data of each camera 211 supplied from the receiving unit 31, respectively, corresponding to the AVC scheme or HEVC scheme It is decoded in a manner that. Decoding processing unit 261-1 through 261-2N is supplied to the output unit 262 and the 2-dimensional image data or depth image data, and a camera-related information and the synchronization shift information constituting the metadata of each camera 211 obtained as a result of the decoding to.
[0167]
　The output unit 262 (synchronization processing unit), based on the synchronization shift information supplied from the decoding section 261-1 to 261-2N, for each reference camera, two-dimensional image data and depth image data of the base camera and the reference camera the synchronously supplied to the conversion unit 192 of FIG. 10.
[0168]
　For example, num_units_in_tick reference camera included in the synchronization shift information is the 1001, Delta_num_units_in_tick is 2002, when time_scale and Delta_time_scale is 60000, the output unit 262, two-dimensional image data and depth image data of the reference camera 2 and supplies to the converter 192 delayed by a frame. The output unit 262 supplies the camera-related information supplied from the decoding section 261-1 to 261-2N to the converter 192.
[0169]
　Thus, since the two-dimensional image data and depth image data of each camera 211 is supplied to the converter 192 is synchronized, to improve product accuracy of three-dimensional data in the conversion unit 192.
[0170]
　(Process description of the decoding apparatus)
　FIG. 16 is a configuration of the conversion unit 192 of FIG. 10, the decoding when the three-dimensional data is the same as that of FIG. 8, except that not corrected using the occlusion 3D data it is a flowchart illustrating a decoding process of the apparatus 173. The decoding process is, for example, when the encoded stream has been transmitted from the encoding device 172, is started.
[0171]
　In step S71 of FIG. 16, receiving unit 31 of the decoding device 173 receives an encoded stream transmitted from the encoding apparatus 172, and supplies the decoding section 191.
[0172]
　In step S72, the decoding unit 191, an encoded stream supplied from the receiving unit 31 is decoded by method corresponding to the coding method in the encoder 172.
[0173]
　In step S73, the decoding unit 191, based on the synchronization shift information of the metadata obtained as a result of decoding, for each reference camera, to synchronize the two-dimensional image data and depth image data of the base camera and the reference camera conversion supplied to the part 192. The output unit 262 supplies the camera-related information of the metadata to the conversion unit 192.
[0174]
　In step S74, the conversion unit 192, two-dimensional image data supplied from the decoding unit 191, by using depth image data, and a camera parameter included in the camera-related information, like the imaging device 11, the three-dimensional data of an object to generate.
[0175]
　Processing in steps S75 through S79 is the same as the processing in step S35 to S39 in FIG. 9, description will be omitted.
[0176]
　As described above, in the transmission system 170, for transmitting the two-dimensional image data and depth image data obtained by each camera 211 as it is encoded, the synchronization of the two-dimensional image data and depth image data of each camera 211 is taken there is a case to not.
[0177]
　However, the transmission system 170, the imaging device 171, detects the synchronization deviation of the two-dimensional image data of each camera, the encoding device 172, the synchronization shift information of the detected synchronization shift, the two-dimensional image data and depth image It is transmitted with the data. Thus, the decoding apparatus 173 can, based on the synchronization shift information to synchronize the two-dimensional image data and depth image data of each camera 211. As a result, the decoding apparatus 173, using the two-dimensional image data and depth image data of N camera 211 is in sync, it is possible to generate three-dimensional data with high accuracy.
[0178]
　
　(Configuration of the third embodiment of the transmission system)
　FIG. 17 is a block diagram showing a configuration example of a third embodiment of a transmission system according to the present disclosure.
[0179]
　In the configuration shown in FIG. 17 are denoted by the same reference numerals are the same as the structural elements of FIG. Redundant description will be omitted as appropriate.
[0180]
　Configuration of the transmission system 280 of FIG. 17, a new image pickup apparatus 281, that the encoding device 282 and the synthesizer 283, is provided, that the decoder 13 replaces the decoder 284, and is generated by the encoding device 12 2-dimensional image data and the viewpoint of the depth image data that is, that it is part of a plurality of viewpoints corresponding to the predetermined display image generation method is different from the configuration of a transmission system 10 of FIG.
[0181]
　In the transmission system 280, a plurality of part of viewpoints of the viewpoint corresponding to the predetermined display image generation system (hereinafter, referred to as a first viewpoint group) coded streams are generated by the encoding device 12, the other portion viewpoint (hereinafter, the referred second viewpoint group) coded streams are generated by the encoding device 282, both of the coded stream are combined (merged).
[0182]
　Specifically, the imaging device 281 of the transmission system 280 is configured similarly to the imaging device 11 of FIG. 1, at least partially captures a subject is identical to the subject in the imaging device 11, the three-dimensional data of the object and supplies to the encoding apparatus 282.
[0183]
　Encoding apparatus 282 is constituted by the conversion unit 21 and configured similarly to conversion unit 291, the encoding unit 22 and configured similarly to the encoding unit 292 and transmission unit 23 configured similarly to the transmission unit 293, that. Encoding apparatus 282, from the three-dimensional data to generate a two-dimensional image data and depth image data of the second aspect groups, encoded and transmitted to the synthesizer 283.
[0184]
　Synthesizer 283, encoded streams of the first viewpoint group that is transmitted from the encoding device 12 (hereinafter, a first partial code stream) of the second viewpoint group transmitted from the encoding apparatus 282 coded stream (hereinafter, a second portion encoded stream) receive. Synthesizer 283 detects the synchronization deviation of the second partial code stream for the first partial code stream, generates viewpoint group synchronization shift information indicating a synchronization shift of the second encoded stream.
[0185]
　Deviation viewpoint group synchronization information, for example, a num_units_in_tick and time_scale represent the second viewpoint group frame rate, constituted by Delta_num_units_in_tick and Delta_time_scale represents the sync.
[0186]
　Synthesizer 283, a second portion encoded stream metadata viewpoint group sync include information. Synthesizer 283 synthesizes the second partial code stream and a first portion encoded stream shifted viewpoint elementary synchronous information is disposed, the encoded streams of the plurality of viewpoints corresponding to the predetermined display image generation method It generates and transmits to the decoder 284.
[0187]
　Configuration of the decoding apparatus 284 are that the decoding unit 32 replaces the decoder 301 is different from the configuration of the decoding device 13 of FIG. 1. Decoder 301 of the decoding device 284, that the shift synchronization information replaces the viewpoint group synchronization deviation information, and, together with the camera-related information, except to provide occlusion 3D data of the metadata to the converter 33, are the same as decoder 191 of FIG. 10, description thereof will be omitted.
[0188]
　(Configuration Example of synthesis apparatus)
　FIG. 18 is a block diagram showing a configuration example of a synthesis apparatus 283 of FIG. 17.
[0189]
　Synthesizer 283 in FIG. 18 is composed of a decoding unit 321, decoding unit 322, synchronization shift detector 323 and the metadata addition unit 324.
[0190]
　Decoding unit 321 of the synthesizer 283 receives the first partial code stream transmitted from the encoding device 12. Decoding unit 321, a first portion encoded stream received (in the example of FIG. 18 MVCD) coding method in the encoding unit 22 decodes a manner corresponding to the two-dimensional image of a first viewpoint group data, to generate a depth image data, and metadata. Decoding unit 321 supplies the two-dimensional image data of one view of the first viewpoint group generated in the shift detection unit 323 synchronization.
[0191]
　Decoding unit 322 receives the second partial code stream transmitted from the encoding apparatus 282. Decoding unit 322, a second portion encoded stream received (in the example of FIG. 18 MVCD) coding method in the encoding unit 292 decodes in a manner that corresponds to the two-dimensional image of the second viewpoint group data, to generate a depth image data, and metadata. Decoding unit 322 supplies the two-dimensional image data of one view of the second viewpoint group generated in the shift detection unit 323 synchronization.
[0192]
　Synchronization shift detector 323, based on the 2-dimensional image data supplied from the decoding unit 321 and the decoding unit 322 detects the synchronization deviation of the two-dimensional image data of the second viewpoint group with respect to the first viewpoint group.
[0193]
　Detection method of the synchronization deviation may be the same as the method of detecting a synchronization shift in the second embodiment, it may be different.
[0194]
　Further, the detection of sync, as well as two-dimensional image data, may be used camera parameters and depth image data. In this case, for example, synchronization shift detector 323, based on the camera parameters included in the camera-related information, one viewpoint and the time of one viewpoint of the second aspect groups of the first viewpoint group converting two-dimensional position of the feature point of the two-dimensional image data of the three-dimensional position. Synchronization shift detector 323, the difference between the time of the two-dimensional image data when the difference between the three-dimensional positions of feature points of the two views is minimized, the two-dimensional image of a second viewpoint group with respect to the first viewpoint group detecting a synchronization shift of the data.
[0195]
　Synchronization shift detector 323, a viewpoint group synchronization shift information representing the detected synchronization error, and supplies the metadata addition unit 324.
[0196]
　The metadata addition unit 324 arranges the viewpoint group synchronization shift information supplied from the detection unit 323 synchronization shift, the second partial code stream transmitted from the encoding device 282 as metadata. The metadata addition unit 324, shift viewpoint group synchronization information to synthesize a second portion encoded stream which is substituted as metadata, a first portion encoded stream transmitted from the encoding device 12. The metadata addition unit 324 (transmitting unit) the encoded stream of the plurality of viewpoints corresponding to the predetermined display image generation method obtained as a result of the synthesis is transmitted to the decoding device 284.
[0197]
　(Process of the description of the synthetic device)
　imaging device 11 and the encoding device 12 of FIG. 17, the encoding processing of the image pickup device 281 and the encoding unit 282 is similar to the encoding process of FIG. 5, description will be omitted .
[0198]
　Figure 19 is a flowchart for explaining the synthesis processing of the synthesis apparatus 283 of FIG. 18.
[0199]
　In step S91 of FIG. 19, the decoding unit 321 of the synthesizer 283, the first partial code stream transmitted from the encoding device 12 is decrypted with method corresponding to the coding method in the encoding unit 22. Decoding unit 321 supplies the two-dimensional image data of one view of the first viewpoint group obtained as the result of decoding to the shift detection unit 323 synchronization.
[0200]
　In step S92, the decoding unit 322, the second partial code stream transmitted from the encoding apparatus 282 and decoding in a manner corresponding to the coding method in the encoding unit 292. Decoding unit 322 supplies the two-dimensional image data of one view of the second viewpoint group obtained as the result of decoding to the shift detection unit 323 synchronization.
[0201]
　In step S93, synchronization shift detector 323, based on the 2-dimensional image data supplied from the decoding unit 321 and the decoding unit 322, a synchronization shift of the two-dimensional image data of the second viewpoint group with respect to the first viewpoint group To detect. Synchronization shift detector 323 supplies the viewpoint group synchronization shift information representing the detected synchronization shift to the metadata addition unit 324.
[0202]
　In step S94, the metadata addition unit 324 arranges the viewpoint group synchronization shift information supplied from the detection unit 323 synchronization shift, the second partial code stream transmitted from the encoding device 282 as metadata.
[0203]
　In step S95, the metadata addition unit 324 combines the first partial code stream transmitted from the encoding device 12, a second portion encoded stream including a deviation viewpoint group synchronization information. The metadata addition unit 324, the encoded streams of the plurality of viewpoints corresponding to the predetermined display image generation method obtained as a result of the synthesis is supplied to the decoder 284, the process ends.
[0204]
　Decoding processing of the decoding device 284, that the shift synchronization information replaces the viewpoint group synchronization deviation information, and, occlusion three-dimensional data by the processing in step S72 is also generated, after the process of step S74, the similar to step S34 in FIG. 9 except that 3-dimensional data is corrected using the occlusion 3D data is similar to the decoding process in FIG. 16.
[0205]
　As described above, in the transmission system 280, a first portion encoded stream generated by the encoding apparatus 12, the decoder 284 is a second portion encoded stream generated is synthesized by the encoding device 282 It is transmitted. Therefore, there is a case where synchronization of the first viewpoint group and the second viewpoint groups encoded stream corresponding to a predetermined display image generation method obtained as a result of the synthesis is not achieved.
[0206]
　However, the transmission system 280, synthesizer 283, a two-dimensional first partial code stream and once decodes the second partial code stream, the first viewpoint group and the second viewpoint group resulting detecting a synchronization shift of the image data. Then, synthesizer 283, a viewpoint group synchronization shift information representing the detected synchronization deviation is transmitted with the first portion encoded stream and a second partial code stream. Accordingly, the decoding device 284 may be based on the viewpoint group sync information to synchronize the two-dimensional image data and depth image data of the first viewpoint group and the second viewpoint group. As a result, the decoding device 284 uses the two-dimensional image data and depth image data of the first viewpoint group and the second viewpoint group is in sync, it is possible to generate three-dimensional data with high accuracy.
[0207]
　
　(Configuration Example of the synthesis apparatus according to the fourth embodiment of the transmission system)
　structure of the fourth embodiment of the transmission system embodying the present disclosure, synthesizing apparatus 283 instead synthesizer 340, point conversion unit 33 of the decoding device 284 replaces the transform unit 380, and viewpoint group sync information, except in place of the coordinate transformation information is the same as that of the transmission system 280 of FIG. 17. Therefore, the following is a description with synthesizer 340 for converting unit 380 only.
[0208]
　Figure 20 is a block diagram showing a configuration example of a synthesizer 340.
[0209]
　In the configuration shown in FIG. 20, the same reference sign is assigned to the same configuration as the configuration in FIG. 18. Redundant description will be omitted as appropriate.
[0210]
　Configuration of the synthesizer 340 of Figure 20, synchronization shift detector 323, the metadata addition unit 324, coordinate transformation data generating unit 341, the point that replaces the metadata addition unit 342, different from the configuration of the synthesizer 283 of FIG. 18 . Synthesizer 340 does not detect the synchronization deviation of the two-dimensional image data for each viewpoint, 3-D the first three-dimensional coordinate system is a three-dimensional coordinate system of the first viewpoint group second viewpoint group generating a coordinate transformation data to be converted into the second three-dimensional coordinate system is a coordinate system.
[0211]
　Specifically, coordinate transformation data generating unit 341 of the synthesizer 340, on the basis of the two-dimensional image data of the first viewpoint group generated by the decoding of the decoder 322, for each viewpoint of the first viewpoint group, obtaining a two-dimensional position of the feature point.
[0212]
　The coordinate transformation data generating unit 341, based on the camera parameter contained in the second aspect groups camera-related information generated by the decoding of the decoding unit 321, the two-dimensional image data, and the depth image data, the above-described formula by (1), we obtain the three-dimensional position in the second three-dimensional coordinate system of the feature point.
[0213]
　Coordinate transformation data generating unit 341, for each viewpoint of the first viewpoint group, based on the three-dimensional position in the two-dimensional position and a second three-dimensional coordinate system of the feature point, to generate the coordinate transformation data. Coordinate transformation data generating unit 341 supplies the coordinate transformation information including a coordinate transformation data for each viewpoint of the first viewpoint group in the metadata addition unit 342.
[0214]
　The metadata addition unit 342 arranges the coordinate conversion information supplied from the coordinate transformation data generating unit 341, the first partial code stream transmitted from the encoding device 12 as metadata. The metadata addition unit 342, coordinate conversion information to synthesize a first portion encoded streams arranged as a metadata, the second partial code stream transmitted from the encoding apparatus 282. The metadata addition unit 342 (transmitting unit) the encoded stream of the plurality of viewpoints corresponding to the predetermined display image generation method obtained as a result of the synthesis is transmitted to the decoding device 284.
[0215]
　(Description of generation processing of coordinate conversion data)
　Figure 21 is a diagram explaining a process for generating a coordinate transformation data by coordinate transformation data generating unit 341 of FIG. 20.
[0216]
　In the example of FIG. 21, first three-dimensional coordinate system, each other axis x perpendicularly intersects A to z A a coordinate system with the coordinate axes, second three-dimensional coordinate system, with each other perpendicularly intersects the axis x B to z B as coordinate, a coordinate system different from the first three-dimensional coordinate system.
[0217]
　It is taken by the virtual camera 361 in one view of the first viewpoint group, first three-dimensional position P in the three-dimensional coordinate system 1 two-dimensional position P on the a characteristic point image A , the following represented by the formula (3).
[0218]
[Number 3]

[0219]
　R Cama | t Cama is of the metadata of the first portion encoded stream, a camera parameter at the first three-dimensional coordinate system of the virtual camera 361.
[0220]
　On the other hand, are imaged by the virtual camera 361, three-dimensional position in the first three-dimensional coordinate system is P 1 is, 3-dimensional position in the second three-dimensional coordinate system is P 1 on the image feature point is' P B is represented by the following formula (4).

claims

[Claim 1]From the three-dimensional data of an object generated from the 2-dimensional image data of a plurality of first viewpoint, the pixels for two-dimensional image data and the third aspect of the plurality of second viewpoint corresponding to the predetermined display image generation method a two-dimensional data generation unit for generating a depth image data representing the depth direction of the position of the subject of,
　the depth image data and the two-dimensional image data of said plurality that are generated by the two-dimensional data generation unit second viewpoint an encoding unit for encoding and
　a transmission unit for transmitting the depth image data and the two-dimensional image data of said plurality of second viewpoint is encoded by the encoding unit
　encoding device comprising a.
[Claim 2]
　The different plurality of first viewpoint and said plurality of second viewpoint
　configured to
　encoding apparatus according to claim 1.
[Claim 3]
　It said plurality of second viewpoint is present at regular intervals on the same circle
　constructed as
　encoding apparatus according to claim 1.
[Claim 4]
　Wherein at least two viewpoints of the plurality of second viewpoint is arranged in a substantially horizontal direction
　is configured to
　encoding apparatus according to claim 1.
[Claim 5]
　The transmission unit, information identifying the two viewpoints arranged in the horizontal direction to the transmission of the plurality of second viewpoint
　configured to
　encoding apparatus according to claim 4.
[Claim 6]
　The transmission unit transmits the three-dimensional data of the occlusion region in the two-dimensional image data of said plurality of second viewpoint
　constructed as
　encoding apparatus according to claim 1.
[Claim 7]
　The two-dimensional data generation unit, based on camera parameters of the plurality of second viewpoint to generate a two-dimensional image data of said plurality of second viewpoint from the 3-dimensional data, the third viewpoint of the camera based on the parameters, to generate the depth image data,
　the transmission unit transmits the camera parameters of the plurality of second viewpoint and the third viewpoint
　is configured as
　codes of claim 1 apparatus.
[8.]
　The number of predetermined display image generation method is the plurality
　configured to
　encoding apparatus according to claim 1.
[Claim 9]
　It said plurality of said second viewpoint third viewpoint is the same
　configured to
　encoding apparatus according to claim 1.
[Claim 10]
　Coding apparatus,
　from the three-dimensional data of an object generated from the 2-dimensional image data of a plurality of first viewpoint, the two-dimensional image data of a plurality of second viewpoint corresponding to the predetermined display image generation method 3 a two-dimensional data generation step of generating the depth image data representing the depth direction of the position of the subject of the respective pixels with respect to the viewpoint of,
　two-dimensional second viewpoint is generated in the plurality by processing of the 2-dimensional data generation step an encoding step of encoding the image data the depth image data,
　a transmission step of transmitting the depth image data and the two-dimensional image data of the encoded plurality of second viewpoint by the process of the encoding step
　encoding method, including.
[Claim 11]
　And the encoded data of the two-dimensional image data of a plurality of first viewpoint corresponding to the predetermined display image generation method, the encoded data of the depth image data indicating the position in the depth direction of the object for each pixel for the second viewpoint a decoding unit for decoding, the
　use and the two-dimensional image data of said plurality of first viewpoint obtained as a result of decoding by the decoder and said depth image data, three-dimensional data to generate three-dimensional data of an object a generating unit,
　on the basis of the generated the three-dimensional data by three-dimensional data generation unit, and the 2-dimensional data generation unit for generating two-dimensional image data as display image data in the predetermined display image generation method
　decoding comprising apparatus.
[Claim 12]
　It said plurality of first viewpoint is present at regular intervals on the same circle
　constructed as
　a decoding apparatus according to claim 11.
[Claim 13]
　Least two viewpoints of said plurality of first viewpoint, arranged substantially horizontally
　configured as
　decoding apparatus according to claim 11.
[Claim 14]
　Receiving unit for receiving information identifying two viewpoints arranged in the horizontal direction of the plurality of first viewpoint
　, further comprising a
　decoding device according to claim 13.
[Claim 15]
　The depth image data and the two-dimensional image data of said plurality of first viewpoint is generated from three-dimensional data of an object generated from the 2-dimensional image data of a plurality of third aspect
　configured as
　claim decoding apparatus according to 11.
[Claim 16]
　The three-dimensional data generation unit, the two-dimensional image data of said plurality of first viewpoint, using a 3-dimensional data of the occlusion region in the two-dimensional image data of the depth image data, and the plurality of first viewpoint, generating a 3-dimensional data of the object
　is constructed as
　a decoding apparatus according to claim 15.
[Claim 17]
　The three-dimensional data generation unit, based on camera parameters of the plurality of first viewpoint and the second viewpoint, to generate a three-dimensional data of the object
　is constructed as
　a decoding apparatus according to claim 11 .
[Claim 18]
　The number of the predetermined display image generation method is the more
　configured as
　decoding apparatus according to claim 11.
[Claim 19]
　It said plurality of first viewpoint and said second viewpoint is the same
　constructed as
　a decoding apparatus according to claim 11.
[Claim 20]
　Decoding apparatus,
　the encoded data of the two-dimensional image data of a plurality of first viewpoint corresponding to the predetermined display image generation method, the depth image data representing the depth direction of the position of the object in each pixel for the second viewpoint a decoding step of decoding the encoded data,
　by using the first two-dimensional image data of the first viewpoint of the decoding by processing resulting in said plurality said decoding step and said depth image data, three-dimensional data of the object and a three-dimensional data generation step of generating,
　based on the generated the three-dimensional data by processing of the three-dimensional data generation step, to generate two-dimensional image data as display image data in the predetermined display image generation method 2 a dimensional data generation step
　decoding method comprising.