Technical field
[0001]
　The present disclosure, generator and generation method, and relates to reproduction apparatus and a reproduction method, in particular, the quality of all the viewing direction of the display image generated by using the omnidirectional image to be able to the same extent generating apparatus and method for generating and, and to a reproducing apparatus and a reproducing method.
BACKGROUND
[0002]
　As the method of reproducing omnidirectional image, by perspective projection by mapping the celestial sphere image in the 3D model, to generate an image of the viewing direction of the visual field range of the viewer as a display image, there is a method of displaying. The omnidirectional image is an image obtained by mapping the image in the horizontal direction 360 degrees around the and vertical directions around 180 ° 3D model.
[0003]
　In this way, only a part of the celestial sphere image is used to generate a display image. For example, if the viewing angle in the horizontal direction is 90 degrees, for the horizontal direction, only 1/4 of the area of ​​the entire omnidirectional image is used to generate a display image. The same applies to the vertical direction. Thus, the ratio of the area used for generating the display image to the entire celestial sphere image is small.
[0004]
　However, the line-of-sight direction of the viewer, because they are indicated by the viewer during playback, the recording apparatus, it is difficult to record only the area used for generating the display image of the celestial sphere image. Accordingly, the recording device records the entire omnidirectional image, by requiring only the region reproducing device is used to generate a display image on the recording apparatus, the transmission amount of the celestial sphere image from the recording apparatus to the reproduction apparatus it is desired to reduce.
[0005]
　Therefore, by dividing the omnidirectional image into a plurality of regions, by generating a coded stream in each divided region, devised to enable the transmission to the playback device only from the recording apparatus encoded streams of the predetermined splitting area is (e.g., see Patent Document 1).
[0006]
　Moreover, the omnidirectional entire image of low resolution, the overlap and separately encoded high resolution omnidirectional image in each divided region having mutually corresponds to the viewing direction of the viewer with the entire celestial sphere image of the low resolution transmitting only the encoded stream of high-resolution full spherical images of the divided regions from the recording apparatus to the reproduction apparatus is also devised (e.g., see Patent Document 2).
[0007]
　In this case, the recording apparatus, the high resolution of the celestial sphere image, since it is sufficient transmit only encoded stream of the divided region used for the generation of the display image, high-resolution full spherical images of all the divided areas it is possible to reduce the transmission amount in comparison with the case of transmitting. Moreover, the generation of the display image, and if the celestial sphere image in the region other than the divided area corresponding to the encoded stream of high-resolution full spherical image which has been transmitted is used, suddenly the viewing direction of the viewer even when changes, it is possible to generate a display image using the omnidirectional image of lower resolution. Furthermore, or in the number the number of the divided regions, by and set a region spanning the end of the celestial sphere image in the divided area, high resolution of the celestial sphere image used for generating the display image all it is possible to increase the proportion of celestial sphere image.
CITATION
Patent Document
[0008]
Patent Document 1: JP 2001-298652 Patent Publication
Patent Document 2: JP 2016-15705 JP
Summary of the Invention
Problems that the Invention is to Solve
[0009]
　However, omnidirectional image, for example, when the image in the horizontal direction 360 degrees around the and vertical directions around 180 degrees is the image by equirectangular the mapped spherical shape on the sphere of the celestial sphere image distorted closer to pole portion (portion corresponding to the north and south poles in the sphere of the globe). For example, a rectangular area of ​​the celestial sphere image, when mapped near pole of the sphere, the shape of the sphere is distorted in a fan shape.
[0010]
　Therefore, even with the same shape and size high resolution omnidirectional image of the divided region of the position on the celestial sphere image divided regions, different shapes and sizes on the perspective projection plane. As a result, different ratio of the area of ​​high-resolution full spherical images of the divided areas in the display image by the gaze direction, it has been difficult to produce a display image of the same degree of quality for all viewing direction.
[0011]
　The present disclosure has been made in view of such circumstances, and is to the quality of all the viewing direction of the display image generated by using the omnidirectional image can be to the same extent .
Means for Solving the Problems
[0012]
　Generating apparatus of the first aspect of the present disclosure, a resolution reduction unit for low resolution of the omnidirectional image, by projecting the omnidirectional image mapped to the 3D model into a plurality of two-dimensional plane, a generating device and a projection unit for generating a plurality of images.
[0013]
　Generating method of the first aspect of the present disclosure, corresponding to the generated device of the first aspect of the present disclosure.
[0014]
　In the first aspect of the present disclosure, omnidirectional image is low resolution, by projecting the omnidirectional image mapped to the 3D model into a plurality of two-dimensional plane, a plurality of images are generated .
[0015]
　Reproducing apparatus of the second embodiment of the present disclosure, at least one image among the plurality of images generated by projecting the omnidirectional image mapped to the 3D model into a plurality of two-dimensional plane, a low resolution a receiving unit for receiving the reduction has been the omnidirectional image, and wherein is the image and lower resolution received by the receiving unit based on at least one of the celestial sphere image, the drawing unit that generates a display image a reproducing apparatus comprising a.
[0016]
　The method of reproducing the second aspect of the present disclosure, corresponding to the reproduction apparatus of the second aspect of the present disclosure.
[0017]
　In the second aspect of the present disclosure, at least one image among the plurality of images generated by projecting the omnidirectional image mapped to the 3D model into a plurality of two-dimensional plane, the lower resolution wherein the omnidirectional image is received with the image and lower resolution has been on the basis of at least one of the omnidirectional image, the display image is generated.
[0018]
　The reproduction apparatus generating device and the second side of the first aspect can be realized by causing a computer to execute a program.
[0019]
　Further, in order to realize a reproducing apparatus of the generator and a second side surface of the first side, a program to be executed by a computer, by transmitting via a transmission medium or by being recorded on a recording medium, provided can do.
Effect of the invention
[0020]
　According to a first aspect of the present disclosure, it is possible to generate an image. Further, according to the first aspect of the present disclosure, the quality of all the viewing direction of the display image generated by using the omnidirectional image may generate an image so that it can be to the same extent.
[0021]
　According to a second aspect of the present disclosure, it is possible to reproduce an image. According to the second aspect of the present disclosure, the quality of all the viewing direction of the display image generated by using the omnidirectional image can be to the same extent.
[0022]
　Here, the advantages described in the present invention is not necessarily limited, it may be any of the effects described in the present disclosure.
BRIEF DESCRIPTION OF THE DRAWINGS
[0023]
FIG. 1 is a diagram illustrating the distortion of the shape of the sphere celestial sphere image.
It is a block diagram showing a configuration example of a first embodiment of a distribution system according to the [2] The present disclosure.
3 is a block diagram showing a configuration example of a production apparatus of FIG.
Is a diagram showing a first example of FIG. 4 two-dimensional plane.
5 is a diagram for explaining the high-resolution image.
6 is a diagram showing a configuration example of a two-dimensional plane table in the first embodiment.
Is a flowchart illustrating a process of generating generator of FIG. 7 Fig.
Is a block diagram showing a configuration example of FIG. 8 the distribution server and the playback apparatus of FIG.
It is a diagram illustrating the mapping by the mapping processing unit of FIG. 9 Fig.
Is a diagram illustrating an example of FIG. 10 displays the image.
11 is a flowchart illustrating the playback processing of the playback apparatus of FIG.
It is a diagram showing a second example of FIG. 12 a two-dimensional plane.
13 is a diagram illustrating an example of a method of generating a full spherical images.
14 is a diagram showing another example of the low-resolution image.
15 is a diagram illustrating the coordinate system of the projection surface.
16 is a diagram illustrating a tan axis projection.
17 is a diagram for explaining a projection point in the perspective projection and tan axis projection.
18 is a diagram showing an example of each pixel of the celestial sphere image on a high-resolution image generated by the perspective projection.
19 is a diagram showing an example of each pixel of the celestial sphere image on tan axis high-resolution image generated by the projection.
FIG. 20 is a diagram showing another example of each pixel of the celestial sphere image on a high-resolution image generated by the perspective projection and tan axis projection.
21 is a diagram illustrating a configuration example of a two-dimensional plane table in the second embodiment.
22 is a block diagram showing a configuration example of hardware of a computer.
FIG. 23 is a block diagram showing an example of a schematic configuration of a vehicle control system.
It is an explanatory diagram showing an example of an installation position of FIG. 24 outside information detection unit and the imaging unit.
DESCRIPTION OF THE INVENTION
[0024]
　Hereinafter, embodiments of the premise and the disclosure of the present disclosure (hereinafter, referred to as embodiments) will be described. The description will be made in the following order.
　0. Premise of the present disclosure (FIG.
　1) 1. First Embodiment: distribution system (FIGS. 2 to
　14) 2. Second Embodiment: distribution system (FIGS. 15 to
　21) 3. Third Embodiment: a computer (Fig.
　22) 4. Application Example (FIGS. 23 and 24)
[0025]
　
　1, omnidirectional image is generated when the image in the horizontal direction 360 degrees around the and vertical directions around 180 degrees is the image by equirectangular mapped sphere, is a diagram illustrating the distortion of the shape of the sphere celestial sphere image.
[0026]
　Left A and 1 B of Figure 1 shows an omnidirectional image 1, the horizontal lines and vertical lines of the celestial sphere image 1, respectively, latitude, and longitude. The central sphere 2 A and in Figure 1 B of FIG. 1 is a sphere omnidirectional image 1 is mapped to the inner surface. Furthermore, the right side of figure A and 1 B of FIG. 1 is a view of the omnidirectional image 1 mapped to sphere 2 from the inside of the sphere 2.
[0027]
　As shown on the left side of A in FIG. 1, the rectangular area 1A in the vicinity of the center of the celestial sphere image 1 is mapped on the inner surface of the sphere 2 as shown in the middle of A 1, FIG. 1 of as shown on the right side of a, a side perpendicular to the pole direction of the region 1A as viewed from the inside of the sphere 2, distortion to the same extent, a curve. That is, the region 1A is an area near the center of the celestial sphere image 1, the mapping position of the pole section direction, are separated by substantially the same distance from the bipolar portion. Therefore, a side perpendicular to the pole direction of the region 1A as viewed from the inside of the sphere 2, distorted equally.
[0028]
　On the other hand, as shown on the left side of the B 1, which are identical in shape and size and the region 1A, the upper region 1B of the omnidirectional image 1, the sphere 2 as shown in the middle of the B 1 When mapped on the inner surface, as shown on the right side of the B 1, the shape of the region 1B as seen from the inside of the sphere 2 will sector. That is, the region 1B is a region in the vicinity of the upper omnidirectional image 1, the mapping position of the pole portion direction, close to the pole on the upper side in the figure, away from extreme portion of the lower side in FIG. Therefore, vertical two sides distortion in the pole direction of the region 1B as seen from the inside of the sphere 2, a curve, the amount of distortion, towards the side is larger closer than the side farther from the pole portion.
[0029]
　As described above, omnidirectional image, when the image in the horizontal direction 360 degrees around the and vertical directions around 180 degrees is the image by equirectangular the mapped spherical shape of the celestial sphere image 1 and size even when the same, by the position of the celestial sphere image 1, different shapes and sizes on the sphere 2. Therefore, even different shapes and sizes on a two-dimensional plane by perspective projection of the celestial sphere image 1 mapped to sphere 2.
[0030]
　
　(Configuration example of a first embodiment of the distribution system)
　Figure 2 is a block diagram showing a configuration example of a first embodiment of a distribution system according to the present disclosure.
[0031]
　Delivery system 10 of FIG. 2 is constituted by the imaging device 11, generator 12, the distribution server 13, the network 14, the reproducing apparatus 15 and the head-mounted display 16,. Delivery system 10 generates a celestial sphere image from the images picked up by the imaging device 11, and displays the display image of the field of view of the viewer by using the omnidirectional image.
[0032]
　Specifically, the photographing device 11 of the distribution system 10 is composed of six cameras 11A-1 to 11A-6. In the following description, when it is not necessary to distinguish the camera 11A-1 to 11A-6, that are collectively camera 11A.
[0033]
　Each camera 11A captures moving images. Imaging device 11 supplies the generator 12 to the six directions of moving images captured by the cameras 11A as a photographic image. The number of cameras imaging device 11 is provided, if a plurality, but is not limited to six.
[0034]
　Generator 12, by a method using equirectangular be reduced resolution to generate a celestial sphere image from the captured image supplied from the imaging device 11. Generator 12, the low-resolution image is a celestial sphere image is lower resolution (YUV image) encoding, to generate a low-resolution stream one.
[0035]
　Also, generator 12 maps the celestial sphere image to a sphere as a 3D model, the omnidirectional image mapped to sphere into five viewing direction the corresponding two-dimensional plane, as the focus of the center of the sphere by perspective projection, it generates five images. Generator 12, the five images were coded respectively as a high resolution image (YUV image), to generate a high-resolution stream five.
[0036]
　Further, generator 12, the position of the two-dimensional plane corresponding to the high resolution image, gradient, and generates a two-dimensional plane information indicating the size. Generator 12, a single low-resolution stream, upload five high-resolution stream, and a two-dimensional plane information to the distribution server 13.
[0037]
　Distribution server 13 is connected to the reproducing apparatus 15 via the network 14. Distribution server 13 stores one low-resolution streams uploaded from generator 12, five high-resolution stream, and a two-dimensional plane information. Distribution server 13, in response to a request from the reproducing apparatus 15, the low-resolution stream stored, high resolution stream, and a two-dimensional plane information to the reproducing apparatus 15 via the network 14.
[0038]
　Reproducing apparatus 15, a single low-resolution stream and the two-dimensional plane information to the distribution server 13, and request via the network 14, one low-resolution stream and the two-dimensional plane information transmitted in response to the request the receive.
[0039]
　The reproduction apparatus 15 has a built-in camera 15A, photographs the markers 16A attached to the head mounted display 16. The reproduction device 15 based on the captured image of the marker 16A, detects the viewer's viewing position in the coordinate system of the 3D model. Further, the reproducing apparatus 15, the detection result of the gyro sensor 16B of the head-mounted display 16, receives from the head-mounted display 16. Reproducing apparatus 15, based on the detection result of the gyro sensor 16B, to determine the viewing direction of the viewer in the coordinate system of the 3D model. Reproducing apparatus 15, based on the viewing position and viewing direction, to determine the viewer's field of view that is located inside the 3D model.
[0040]
　The reproduction apparatus 15, based on a two-dimensional plane information and the viewing range of the viewer, a single high-resolution stream of five high-resolution stream request via the network 14, in response to the request receive a high resolution stream one transmitted.
[0041]
　Reproducing apparatus 15 decodes the one of the low-resolution streams and one of the high resolution stream received. Reproducing apparatus 15 maps the low-resolution image obtained as a result of decoding to a sphere as a 3D model by mapping the high-resolution image on a two-dimensional plane as a 3D model of the internal sphere, to generate a 3D model image .
[0042]
　The reproduction apparatus 15 includes, as a focus of the viewing position, by perspective projection of 3D model image the field of view of the viewer, generates an image of the visual field range of the viewer as a display image. Reproducing apparatus 15 supplies the display image on the head mounted display 16.
[0043]
　Head-mounted display 16 is mounted on the head of the viewer, displaying the display image supplied from the reproducing apparatus 15. The head-mounted display 16, the marker 16A is attached to be captured by the camera 15A. Therefore, the viewer, a head-mounted display 16 in a state of mounting on the head, it is possible to specify the viewing position by moving. Further, the head-mounted display 16, a gyro sensor 16B is incorporated, the detection result of the angular velocity by the gyro sensor 16B is transmitted to the reproducing apparatus 15. Therefore, the viewer, by rotating the head wearing the head-mounted display 16, it is possible to specify the line-of-sight direction.
[0044]
　In the distribution system 10, delivery method from the distribution server 13 to the reproducing apparatus 15 may be any method. Delivery method, for example, MPEG-DASH - if it is (Moving Picture Experts Group phase Dynamic Adaptive Streaming over HTTP) method using a distribution server 13 is a HTTP (HyperText Transfer Protocol) server, the reproducing apparatus 15 MPEG- DASH is a client.
[0045]
　(Configuration Example of generation apparatus)
　FIG 3 is a block diagram showing a configuration example of a generator 12 of FIG.
[0046]
　Generator 12 of FIG. 3, the stitching processing unit 21, the mapping processing unit 22, the low-resolution conversion unit 23, an encoder 24, setting unit 25, the perspective projection unit 26-1 to 26-5, the encoder 27-1 through 27- 5, the table generation unit 28, and a the transmitting unit 29.
[0047]
　Stitching processing unit 21, for each frame, and the same color and brightness of the six directions of the captured image supplied from the camera 11A of Figure 2, to connect the overlap is removed. Stitching 21 supplies the captured image in units of frames obtained as a result to the mapping process section 22.
[0048]
　Mapping processor 22, by a method using equirectangular, generates an omnidirectional image from the captured image supplied from the stitching processing unit 21. Specifically, the mapping processing unit 22 maps a sphere centered on the predetermined viewpoint captured image as a texture, to produce an image of equirectangular of the sphere as a celestial sphere image. Thus, the shape of the celestial sphere image generated by the mapping process section 22 is a rectangular suitable for coding.
[0049]
　Mapping processing unit 22 supplies the omnidirectional image into low-resolution conversion unit 23 and the perspective projection unit 26-1 to 26-5. Incidentally, stitching processing unit 21 and the mapping processor 22 may be integrated.
[0050]
　Resolution reduction unit 23, the omnidirectional image supplied from the mapping unit 22 to the low resolution, to generate a low-resolution image. Resolution reduction unit 23 supplies the low-resolution images generated in the encoder 24.
[0051]
　Encoder 24 (the low-resolution coding section) is encoded with the encoding method such as a low-resolution image supplied from the low-resolution unit 23 MPEG2 (Moving Picture Experts Group phase 2) system or AVC (Advanced Video Coding) scheme , to produce a single low-resolution stream. The encoder 24 supplies one low resolution stream to the transmission unit 29.
[0052]
　Setting unit 25 sets the two-dimensional plane information corresponding to the five viewing direction. Setting unit 25 supplies the respective two-dimensional plane information to perspective projection unit 26-1 to 26-5. The setting unit 25 supplies the five 2-dimensional plane information table generating unit 28.
[0053]
　Perspective projection unit 26-1 to 26-5, respectively, to map the omnidirectional image supplied from the mapping unit 22 to a sphere. Perspective projection unit 26-1 to 26-5, respectively, the center of the sphere as the focus, perspective the omnidirectional image mapped to sphere, the two-dimensional plane represented by two-dimensional plane information supplied from the setting unit 25 by projecting, to generate an image. Thus, the generated image is an omnidirectional image mapped to a sphere with an image viewed from the center of the sphere in a predetermined viewing direction. Perspective projection unit 26-1 to 26-5, respectively, supplied to the encoder 27-1 to 27-5 of the generated image as a high-resolution image.
[0054]
　Encoder 27-1 to the encoder 27-5 (high resolution coding section), respectively, the high-resolution image supplied from the perspective projection unit 26-1 to 26-5, an encoding method such as MPEG2 method and AVC method encoding, to generate a high-resolution stream one.
[0055]
　In this case, for example, between five high-resolution stream generated by the encoder 27-1 to 27-5, the synchronization point, such as the first picture and the IDR picture of the GOP (Group of Picture) is the same. Encoder 27-1 to 27-5, respectively, and supplies one generated in a high-resolution stream to the transmission unit 29.
[0056]
　In the following description, when it is not necessary to distinguish the perspective projection unit 26-1 to 26-5, that perspective projection unit 26 them together. Similarly, as the encoder 27 are collectively encoders 27-1 to 27-5.
[0057]
　Table generation unit 28 generates a two-dimensional plane table includes five two-dimensional plane information supplied from the setting unit 25, and supplies this to the transmission unit 29.
[0058]
　Transmission unit 29, one low-resolution stream supplied from the encoder 24, the two-dimensional plane table supplied from the high-resolution stream, and the table generation unit 28 of the total of five supplied from the respective encoders 27, FIG. in 2 of the distribution server 13 to upload (send).
[0059]
　(First example of a two-dimensional plane)
　FIG. 4 is a diagram showing an example of five two-dimensional plane is set by the setting unit 25 of FIG.
[0060]
　A of the Figure 4 B in FIG. 4, respectively, perspective view of the sphere as a two-dimensional plane is a 3D model that is set therein, a top view of a horizontal cutting plane.
[0061]
　In the example of FIG. 4, omnidirectional image is an omnidirectional image generated from the photographed image obtained by photographing the concert hall. Further, when the omnidirectional image is mapped to sphere 40 passes through the center O of the sphere 40, the horizontal angle between the reference axis on a horizontal plane passing through the center O, -90 °, -45 °, 0 ° , 45 degrees, in a direction that is 90 degrees, there is a full spherical image of a stage arranged in the concert hall. That is, the horizontal angle between the reference axis of the viewing direction is assumed to be important for the viewer to view the position of the center O, -90 °, -45 °, 0 °, 45 °, 90 ° .
[0062]
　Accordingly, as shown in B of A and 4 of Figure 4, the setting unit 25, passes through the center O of the sphere 40, the horizontal angle between the reference axis, -90 °, -45 °, 0 °, 45 degrees, a line is 90 degrees, and a normal line passing through the center, so that the adjacent ones intersect, set the internal two-dimensional plane 41-45 spheres 40. Therefore, a portion of the celestial sphere image perspectively projected in the adjacent of the two-dimensional plane 41 through 45 overlap.
[0063]
　Further, in the example of FIG. 4, the absolute value of the horizontal angle between the normal line and the reference axis passing through the center of the two-dimensional plane 41-45 is 90 degrees or less. Accordingly, the reproducing apparatus 15, even with the high-resolution images corresponding to all the two-dimensional plane, and generates a display image corresponding to all of the line of sight direction of the horizontal direction 360 degrees around the and vertical directions around 180 degrees that can not.
[0064]
　In the example of FIG. 4, the vertical angle between the normal line and the reference axis passing through the center of the two-dimensional plane 41-45 are all 0 degrees, there is no inclination of the two-dimensional plane 41-45.
[0065]
　(High Description resolution image)
　FIG. 5 is a diagram for explaining the high-resolution image generated by the perspective projection unit 26 of FIG.
[0066]
　In the example of FIG. 5, setting unit 25 sets the two-dimensional plane information of the two-dimensional plane 41-45 in FIG. Also, B in A and 5 of FIG. 5 is a top view of the ball 40 to omnidirectional image is mapped, representing a small black circles pixels in FIG.
[0067]
　As shown in A of FIG. 5, by celestial sphere image for each divided region having overlapping the adjacent ones with each other is divided, if the high-resolution image is generated, overlapping adjacent high resolution image region mapping positions on the sphere 40 of each pixel of the (sampling point) is the same.
[0068]
　Specifically, the mapping position of the ball 40 of the pixel 61A and the pixel 62A of overlapping region between the high resolution image 61 of the adjacent divided area high resolution image 62 is the same. Similarly, mapping positions on the sphere 40 of the pixel 62B and the pixel 63A of overlapping region between the high resolution image 62 of the adjacent divided area high resolution image 63 is the same. That is, the position of the celestial sphere image pixels 61A and pixels 62A are the same, the position of the celestial sphere image pixel 62B and the pixel 63A is the same.
[0069]
　Thus, by adhering by superimposing each pixel to each other in the overlapping region of the high resolution image 61 to 63, it is possible to generate a part of the celestial sphere image. In A of FIG. 5, for clarity, illustrates the high resolution image 61 and the high resolution image 63 away from the ball 40.
[0070]
　On the other hand, as shown in B of FIG. 5, the mapping positions on the sphere 40 of each pixel of the high resolution image 82 to 84 are perspective projection into a two-dimensional plane 42 through 44 by the perspective projection unit 26 is different.
[0071]
　Specifically, different from the mapping positions on the sphere 40 of the pixel 82A of the high-resolution image 82, the mapping position of the pixel 83A of the high resolution image 83 to be mapped to the nearest of the mapping position. Similarly, different mapping positions on the sphere 40 of the pixel 83B of the high resolution image 83, the mapping position of the pixel 84B of the high resolution image 84 to be mapped to the nearest of the mapping position. Therefore, even if stuck by superimposing high-resolution image 82 to 84, it is impossible to generate a portion of the celestial sphere image.
[0072]
　In B of FIG. 5, only the high resolution image 82 to 84 perspective projection into a two-dimensional plane 42 through 44 have been shown, for perspective projection high resolution image into a two-dimensional plane 41 and the two-dimensional plane 45 also it is similar to the high resolution image 82 to 84.
[0073]
　(Configuration example of a two-dimensional plane table)
　FIG. 6 is a diagram showing a configuration example of a two-dimensional plane table generated by the table generation unit 28 of FIG.
[0074]
　In the example of FIG. 6, a two-dimensional plane information includes azimuth and elevation as information indicating the position of the two-dimensional plane 41-45 in FIG. 4, includes a rotation angle as the information indicating the inclination, lateral as information indicating the size including the angle of view and the vertical angle of view.
[0075]
　Incidentally, azimuth and elevation, respectively, the horizontal angle between the line connecting the centers of the center O and 2-dimensional plane 41-45 of the sphere 40, the reference axis on a horizontal plane passing through the center O, the vertical angle it is. Rotation angle is the angle of the rotation direction of the two-dimensional plane 41 or 45 of the line connecting the center and the center O of the two-dimensional plane 41-45 when the axis. Horizontal angle of view, the two ends of the transverse direction of the two-dimensional plane 41-45 is the angle of each and the center O and a line connecting the vertical angle of view, in the longitudinal direction of the two-dimensional plane 41-45 2 One end is the angle of each and the center O and a line connecting.
[0076]
　In this case, as shown in FIG. 6, the two-dimensional plane table, a unique ID is registered in each of the two-dimensional plane 41-45. In the example of FIG. 6, from 1 to two dimensional plane 41 to 45 are ID is sequentially applied to the two-dimensional plane table, 1 to 5 is registered as ID.
[0077]
　Further, the two-dimensional plane table in association with the ID, a 2-dimensional plane information of the two-dimensional plane corresponding to the ID, the number of horizontal pixels is the number of horizontal pixels of the high resolution image of the two-dimensional plane and vertical pixel number and the number of pixels in the vertical direction is registered.
[0078]
　Specifically, a two-dimensional plane 41 through 45, respectively, through the center O of the sphere 40, the horizontal angle of -90 degrees from the reference axis, -45 °, 0 °, 45 °, is 90 ° , the vertical angle line are all 0 degrees, and a normal line passing through the center, is set so as not tilt. Therefore, in association with each ID of "1" to "5", the azimuth angle "-90 degree" azimuthal angle "-45 degree" azimuthal angle "0 degree" azimuth "45 degrees", azimuth " 90 degrees "is registered. Further, in association with the ID "1" to "5", elevation "0 degree" and the rotation angle of "0 degree" it is registered.
[0079]
　In the example of FIG. 6, the two-dimensional plane 41 to the horizontal angle of 45 and Tatega angle is 90 degrees, the horizontal pixel number and vertical pixel number is 1024. Therefore, in association with the ID "1" to "5", the horizontal angle "90 degrees", Tatega angle "90 degrees", the horizontal pixel number "1024", and the vertical pixel number "1024" is registered.
[0080]
　(Process description generation apparatus)
　FIG. 7 is a flowchart illustrating a process of generating producer 12 of FIG.
[0081]
　In step S11 in FIG. 7, the stitching processing unit 21, for each frame, and the same color and brightness of the six directions of the captured image supplied from the camera 11A of Figure 2, to connect the overlap is removed. Stitching 21 supplies the captured image in units of frames obtained as a result to the mapping process section 22.
[0082]
　In step S12, the mapping processing unit 22, by a method using equirectangular, it generates an omnidirectional image from the captured image supplied from the stitching processing unit 21. Mapping processing unit 22 supplies the omnidirectional image into low-resolution conversion unit 23 and the perspective projection unit 26-1 to 26-5.
[0083]
　In step S13, the low-resolution unit 23, the omnidirectional image supplied from the mapping unit 22 to the low resolution, to generate a low-resolution image. Resolution reduction unit 23 supplies the low-resolution images generated in the encoder 24.
[0084]
　In step S14, the encoder 24, the low-resolution image supplied from the low-resolution unit 23 encodes and generates a single low-resolution stream. The encoder 24 supplies one low resolution stream to the transmission unit 29.
[0085]
　In step S15, the setting unit 25 sets the two-dimensional plane information corresponding to the five viewing direction. Setting unit 25, the respective two-dimensional plane information supplied to the perspective projection unit 26, and supplies the five 2-dimensional plane information table generating unit 28.
[0086]
　In step S16, the perspective projection unit 26, the omnidirectional image supplied from the mapping unit 22 maps a sphere, the center of the sphere as the focus, the omnidirectional image mapped to a sphere, setting unit 25 by perspective projection into a two-dimensional plane represented by two-dimensional plane information supplied from, for generating an image. Each perspective projection unit 26 supplies each encoder 27 the generated image as a high resolution image.
[0087]
　In step S17, each encoder 27 encodes the high-resolution image supplied from the perspective projection unit 26, generates a high-resolution stream one, and supplies this to the transmission unit 29.
[0088]
　In step S18, the table generation unit 28 generates a two-dimensional plane table includes five two-dimensional plane information supplied from the setting unit 25, and supplies this to the transmission unit 29.
[0089]
　In step S19, the transmission section 29, one of the low-resolution stream, the two-dimensional plane table supplied from the respective encoders 27 high-resolution streams total of five supplied from and the table generation unit 28, which is supplied from the encoder 24 and upload to the distribution server 13.
[0090]
　(Configuration Example of the distribution server and the playback device)
　FIG. 8 is a block diagram showing a configuration example of the distribution server 13 and the playback device 15 in FIG. 2.
[0091]
　As shown in FIG. 8, the distribution server 13 is constituted by the receiving unit 101, a storage 102, transmitter 103 and transmitter 104,.
[0092]
　Receiver 101, one low-resolution streams uploaded from generator 12 of FIG. 2, five high-resolution stream, and receives the two-dimensional plane table, and supplies the storage 102.
[0093]
　Storage 102 stores one low-resolution stream supplied from the receiving unit 101, five high-resolution stream, and a two-dimensional plane table.
[0094]
　Transmitter 103 in response to a request from the reproducing apparatus 15 reads out the one of the low-resolution stream and two-dimensional plane table from the storage 102, and transmits to the playback apparatus 15 via the network 14.
[0095]
　Transmitting section 104 in response to a request from the playback device 15 reads the high-resolution stream one from the storage 102, and transmits to the playback apparatus 15 via the network 14. Note that changing of the high resolution stream to be transmitted is performed in the synchronization point. Thus, changes in the high-resolution stream to be transmitted is performed in several tens of frames from several frames.
[0096]
　As described above, the synchronization point between the five high-resolution stream is identical. Accordingly, the transmission unit 104, by switching the high-resolution stream to be transmitted in a synchronization point, in the reproduction apparatus 15, it is possible to easily switch the high-resolution image to be reproduced.
[0097]
　Reproducing apparatus 15 is constituted by the camera 15A, the receiving unit 121, a decoder 122, receiving unit 123, a decoder 124, the mapping processing unit 125, rendering unit 126, receiving portion 127 and the line-of-sight detecting unit 128.
[0098]
　Receiver 121 of the reproducing apparatus 15, a low-resolution stream and the two-dimensional plane information one to the distribution server 13, and requests via the network 14. Receiver 121 (receiving unit) receives the low-resolution stream and the two-dimensional plane information of a single transmitted from the transmitting unit 103 in response to the request. Receiver 121, one low resolution stream is supplied to the decoder 122, and supplies the two-dimensional plane information to line-of-sight detecting unit 128.
[0099]
　Decoder 122 (low resolution decoder) decodes the low-resolution stream supplied from the receiving unit 121 to generate a low resolution image. The decoder 122 supplies the low-resolution image to the mapping processing unit 125.
[0100]
　Receiving unit 123, the sight line detection unit 128 acquires the selected surface information indicating the ID of the selected surface is one of five two-dimensional plane. Receiving unit 123, based on the selection surface information, out of the five high-resolution stream, a single high-resolution stream selection plane specified by the selected face information, the request via the network 14. Receiving unit 123 receives the high-resolution stream of one transmitted from the transmitting unit 104 in response to the request, and supplies to the decoder 124.
[0101]
　Decoder 124 (high resolution decoder) decodes the high-resolution stream of one supplied from the reception unit 123, to generate a high resolution image. The decoder 124 supplies the high-resolution image to the mapping processing unit 125.
[0102]
　Mapping processing unit 125, based on the two-dimensional plane information of the selected plane supplied from the sight line detecting unit 128 sets the selected face as the 3D model in the interior of the sphere that is set in advance as a 3D model. Mapping processing unit 125, a low-resolution image supplied from the decoder 122, mapped as a texture to a sphere as a 3D model. Further, the mapping processing unit 125, a two-dimensional plane as a 3D model, maps the high resolution image supplied from the decoder 124 as texture. Mapping processing unit 125 supplies the drawing unit 126 a 3D model image texture to a sphere and the selected face is mapped.
[0103]
　Rendering unit 126, the 3D model image supplied from the mapping unit 125, as the focus of the viewing position supplied from the sight line detecting unit 128, by perspective projection in the visual field range of the viewer, the viewing range of the viewer image is generated as a display image. That is, the drawing unit 126 generates a mapped image into a sphere 40 or the two-dimensional plane visible through the viewing range of viewing position as the display image. Rendering unit 126 supplies the display image on the head mounted display 16.
[0104]
　Receiving unit 127 receives the detection result of the gyro sensor 16B in FIG. 2 from the head-mounted display 16, and supplies the line-of-sight detecting unit 128.
[0105]
　Visual axis detection unit 128, based on the detection result of the gyro sensor 16B supplied from the receiving unit 127, determines the viewing direction of the viewer in the coordinate system of the 3D model. Further, the sight line detection unit 128 acquires the captured image of the markers 16A from the camera 15A, based on the captured image to detect a viewing position in the coordinate system of the 3D model.
[0106]
　Line-of-sight detection section 128 (selecting unit) viewing position and viewing direction in the coordinate system of the 3D model, and, based on the two-dimensional plane information supplied from the receiving unit 121, of the five 2-dimensional plane, the viewing one two-dimensional plane corresponding to the closest normal to the line of sight is determined on the selected surface.
[0107]
　Specifically, the line-of-sight detection unit 128 is closest to the rotation angle in the horizontal and vertical angles as well as line of sight forms a sight line and the reference axis extending in the viewing direction from the viewing position, corresponding to the azimuth and elevation and rotation angle the ID of the two-dimensional plane is obtained as the ID of the selected surface.
[0108]
　Thus, the sight line detection unit 128, as the proportion of high-resolution image is perspectively projected in the visual field range of the viewer is the highest, it is possible to choose a two-dimensional plane corresponding to the high resolution image as the selected surface . Visual axis detection unit 128 supplies the selected surface information to the receiving section 123 supplies the two-dimensional plane information of the selected surface to the mapping processing unit 125.
[0109]
　Further, the sight line detection unit 128, based on the viewing position and the viewing direction in the coordinate system of the 3D model to determine the field of view of the viewer in the coordinate system of the 3D model. Visual axis detection unit 128 supplies the drawing unit 126 a viewing position and viewing range of the viewer.
[0110]
　(Description of the mapping)
　9 is a diagram illustrating the mapping by the mapping processing unit 125 in FIG. 8.
[0111]
　In the example of FIG. 9, the viewing position is the center O of the sphere 40. In this case, closest to the rotation angle of the eggplant horizontal and vertical angles as well as line of sight 141 of the sight 141 and the reference axis extending in the viewing direction from the center of O, corresponding to the azimuth and elevation and angle of rotation, the arrows from the center O 142 the line extending in the direction, a two-dimensional plane 143 which the normal line passing through the center is selected as the selected surface.
[0112]
　Accordingly, the mapping processing unit 125 sets the two-dimensional plane 143 as a 3D model inside the sphere 40. Then, the mapping processing unit 125, a low-resolution image 151 is mapped as a texture to a sphere 40 as the 3D model, maps the high resolution image 152 as a texture on a two-dimensional plane 143 as a 3D model.
[0113]
　As described above, the two-dimensional plane 143 is disposed inside the sphere 40. Accordingly, the drawing unit 126, from the direction in which both the low-resolution image 151 high-resolution image 152 is present, when performing perspective projection from the two-dimensional plane 143 on the inside of the field of view, high-resolution image in comparison with the low-resolution image 151 152 can be used preferentially.
[0114]
　(Display Example of Image)
　FIG. 10 is a diagram showing an example of a display image.
[0115]
　In the example of FIG. 10, the viewing position is the center O of the sphere 40. Further, the rectangular 10 shows a block of 16 pixels × 16 pixels of the high resolution image or low-resolution images. This block, for example, when the encoding scheme of the high-resolution image and the low resolution image is AVC scheme, the coding unit.
[0116]
　As shown in A of FIG. 10, if the viewer views the predetermined two-dimensional plane as the front, the center of the display image 170 high resolution image 171 is arranged. In the example of A of FIG. 10, because the size of the display image 170 is larger than the size of the high resolution image 171, the both ends of the display image 170, the low-resolution image 172 is arranged.
[0117]
　As shown in B of FIG. 10, when the viewing direction of the viewer is obliquely upward, in the example of B of a two-dimensional plane (FIG. 10 corresponding to the closest normal to the viewing, elevation angle is 45 degrees 2 high resolution image 181 of the dimension plane) is located in the center of the display image 180. Further, as shown in B of FIG. 10, in a region other than the high resolution image 181 of the display image 180, the low-resolution image 182 is arranged. As shown in B of FIG. 10, in the display image 180, the density of the blocks of the low-resolution image 182 that is mapped to the vicinity of the pole portion of the sphere 40, the low-resolution image is mapped to the vicinity of the center of the sphere 40 182 greater than the density of the block.
[0118]
　As shown in C of FIG. 10, when the viewing direction of the viewer is an upper direction, in the C examples of two-dimensional plane (FIG. 10 corresponding to the closest normal to the viewing, 2-dimensional elevation is 90 degrees high resolution image 181 of the plane) is located in the center of the display image 190. Further, as shown in C of FIG. 10, in a region other than the high resolution image 191 of the display image 190, the low-resolution image 192 is arranged.
[0119]
　The C in the example of FIG. 10, since the elevation angle of the two-dimensional plane corresponding to the high resolution image is 90 degrees, the region corresponding to the pole direction of the ball 40 in the display image 190 is a high resolution image 191. That is, in the display image 190, relatively large distortion occurs in the low-resolution image, the region corresponding to the pole direction of the ball 40 is a high resolution image 191 that distortion does not occur. Accordingly, it is a region corresponding to the pole direction of the ball 40 in comparison with the case of the low resolution image 192, to improve the image quality of the display image 190.
[0120]
　Note that, as shown in C of A through 10 in FIG. 10, the low-resolution image 172 (182 and 192) is smaller in resolution than the high resolution image 171 (181 and 191). Therefore, the size of the area of ​​the display image 170 (180, 190) within the block is placed in the low resolution image 172 (182 and 192), the display image 170 of blocks are arranged in the high-resolution image 171 (181 and 191) It is larger than the size of the area in (180, 190).
[0121]
　As described above, in the example of C of A through 10 in FIG. 10, the display image 170 (180, 190) of the total area of ​​the high resolution image 171 to the center of the resolution greatly influences the visual (181 and 191 ) are arranged. Therefore, as compared with the case of low-resolution image 172 (182, 192) is arranged in the center of the display image 170 (180, 190), it is possible to improve the image quality of the display image 170 (180, 190).
[0122]
　(Description of the processing of the reproduction apparatus)
　FIG. 11 is a flowchart illustrating the playback processing of the playback apparatus 15 of FIG. The regeneration process is started, for example, in response to a request of the viewer.
[0123]
　In step S31 in FIG. 11, the receiving unit 121 of the reproducing apparatus 15, the distribution server 13 requests the two-dimensional plane information, it receives the two-dimensional plane information transmitted from the transmitting unit 103 in response to the request. Receiving unit 121 supplies the two-dimensional plane information to line-of-sight detecting unit 128.
[0124]
　In step S32, receiving unit 127 receives the detection result of the gyro sensor 16B in FIG. 2 from the head-mounted display 16, and supplies the line-of-sight detecting unit 128.
[0125]
　In step S33, line-of-sight detecting unit 128, based on the detection result of the gyro sensor 16B supplied from the receiving unit 127, determines the viewing direction of the viewer in the coordinate system of the 3D model.
[0126]
　In step S34, line-of-sight detecting unit 128 obtains the captured image of the markers 16A from the camera 15A, it detects the viewing position in the coordinate system of the 3D model based on the captured image.
[0127]
　In step S35, the sight line detection unit 128, the viewing position and the viewing direction in the coordinate system of the 3D model, and, based on the two-dimensional plane information supplied from the receiving unit 121, of the five 2-dimensional surface viewers determining the two-dimensional plane of the line of sight one closest to the selection plane. Visual axis detection unit 128 supplies the reception unit 123 to select surface information of the selected surface, it supplies the two-dimensional plane information of the selected surface to the mapping processing unit 125.
[0128]
　In step S36, the sight line detection unit 128, based on the viewing position and the viewing direction in the coordinate system of the 3D model to determine the field of view of the viewer in the coordinate system of the 3D model. Visual axis detection unit 128 supplies the drawing unit 126 a viewing position and viewing range of the viewer.
[0129]
　In step S37, the receiving unit 123 requests the one of the high resolution stream selection plane identified by the selection surface information supplied from the sight line detecting unit 128 to the distribution server 13, the transmission unit 104 in response to the request receive a high resolution stream one transmitted. Receiving unit 123 supplies the high-resolution stream of one received in the decoder 124.
[0130]
　In step S38, the decoder 124 decodes the high-resolution stream of one supplied from the reception unit 123, to generate a high resolution image. The decoder 124 supplies the high-resolution image to the mapping processing unit 125.
[0131]
　In step S39, the mapping processing unit 125, based on the two-dimensional plane information of the selected plane supplied from the sight line detecting unit 128 sets the selected face as the 3D model in the interior of the sphere that is set in advance as a 3D model.
[0132]
　In step S40, the mapping processing unit 125, a high-resolution image supplied from the decoder 124, mapped as a texture on the selected surface set as a 3D model.
[0133]
　In step S41, the receiving unit 121 requests the low-resolution streams one to the distribution server 13 receives a low-resolution stream of one transmitted from the transmitting unit 103 in response to the request. Receiving unit 121 supplies one of the low-resolution streams to the decoder 122.
[0134]
　In step S42, the decoder 122 decodes the low-resolution stream supplied from the receiving unit 121 to generate a low resolution image. The decoder 122 supplies the low-resolution image to the mapping processing unit 125.
[0135]
　In step S43, the mapping processing unit 125, a low-resolution image supplied from the decoder 122, mapped as a texture to a sphere as a 3D model. Mapping processing unit 125 supplies the drawing unit 126 a 3D model image texture sphere and two-dimensional plane is mapped.
[0136]
　In step S44, the drawing unit 126, the 3D model image supplied from the mapping unit 125, as the focus of the viewing position supplied from the sight line detecting unit 128, by perspective projection in the visual field range of the viewer, the viewer generating an image of the visual field range as a display image.
[0137]
　In step S45, the drawing unit 126 displays by sending the display image on the head mounted display 16. In step S46, the reproducing apparatus 15, whether to finish playback, determining whether the end of reproduction is requested, for example, by a viewer.
[0138]
　If it is determined not to end the playback in step S46, the process returns to step S31, until it is determined to end the reproduction, processing in steps S31 to S46 are repeated. On the other hand, if it is determined to terminate the playback in step S46, the process ends.
[0139]
　Incidentally, in the above description, the number of two-dimensional plane has been as a five, the number of two-dimensional plane is not limited to five. The greater the number of two-dimensional plane is large, the reproduction apparatus 15 is capable of generating a display image by using the high-resolution image corresponding to the normal closer sight of the viewer. Thus, the proportion of the high-resolution image in the display image is increased, thereby improving the image quality of the display image. However, since the number of high-resolution stream is increased, throughput of the generating device 12 for generating a storage capacity and a high-resolution stream storage 102 required is increased.
[0140]
　Further, the two-dimensional plane table, two-dimensional plane information, horizontal pixel number, and other than the fixed value of the number of vertical pixels may be registered. Further, two-dimensional plane may be set in one or more frames may be set in units of scenes.
[0141]
　(Second example of a two-dimensional plane)
　FIG. 12 is a diagram showing an example of a two-dimensional plane when the number of two-dimensional plane is other than five.
[0142]
　In FIG. 12, arrows indicate the normal line passing through the center of each 2-dimensional plane.
[0143]
　Figure 12 as shown in A, setting unit 25, as a two-dimensional plane, it is also possible to set the six surfaces 211 to 216 of the cube 210 about the center O of the sphere 40. In this case, the normal line passing through the center of the six 2-dimensional plane, through the center O, a total of six lines in both directions of three axes orthogonal to each other. Further, the horizontal field angle and Tatega angle of all the two-dimensional plane is 90 degrees, the two-dimensional plane do not overlap.
[0144]
　That is, in this case, a high resolution image of the two-dimensional plane is an image obtained by dividing the omnidirectional image generated by cube mapping in terms units cube as a 3D model. Note that the cube mapping, the image is mapped to a cube as a 3D model, a developed view of a cube on which an image is mapped is a method of generating omnidirectional images to omnidirectional image.
[0145]
　Further, as shown in B of FIG. 12, setting unit 25, so that the normal line passing through the center of each 2-dimensional plane, is a line passing through the 12 middle point and the center O of each side of the cube 210, it is also possible to set the 12 two-dimensional plane. In this case, as compared with the case of A in FIG. 12, because the angle between adjacent two-dimensional plane becomes smaller, the reproducing apparatus 15 may be a selected surface 2D plane corresponding to the normal closer sight . As a result, the ratio of the high-resolution image in the display image is increased, thereby improving the image quality of the display image.
[0146]
　Furthermore, as shown in C of FIG. 12, setting unit 25, the normal line passing through the center of the two-dimensional plane, a line passing through the 12 middle point and the center O of each side of the cube 210, and cube 210 as is six surfaces 211 to a line passing through the center and the center O of 216, it is also possible to set the 18 two-dimensional plane. In this case, two-dimensional plane is a two-dimensional plane in the case of surface 211 to 216 and Figure 12 B.
[0147]
　In the example of C in A and 12 in FIG. 12, by using all the high-resolution image corresponding to the surface 211 to 216, corresponding to all of the viewing direction in the horizontal direction 360 degrees around the and vertical 180 degrees around it is possible to generate a display image.
[0148]
　(Another example of a method of generating a full spherical image)
　In the above description, omnidirectional image has been produced by a method using equirectangular generation method celestial sphere image, the method but it is not limited.
[0149]
　Figure 13 is a diagram illustrating an example of a method of generating equirectangular omnidirectional image other than the method using a.
[0150]
　The method of generating the full spherical image of FIG. 13, as shown in A of FIG. 13, the captured image is mapped to regular octahedron 230 as a 3D model. Then, the shape of the mapped image on each side 231 to 238 of the equilateral triangle of the octahedron 230 are deformed in a right triangle, as shown in B of FIG. 13, by combining the images 241 to 248 of each right-angled triangle , omnidirectional image 240 square is produced.
[0151]
　If omnidirectional image is generated by the generation method of FIG. 13, a two-dimensional plane 261 is disposed within the octahedron 230. Also, 3D model low-resolution image is mapped by the mapping processor 125 is a regular octahedron 230.
[0152]
　Method for generating a celestial sphere image, in addition to the method of FIG. 13, may be a cube mapping. Further, a method of generating a low resolution before omnidirectional image of the low resolution image, a method of generating a full spherical image used for generating a high-resolution image may be different.
[0153]
　(Other examples of low-resolution image)
　FIG. 14 is a diagram showing another example of the low-resolution image.
[0154]
　Image In the above description, the low-resolution image is a was a an omnidirectional image of one viewpoint lower resolution image, which was reduced resolution respectively omnidirectional image of the view for the viewpoint and the right-eye it may be a synthetic (packing) image.
[0155]
　Specifically, as shown in A of FIG. 14, the low-resolution image, for example, an omnidirectional image of the view for the right eye and the low resolution image 421 obtained by the low resolution of the omnidirectional image of the view for the left eye a low resolution and low-resolution image 422 is laterally (horizontally) to be packed the packed image 420 (low resolution packed image) may be.
[0156]
　Further, as shown in B of FIG. 14, the low-resolution image, for example, packed image to the low-resolution image 422 of the viewpoint for the right eye and the low-resolution image 421 of the viewpoint for the left eye is packed in a longitudinal direction (vertical direction) 440 may be a (low resolution packed image).
[0157]
　Similarly, high-resolution images of each of the two-dimensional plane, the high resolution image and a high resolution image horizontally or vertically in the packed packing images (high resolution perspective for the right eye perspectives for the left eye of the two-dimensional plane it may be a packed image). High-resolution image of the view for the left eye of the predetermined two-dimensional plane, the omnidirectional image of the view for the left eye that is mapped to a sphere as a focus the center of the sphere is obtained by perspective projection in the two-dimensional plane is an image. The high resolution image for the right eye of the predetermined two-dimensional plane, the omnidirectional image of the view for the right eye that is mapped to a sphere as a focus the center of the sphere is obtained by perspective projection in the two-dimensional plane is an image.
[0158]
　When the low-resolution image and a high-resolution image is a packed image, the mapping processor 125 of FIG. 8, the low resolution of the viewpoint of the low-resolution image and the right-viewpoint for the left eye packing image obtained as a result of decoding by the decoder 122 to separate the image. Further, the mapping processing unit 125 separates the packed image obtained as a result of decoding by the decoder 124 to a high resolution image of the high resolution image and the viewpoint of the right eye perspectives for the left eye. Then, the mapping processing unit 125 generates a 3D model image for each viewpoint for the viewpoint and the right eye left eye, the drawing unit 126, for each of the viewpoints and the viewpoint of the right eye left eye, using the 3D model image Te to generate a display image.
[0159]
　Thus, the head-mounted display 16, if a 3D capable display, a viewpoint and a display image of the view for the right eye left eye, respectively, the image for the left eye, by displaying an image for the right eye, a display image it is possible to 3D display.
[0160]
　As described above, generator 12 of the distribution system 10 generates a high resolution image by perspective projection omnidirectional image into a plurality of two-dimensional plane. Thus, the shape of the 3D model of the high-resolution image is not distorted.
[0161]
　Thus, the reproducing apparatus 15 can generate a display image using a high-resolution image having isotropic. As a result, it is possible to the quality of all the viewing direction of the display image to the same extent. Also be performed using the region of the high-resolution image in the display image, distortion of the general image processing such as the face recognition processing greatly affects the accuracy with high precision.
[0162]
　Moreover, since the movement of the movement and a high resolution image of the 3D model matches, it is possible to perform coding using motion compensation for the high-resolution image with high accuracy. Furthermore, it is possible to perform uniform bit allocation in a high-resolution image.
[0163]
　The reproduction device 15 receives requests only to the distribution server 13 high resolution stream 2D ​​plane corresponding to the viewing direction. Therefore, as compared with the case of receiving a high-resolution stream of all the two-dimensional plane, to reduce the transmission amount between the reproducing apparatus 15 and the delivery server 13.
[0164]
　Further, generator 12, the entire omnidirectional image and the lower resolution to produce a lower resolution image. Thus, generator 12, and if also the corresponding to the area other than the area of ​​the display image high resolution image, even when the viewing direction of the viewer is changed suddenly, a display image with a low resolution image it can be generated. The distribution system 10 may be compatible with the reproducing apparatus for reproducing only the omnidirectional image overall code stream.
[0165]
　Also, generator 12 receives the high-resolution stream selection surface and the low-resolution stream, to generate a display image. Thus, receiving an encoded stream of one resolution of omnidirectional image, as compared with the case of generating a display image, in the same transmission amount and the processing amount, it is possible to improve the resolution of the display image.
[0166]
　Further, generator 12 may be any position, inclination, and the two-dimensional plane can be configured in size. Thus, generator 12 may generate a high resolution image corresponding to any viewing direction. In contrast, when generating a high resolution image by dividing the omnidirectional image generated by a method using equirectangular, corresponding to the region extending laterally in the omnidirectional image, the sphere 40 it is difficult to make the image to be mapped to the pole produced as a high resolution image.
[0167]
　Also, generator 12 from the captured image, to generate an omnidirectional image to be used in generating a low-resolution image to generate a high-resolution image from the omnidirectional image. Accordingly, the reproducing apparatus 15, the high resolution image and using a low-resolution image, the display area of ​​the region and a low-resolution image of high resolution images as compared with the case of the captured image itself and the high-resolution image is not continuous, incongruity image it can be generated.
[0168]
　The two-dimensional plane is at a high density in response to the critical range in the range of sight directions can see an object which is assumed to be important setting for the viewer, it corresponds to a range of other critical range Te either configured at low density, or, may not be set. In this case, when the line of sight of the viewer is in the critical range, the reproducing apparatus 15 can generate a display image using a high-resolution two-dimensional plane corresponding to the normal closer sight. Thus, the proportion of the high-resolution image in the display image is increased, thereby improving the image quality of the display image. Further, since the number of high-resolution stream corresponding to a range of other critical range is small, it is possible to suppress the increase in the number of high-resolution stream.
[0169]
　The high-resolution stream, a low resolution stream, and the two-dimensional plane table is not stored in the storage 102, the real time delivery (Live) to the reproduction apparatus 15 via the transmission unit 103 and the transmitter 104 from the receiver 101 it may be so.
[0170]
　Furthermore, in the above description, the captured image has been assumed to be a moving image, it may be still images. Further, the number of selected faces may not be one.
[0171]
　Distribution system 10, instead of the head-mounted display 16, may have a stationary display. In this case, the playback apparatus 15 does not have a camera 15A, the viewing position and viewing direction, the viewer is input by operating a controller connected to the reproducing apparatus 15 or stationary display.
[0172]
　The distribution system 10, instead of the playback device 15 and the head-mounted display 16, may have the mobile terminal. In this case, the mobile terminal performs processing of camera 15A other playback device 15 to display the display image on the display with the mobile terminal. Viewer type by changing the attitude of the mobile terminal the viewing position and viewing direction, the mobile terminal, by detecting the attitude of the mobile terminal to the gyro sensor built, the input viewing position and viewing direction get.
[0173]
　
　configuration of the second embodiment of a distribution system according to the present disclosure, except that instead of tan-axis projection of the perspective projection (the details will be described later) is performed, the distribution of FIG. 2 configuration of the system 10 and the same. Therefore, in the following description, only the tan axis projection.
[0174]
　(The coordinate system of the description of the projection plane)
　Fig. 15 is a diagram for explaining the coordinate system of the projection surface.
[0175]
　In the second embodiment, the projection plane, generator 12, when generating the high resolution image, two-dimensional plane to tan shaft projecting the omnidirectional image which is mapped to a sphere or a reproducing apparatus 15 , when generating the display image, a visual field range of tan-axis projection of the 3D model image.
[0176]
　In the example of FIG. 15, the 3-dimensional xyz coordinate system of the 3D model, the projection plane 501 z is -1.0 is set. In this case, the center O'of the projection plane 501 is the origin, the horizontal direction of the projection plane 501 and s directions, a two-dimensional st coordinate system for the vertical direction and t direction, the coordinate system of the projection plane 501.
[0177]
　In the following, xyz coordinate system of the origin O from st coordinate system of coordinates (s, t) a vector 502 toward the, the coordinates (s, t), -1.0 is the distance from the origin O to the projection plane 501 using a vector (s, t, -1.0) called.
[0178]
　(Tan axis projection description)
　FIG. 16 is a diagram for explaining a tan shaft projection (tangent axis projection).
[0179]
　Figure 16 is a view of the projection plane 501 in the negative direction of z. In the example of FIG. 16, the st coordinate system, the minimum value of s values ​​and t values ​​of the projection plane 501 is -1.0 and the maximum value is 1.0.
[0180]
　In this case, the perspective projection, the projection vector is a vector directed from the origin O to the projection point on the projection plane 501 (s', t'-1.0) and such that projection points are set onto the projection surface 501. Incidentally, s'is the value of each predetermined interval provided in a range of s values ​​from -1.0 to 1.0, t'the predetermined intervals provided in the range of t values ​​from -1.0 to 1.0 it is of value. Therefore, the projection point in the perspective projection is uniform on the projection plane 501.
[0181]
　In contrast, when the angle of the projection plane 501 and .theta.w (in the example of FIG. 16 [pi / 2), the tan axis projection, the projection vector is the vector (tan (s'* θw / 2), tan (t' * .theta.w / 2), so that -1.0), the projected points are set onto the projection surface 501.
[0182]
　Specifically, the vector (tan (s'* θw / 2), tan (t'* θw / 2), -1.0) is, s'* a .theta.w / 2 and theta, a t'* θw / 2 φ If you, become a vector (tanθ, tanφ, -1.0). At this time, when the angle of view θw approaches to π, tanθ and tanφ diverges to infinity. Accordingly, as tan .theta or tan [phi does not diverge to infinity, vector (tanθ, tanφ, -1.0) is corrected to a vector (sinθ * cosφ, cosθ * sinφ, -cosθ * cosφ), the projection vector is the vector (sin [theta * cosφ, cosθ * sinφ, -cosθ * cosφ) and such that projection points are set onto the projection surface 501. Therefore, the tan axis projection, each other projection vector corresponding to the projection point adjacent angle are the same.
[0183]
　Similarly to the logarithmic axis (log scale), tan (s'* θw / 2), tan (t'* θw / 2) is, s'of tan axis, seen as a t'. Accordingly, in the present specification, the projection vector is the vector (tan (s'* θw / 2), tan (t'* θw / 2), -1.0) projection to be, is referred to as a tan-axis projection.
[0184]
　(Description of the projection point in the perspective projection and tan axis projection)
　17 is a diagram for explaining a projection point in the perspective projection and tan axis projection.
[0185]
　Figure 17 is a view of the projection plane 501 in the negative direction of y. In the example of FIG. 17, the projection point on the projection surface 501 is disposed nine.
[0186]
　In this case, as shown in FIG. 17, in the perspective projection, the projection points P1 to P9 are arranged at the same distance d on the projection plane 501. Thus, corresponding to the projection point P1 to P9, distance d'of P1' to P9' point on celestial sphere image mapped to a sphere 511 is not equidistant. That is, the interval d'depends on the projection vector, the wide spacing d'closer projection point to the center of the projection surface 501.
[0187]
　In contrast, in the tan-axis projection, the projection point Q1 to Q9 are arranged such that the angle of each other projection vector of the projection point adjacent is the same angle alpha. Thus, corresponding to the projection point Q1 to Q9, spacing between points Q1' to Q9' on omnidirectional image mapped on the sphere 511 is the same distance D''.
[0188]
　(Example of each pixel of the celestial sphere image on a high-resolution image generated by the perspective projection and tan axis projection)
　18, an example of each pixel of the celestial sphere image on a high-resolution image generated by the perspective projection it is a diagram showing a, FIG. 19 is a diagram showing an example of each pixel of the celestial sphere image on a high-resolution image generated by the tan-axis projection.
[0189]
　In FIG 18 and FIG 19, Aru boundaries of each pixel of the celestial sphere image on a high-resolution image shown by a white line. This is the same in FIG. 20 to be described later. In the example of FIGS. 18 and 19, two-dimensional plane of the projection plane, a six surfaces 211 to 216 A in FIG. 12. This is the same in FIG. 21 to be described later.
[0190]
　In this case, as shown in FIG. 18, each pixel of the celestial sphere image mapped to sphere, the whole sky on a high-resolution image 521 to 526 is generated by being perspectively projected on each side 211 to 216 the density of sphere image pixel 530 (rectangle enclosed by drawing a white line) is higher the closer to the center of the screen.
[0191]
　That is, in the interval between the projection point are the same on the surface 211 to 216 perspective projection, the center of density per angle point corresponding to the projection point on the celestial sphere image mapped to sphere, the surface 211 to 216 as the projection point lower close to. Thus, the density of the pixel 530 of the celestial sphere image on a high resolution image 521 to 526, the higher the closer to the center of the screen.
[0192]
　In contrast, as shown in FIG. 19, each pixel of the celestial sphere image mapped to sphere, high resolution image 541 to the upper 546 is generated by being tan axis projection on each side 211 to 216 the density of the pixel 550 of the celestial sphere image (rectangle enclosed by drawing a white line) is substantially uniform.
[0193]
　That is, corresponding to the projection point, the frequency is the same tan-axis projection point on the celestial sphere image mapped to a sphere, compared to the perspective projection, the projection points on celestial sphere image mapped to sphere density per angle point corresponding to become uniform. Thus, the density of the pixel 550 of the celestial sphere image on a high resolution image 541 to 546 become substantially equal. As a result, the image quality of the high resolution image 541 to 546 is improved in comparison with the high-resolution image 521 to 526 density is not uniform in the pixels of the celestial sphere image.
[0194]
　Since the boundary of each pixel 550 of the high-resolution image 541 and the high-resolution image 546 is not drawn concentric, high-resolution image 541 and the high-resolution image 546 is different from the image taken by the fisheye lens. Further, each with a high-resolution image 521 to 526, each of the high resolution image 541 to 546, for the same area of ​​the omnidirectional image is an image that is projected, are interconvertible is.
[0195]
　Figure 20 is a diagram showing another example of each pixel of the celestial sphere image on a high-resolution image generated by the perspective projection and tan axis projection.
[0196]
　In the upper part of FIG. 20, angle θw of the two-dimensional plane, 60 °, 90 °, 120 °, 150 °, high-resolution image 561 to 565 generated by the perspective projection in the case of 170 ° is shown is there.
[0197]
　Further, in the lower part of FIG. 20, angle θw of the two-dimensional plane, 60 °, 90 °, 120 °, 150 °, high-resolution images 571 through 575 produced by the tan-axis projection when it is 170 ° It is shown.
[0198]
　As shown in the upper part of FIG. 20, the difference due to the position of the perspective projection height on resolution image 561 to 565 of the pixel density of the celestial sphere image generated by, the screen becomes larger the larger the angle .theta.w. On the other hand, as shown in the lower part of FIG. 20, the density of the pixel of the celestial sphere image with high-resolution images 571 through 575 produced by the tan-axis projection is nearly uniform within the screen. Therefore, the larger the angle .theta.w, high-resolution image 571 to 575 generated by the tan-axis projection, the image quality is improved compared to the high resolution image 561 to 565 generated by the perspective projection.
[0199]
　The density of the pixel of the center of the screen omnidirectional image of the high resolution image 561 to 565 generated by the perspective projection is higher the larger the angle .theta.w. Thus, for example, the center of the screen resolution of the high resolution image 563 angle θw is at 120 °, the angle θw is reduced as compared with the high-resolution image 562 is 90 °.
[0200]
　On the other hand, the density of the pixels of the high resolution image 571 or 575 whole celestial sphere image generated by tan axis projection is higher the larger the angle .theta.w, nearly uniform in the screen. Therefore, the tan axis projection can be suppressed as compared with a perspective projection, that the density of pixels as the center of the screen celestial sphere image angle θw is greater increases.
[0201]
　For example, an angle θw is 120 °, the density of the pixel of the celestial sphere image in the center of the screen of the high resolution image 573 generated by the tan-axis projection, angle θw is the 90 °, generated by the perspective projection substantially equal to the density of the pixel of the celestial sphere image in the center of the screen of the high resolution image 562 that. As a result, in the tan-axis projection, as compared with a perspective projection, it is possible to improve the resolution of key areas of the center of the screen.
[0202]
　Incidentally, the illustration is omitted, the high resolution image 521 to 526 is mapped as a texture on each face of the cube, perspective projection as the boundary of two adjacent faces that is projected in the center of the screen is performed, the boundary more, the density of the pixels of the celestial sphere image is perspectively projected increases near. Similarly, when the perspective projection as the boundary of the three surfaces adjacent is projected in the center of the screen is performed, the closer to the boundary, the density of the pixel of the celestial sphere image perspectively projected increases. That is, the density of the pixel of the celestial sphere image high resolution image 521 to 526 are perspective projection from the vicinity of the apex of the cube mapped is higher than in other regions.
[0203]
　(Configuration example of a two-dimensional plane table)
　FIG. 21 is a diagram showing a configuration example of a two-dimensional plane table in the second embodiment.
[0204]
　2 dimensional planar table arrangement of Figure 21, except that the new method of projection onto a two-dimensional plane is registered as a projection system, which is the same as that of FIG. Specifically, in the second embodiment, since the projection method to two-dimensional plane is a tan axis projection, the two-dimensional plane table in FIG. 21, in association with 1 to 5 as ID, projection tan-axis projection is registered as a method.
[0205]
　Incidentally, also in the first embodiment, as in the second embodiment, to the projection system in a two-dimensional plane table may be registered, in the second embodiment, in the first embodiment Similarly, the projection system in a two-dimensional plane tables may not be registered.
[0206]
　As described above, in the second embodiment, since the high-resolution image is generated by tan shaft projecting the omnidirectional image mapped to sphere in a two-dimensional plane, to improve the quality of high-resolution images it can. Moreover, because it produces a display image by tan shaft projecting the 3D model image the field of view, it is possible to improve the image quality of the display image.
[0207]
　Incidentally, the projection is performed when generating the high resolution image and the display image, perspective projection may be other than tan-axis projection. The projection system for each two-dimensional plane may be different.
[0208]
　
　(Description of the computer according to the present disclosure)
　series of processes described above has the may be executed by hardware or can be executed by software. In the case of executing the series of processes by software, a program constituting the software is installed in a computer. Here, the computer includes a computer incorporated in dedicated hardware, by installing various programs, which can execute various functions include, for example, such as a general-purpose personal computer.
[0209]
　Figure 22 is a block diagram showing a configuration example of hardware of a computer that executes the series of processes described above.
[0210]
　In the computer 900, CPU (Central Processing Unit) 901, ROM (Read Only Memory) 902, RAM (Random Access Memory) 903 are connected to each other via a bus 904.
[0211]
　The bus 904 is further input and output interface 905 is connected. Output interface 905, an input unit 906, output unit 907, storage unit 908, a communication unit 909, and a drive 910 are connected.
[0212]
　Input unit 906 includes a keyboard, a mouse, and a microphone. The output unit 907 includes a display and a speaker. Storage unit 908, a hard disk and a nonvolatile memory. The communication unit 909 includes a network interface. Drive 910 drives a magnetic disk, an optical disk, a magneto-optical disk, or a removable medium 911 such as a semiconductor memory.
[0213]
　In configured computer 900 as described above, CPU 901 is, for example, a program stored in the storage unit 908, output interface 905 and the bus 904 and executes the loaded into RAM 903, the above-described a series of processing is performed.
[0214]
　Program computer 900 (CPU 901) is executed, for example, can be provided by being recorded on the removable medium 911 as a package medium or the like. Further, the program may be provided via a local area network, the Internet, or digital satellite broadcasting, a wired or wireless transmission medium.
[0215]
　In the computer 900, programs, by loading the removable medium 911 into the drive 910, can be installed via the input output interface 905, the storage unit 908. The program via a wired or wireless transmission medium and received by the communication unit 909, can be installed in the storage unit 908. Alternatively, the program may be in the ROM902 and the storage unit 908 installed in advance.
[0216]
　The program may be a program in which processes are performed in time series in the order described herein, the necessary timing such as when parallel or call was made by the computer 900 to execute in the process it may be a program to be carried out.
[0217]
　
　technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure, mobile vehicles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility, aircraft, drones, ships, robots, construction equipment, of any type, such as agricultural machines (tractors) it may be implemented as a device mounted to the body.
[0218]
　Figure 23 is a block diagram showing a schematic configuration example of a vehicle control system 7000 which is an example of a mobile control system technology according to the present disclosure may be applied. The vehicle control system 7000 includes a plurality of electronic control units connected through a communication network 7010. In the example shown in FIG. 23, the vehicle control system 7000 includes a drive system control unit 7100, the body system control unit 7200, a battery control unit 7300, the vehicle exterior information detection unit 7400, vehicle information detection unit 7500, and the integrated control unit 7600 . Communication network 7010 for connecting these plurality of control units, for example, CAN (Controller Area Network), LIN (Local Interconnect Network), conforming to a LAN (Local Area Network) or FlexRay (registered trademark) any standard such as it may be a vehicle-mounted communication network.
[0219]
　Each control unit includes a microcomputer for performing arithmetic processing in accordance with various programs, a storage unit for storing the parameters and the like used in the program or various operations are performed by the microcomputer, a drive circuit for driving the device for various control target equipped with a. Each control unit is provided with a network I / F for communicating with other control units via a communication network 7010, with the vehicle outside the device or sensor or the like, a wired or wireless communication a communication I / F for communication. In Figure 23, a functional configuration of the integrated control unit 7600, a microcomputer 7610, a general-purpose communication I / F7620, dedicated communication I / F7630, the positioning unit 7640, a beacon receiving unit 7650, interior equipment I / F7660, audio image output unit 7670, vehicle network I / F7680 and a storage unit 7690 is illustrated. Similarly other control unit comprises a microcomputer, a communication I / F and a storage unit or the like.
[0220]
　Driving system control unit 7100 controls the operation of the device associated with the vehicle drive system in accordance with various programs. For example, the driving system control unit 7100, the driving force generating device for generating a driving force of a vehicle such as an internal combustion engine or the driving motor, the driving force transmission mechanism for transmitting a driving force to the wheels, the steering angle of the vehicle adjusting steering mechanism, and functions as a control device of the braking device or the like to generate a braking force of the vehicle. Driving system control unit 7100, ABS (Antilock Brake System), or ESC may function as (Electronic Stability Control) control device, such as.
[0221]
　The driving system control unit 7100, the vehicle state detecting unit 7110 is connected. The vehicle state detecting unit 7110, for example, a gyro sensor for detecting an angular velocity of the body of the shaft rotary motion, the acceleration sensor detects the acceleration of the vehicle or the operation amount of the accelerator pedal, the brake pedal operation amount, the steering wheel steering angle, includes at least one of the sensors for detecting the rotational speed of the engine rotational speed or wheel. Driving system control unit 7100, performs arithmetic processing by using the signal input from the vehicle state detecting unit 7110, and controls the internal combustion engine, drive motor, the electric power steering system or the braking device or the like.
[0222]
　Body system control unit 7200 controls the operation of the camera settings device to the vehicle body in accordance with various programs. For example, the body system control unit 7200, a keyless entry system, a smart key system, the power window device, or headlamp, back lamps, it functions as a control device of the brake lamp, turn signals or various lamps fog lamp or the like. In this case, the body system control unit 7200, a signal of a radio wave or various switches is transmitted from wireless controller to replace the key can be entered. Body system control unit 7200 receives an input of these radio or signal, the door lock device for a vehicle, the power window device, controls the lamp.
[0223]
　Battery control unit 7300 controls the secondary battery 7310 is a power source of the drive motor in accordance with various programs. For example, the battery control unit 7300, a battery assembly including a secondary battery 7310, the battery temperature, information such as the remaining capacity of the battery output voltage or battery is input. Battery control unit 7300 performs arithmetic processing using these signals, controls the cooling device or the like provided in the temperature control or the battery device for a secondary battery 7310.
[0224]
　Outside information detection unit 7400 detects the external information of the vehicle having the vehicle control system 7000. For example, the outside information detection unit 7400, at least one of the imaging unit 7410 and the outside information detection unit 7420 are connected. The imaging unit 7410, ToF (Time Of Flight) camera, a stereo camera, a monocular camera, includes at least one of the infrared camera and other camera. The outside information detection unit 7420, for example, environmental sensors for detecting the current weather or weather, or other vehicles around the vehicle having the vehicle control system 7000, for detecting an obstacle or a pedestrian or the like of it includes at least one of the peripheral information detection sensor.
[0225]
　Environment sensors, for example, a raindrop sensor for detecting rain, fog sensor for detecting fog, may be at least one of snow sensor for detecting a sunshine sensor, and snow to detect the sunshine degree. Ambient information detection sensor, an ultrasonic sensor, a radar device and LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) may be at least one of the devices. These imaging unit 7410 and the outside information detection unit 7420 may be provided as an independent sensor or device, a plurality of sensors or devices may be provided as an integrated unit.
[0226]
　Here, FIG. 24 shows an example of the installation position of the imaging unit 7410 and the outside information detection unit 7420. Imaging unit 7910,7912,7914,7916,7918, for example, the front nose of the vehicle 7900, side mirrors, rear bumper, is provided on at least one position of the upper portion of the front glass of the back door and the vehicle interior. Imaging unit 7918 provided in the upper portion of the front glass of the image pickup unit 7910 and the passenger compartment provided in the front nose mainly acquires a forward image of the vehicle 7900. Imaging unit 7912,7914 provided in the side mirror is mainly to acquire an image of the side of the vehicle 7900. Imaging unit provided in the rear bumper or the back door 7916 mainly obtains the rear image of the vehicle 7900. Imaging unit 7918 provided in the upper part of the windshield in the vehicle interior mainly preceding vehicle or a pedestrian, an obstacle, a traffic used to detect such traffic signs or traffic lane.
[0227]
　Incidentally, in FIG. 24 is an example of the imaging ranges of the imaging unit 7910,7912,7914,7916 are shown. Imaging range a represents an imaging range of the imaging unit 7910 provided in the front nose, imaging range b, c are each an imaging range of the imaging unit 7912,7914 provided on the side mirror, an imaging range d is It shows the imaging range of the imaging unit 7916 provided in the rear bumper or a back door. For example, by being superimposed image data captured by the imaging unit 7910,7912,7914,7916, bird's-eye view image is obtained viewed vehicle 7900 from above.
[0228]
　Front of the vehicle 7900, the rear, side, outside information detector 7920,7922,7924,7926,7928,7930 provided on the top of the windshield of the corner and the vehicle compartment may be, for example, an ultrasonic sensor or a radar device. Front nose of the vehicle 7900, the rear bumper, outside information detector 7920,7926,7930 provided on the top of the windshield of the back door and the passenger compartment may be, for example, LIDAR device. These outside information detection section 7920 to 7930 is primarily the preceding vehicle, is used to detect a pedestrian or the like, or an obstacle.
[0229]
　Referring back to FIG. 23. Outside information detection unit 7400, causes image the outside of the image to the imaging unit 7410, receives the image data captured. Also, outside information detection unit 7400 receives the detection information from the vehicle exterior information detection unit 7420 are connected. Outside information detection unit 7420 is an ultrasonic sensor, if a radar or LIDAR system, outside information detection unit 7400, as well to transmit ultrasonic waves or electromagnetic waves or the like, receives information of the received reflected wave. Outside information detection unit 7400, based on the received information, human, cars, obstacle, may be performed object detection process or distance detecting process such as characters on the label or the road surface. Outside information detection unit 7400, based on the received information, rainfall, fog or road conditions, etc. may be performed to recognize environmental recognition process. Outside information detection unit 7400, based on the received information, may calculate a distance to the outside of the object.
[0230]
　Also, outside information detection unit 7400, based on the received image data, human, cars, obstacle, may be performed to recognize letters or the like on the label or the road surface image recognition process or the distance detection process. Outside information detection unit 7400, the process performs such distortion correction or alignment with respect to the received image data, synthesizes the image data captured by different image capturing unit 7410, and generates an overhead image or panoramic image it may be. Outside information detection unit 7400, by using the image data captured by different image capturing unit 7410 may perform the viewpoint conversion process.
[0231]
　Vehicle information detection unit 7500 detects the vehicle information. The vehicle information detection unit 7500, for example, the operation detection unit 7510 for detecting the state of the driver are connected. Operation detection unit 7510, a camera for imaging a driver may include a microphone for collecting a sound of a biological sensor or the passenger compartment to detect the biological information of the driver. Biological sensor may, for example, provided on the seat surface or the steering wheel or the like, it detects the biological information of the driver gripping the rider or the steering wheel sitting on the seat. Vehicle information detection unit 7500, based on the detection information input from the operation detection section 7510 may calculate the degree of fatigue or concentration degree of the driver, determine the driver is not drowsy it may be. Vehicle information detection unit 7500 may perform processing such as noise canceling processing for collected sound signal.
[0232]
　Integrated control unit 7600 controls the overall operation of the vehicle control system 7000 according to various programs. The integrated control unit 7600, input unit 7800 is connected. The input unit 7800 is, for example, a touch panel, button, microphone, switch or lever or the like, is realized by an input operation which can be the device by the rider. The integrated control unit 7600, the data may be input obtained by recognizing speech a speech input by a microphone. Input unit 7800, for example, may be a remote control device using infrared rays or other radio waves, there external connection device such as a mobile phone or PDA (Personal Digital Assistant) corresponding to the operation of the vehicle control system 7000 it may be. Input unit 7800, for example, may be a camera, in which case the passenger can input information by gestures. Alternatively, data occupant is obtained by detecting the motion of the wearable device mounted may be input. Further, the input unit 7800, for example, based on information entered by the rider or the like using the input unit 7800 of the generated input signal may include an input control circuit for outputting the integrated control unit 7600. Passenger or the like, by operating the input unit 7800, and instructs the input processing operation of various types of data to the vehicle control system 7000.
[0233]
　Storage unit 7690 is, ROM (Read Only Memory) that stores various programs executed by the microcomputer, and various parameters, RAM for storing the operation result or the sensor value or the like (Random Access Memory) may be contained. The storage unit 7690 is, HDD (Hard Disc Drive) such as a magnetic storage device, a semiconductor storage device may be realized by an optical storage device, or magneto-optical storage device or the like.
[0234]
　General-purpose communication I / F7620 is a universal communication I / F which mediates communication between the various devices present in the external environment 7750. General-purpose communication I / F7620 is, GSM (Global System of Mobile communications), WiMAX, LTE (Long Term Evolution) or LTE-A (LTE-Advanced) cellular communication protocol such as, or wireless LAN (Wi-Fi (registered trademark) also referred to), it may implement other wireless communication protocol such as Bluetooth (registered trademark). General-purpose communication I / F7620, for example, via a base station or access point, connected to an external network (e.g., Internet, cloud network or operator-specific network) devices existing on (e.g., an application server or the control server) it may be. Further, general-purpose communication I / F7620, for example by using a P2P (Peer the To Peer) technology, terminals present in the vicinity of the vehicle (e.g., the driver, the pedestrian or the shop terminal, or MTC (Machine Type Communication) terminal) it may be connected to.
[0235]
　Dedicated communication I / F7630 is a communication I / F that support the communication protocol developed for the purpose of use in a vehicle. Dedicated communication I / F7630 is, for example, a combination of the IEEE1609 of IEEE802.11p and the upper layer of the lower layer WAVE (Wireless Access in Vehicle Environment), a standard protocol such as DSRC (Dedicated Short Range Communications), or a cellular communications protocol it may be implemented. Dedicated communication I / F7630 is, typically, vehicle-to-vehicle (Vehicle to Vehicle) communication, road-to-vehicle (Vehicle to Infrastructure) communication, communication and step-to-vehicle (Vehicle to Pedestrian between the vehicle and the house (Vehicle to Home) ) performs V2X communication is a concept including one or more of the communication.
[0236]
　Positioning unit 7640 is, for example, GNSS (Global Navigation Satellite System) GNSS signals from satellites (eg, GPS (Global Positioning System) GPS signals from the satellites) and performing positioning by receiving, latitude of the vehicle, longitude and altitude It generates position information including the. Incidentally, the positioning unit 7640 may identify the current position by exchanging signals with a wireless access point, or a cellular telephone with a positioning function, may acquire the positional information from the terminal, such as PHS or a smartphone.
[0237]
　Beacon reception unit 7650 receives, for example, radio waves or electromagnetic waves transmitted from the road radio station installed on such as the current position, traffic jam, and acquires the information of the road closures or required time or the like. The function of the beacon receiver 7650 may be included in a dedicated communication I / F7630 mentioned above.
[0238]
　Interior equipment I / F7660 is a communication interface that mediates the connection between the various car devices 7760 that are present in the vehicle and a microcomputer 7610. Interior equipment I / F7660 is a wireless LAN, Bluetooth (registered trademark), may establish a wireless connection using a wireless communication protocol such as NFC (Near Field Communication) or WUSB (Wireless USB). Also, interior equipment I / F7660 via a connection terminal (not shown) (and, if necessary cables), USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface), or MHL (Mobile High-definition Link ) a wired connection may be established such. Vehicle equipment 7760, for example, a mobile device or wearable device passenger has, or may include at least one of which is carried on the vehicle or mounting is information equipment. Also, the car device 7760 may include a navigation device that performs route search to any destination. Interior equipment I / F7660 is between these vehicle devices 7760, for exchanging control signals or data signals.
[0239]
　Vehicle network I / F7680 is an interface that mediates communication between the microcomputer 7610 and the communications network 7010. Vehicle network I / F7680 is according to a specific protocol supported by the communication network 7010 to transmit and receive signals and the like.
[0240]
　The microcomputer 7610 of the integrated control unit 7600, via at least one of a general-purpose communication I / F7620, dedicated communication I / F7630, the positioning unit 7640, a beacon receiving unit 7650, interior equipment I / F7660 and vehicle network I / F7680 based on the information acquired Te, in accordance with various programs, for controlling the vehicle control system 7000. For example, the microcomputer 7610, based on the vehicle outside of the information obtained, the drive force generation unit, and calculates a control target value of the steering mechanism or braking device, outputs a control command to the driving system control unit 7100 it may be. For example, the microcomputer 7610, the collision avoidance or cushioning of the vehicle, follow-up running based on inter-vehicle distance, vehicle speed maintained running, functions realized in the vehicle collision warning, or ADAS including lane departure warning of the vehicle (Advanced Driver Assistance System) it may be carried out cooperative control for the purpose of. The microcomputer 7610, the driving force generating device on the basis of the information around the vehicle to be acquired, by controlling the steering mechanism or braking device, autonomously traveling without depending on the operation of the driver automatic or the like may be cooperative control is performed for the purpose of operation.
[0241]
　The microcomputer 7610 is general-purpose communication I / F7620, information obtained via at least one of a dedicated communication I / F7630, the positioning unit 7640, a beacon receiving unit 7650, interior equipment I / F7660 and vehicle network I / F7680 the basis to generate a three-dimensional distance information between the object of the structure and the person of the vehicle and the surrounding, may create a local map information including the peripheral information of the current position of the vehicle. The microcomputer 7610 is based on the information obtained, the collision of the vehicle, predicting the risk of entry such as to close or closures of the road such as a pedestrian, may generate a warning signal. Warning signals, for example, or to generate a warning sound may be a signal for or to light the warning lamp.
[0242]
　Audio and image output unit 7670 transmits, to the passenger or outside of the vehicle, at least one of the output signal of the voice and image to be output device to inform a visually or aurally information. In the example of FIG. 23, as an output device, an audio speaker 7710, display unit 7720 and the instrument panel 7730 is illustrated. Display unit 7720 is, for example, may include at least one of the on-board display and head-up display. Display unit 7720 may have a AR (Augmented Reality) display function. Output device, other than those devices, headphone, wearable devices eyeglass display like the rider mounted, may be another device such as a projector or lamp. If the output device is a display device, a display device, the information received from the obtained results or other control unit by processing of the microcomputer 7610 has performed, text, images, tables, graphs, etc., in various forms visually display. Also, if the output device is an audio output device, audio output device, aurally converts the audio signal consisting of audio data or sound data or the like which is reproduced into an analog signal.
[0243]
　In the example shown in FIG. 23, at least two control units connected via a communication network 7010 may be integrated as one control unit. Alternatively, individual control units may be constituted by a plurality of control units. Further, the vehicle control system 7000 may be provided with a separate control unit (not shown). In the above description, some or all of any of the control units is responsible function, may be provided to other control units. That is, if designed to be transmission and reception of information via a communications network 7010, predetermined arithmetic processing may be adapted to be performed in any of the control units. Similarly, one of the connected sensor or device control unit is connected to the other control units, a plurality of control units may send and receive detection information with each other via a communication network 7010 .
[0244]
　Incidentally, the computer program for realizing each function of the distribution system 10 according to the present embodiment described with reference to FIGS. 1 to 21 may be implemented in any of the control unit or the like. Further, such a computer program is stored, it is also possible to provide a computer readable recording medium. The recording medium may be a magnetic disk, an optical disk, a magneto-optical disk, a flash memory or the like. The computer program, without using the recording medium, for example may be distributed via a network.
[0245]
　The vehicle control system 7000 described above, the case of applying the delivery system 10 according to this embodiment described with reference to FIGS. 1 to 21, for example, imaging apparatus 11 of the distribution system 10, at least a portion of the imaging unit 7410 It corresponds to. Also, generator 12, the distribution server 13, the reproducing apparatus 15 is integrated, which corresponds to the microcomputer 7610 and the storage unit 7690. Head-mounted display 16 corresponds to the display unit 7720. When applying the delivery system 10 to the integrated control unit 7600, the network 14, the camera 15A, the markers 16A and the gyro sensor 16B, is not provided, the line of sight of the viewer by operating the input unit 7800 of the passenger is viewers direction and the viewing position are input. As described above, the distribution system 10, by applying to the integrated control unit 7600 of the application example shown in FIG. 23, the image quality of all viewing direction of the display image generated by using the omnidirectional image same it can be to such an extent.
[0246]
　Moreover, at least some of the components are integrated circuit modules comprised of modules for the integrated control unit 7600 (e.g., one of the die shown in FIG. 23 of the delivery system 10 described with reference to FIGS. 1 to 21 may be implemented in). Alternatively, the delivery system 10 described with reference to FIGS. 1 to 21 may be realized by a plurality of control units of the vehicle control system 7000 shown in FIG. 23.
[0247]
　Further, in this specification, a system includes a plurality of components (devices, modules (components) or the like) means a set of, it does not matter whether or not there is in the same housing all components. Therefore, housed in a separate enclosure, a plurality of devices connected via a network, and one device in which a plurality of modules within a single casing is housed, both of which the system .
[0248]
　Note that the effect described herein is not intended to be limited to a merely illustrative, there may be other effects.
[0249]
　Further, embodiments of the present disclosure is not intended to be limited to the embodiments described above, but various modifications are possible without departing from the scope of the present disclosure.
[0250]
　For example, the present disclosure, one function shared by a plurality of devices via a network, it is possible to adopt a configuration of cloud computing which processes jointly.
[0251]
　Further, each step described in the above flowcharts may be executed by one device, it can be performed by allocating a plurality of apparatuses.
[0252]
　Further, when a plurality of processes are included in one step, the plurality of processes included in the one step may be executed by one device, it can be performed by allocating a plurality of apparatuses.
[0253]
　The present disclosure can also be configured as below.
[0254]
　(1)
　and the resolution reduction unit for low resolution of the omnidirectional image,
　by projecting the omnidirectional image mapped to the 3D model into a plurality of two-dimensional plane, and a projection unit for generating a plurality of images
　generating device comprising a.
　(2)
　the low resolution has been the celestial sphere image by resolution reduction unit encodes the low-resolution coding section that generates a low-resolution stream,
　the plurality of images generated by the projection unit Numerals However, the high resolution encoding unit for generating a high-resolution stream
　and a transmitter for transmitting said low-resolution stream generated by the low-resolution coding section, the high-resolution stream generated by the high-resolution coding section
　further comprising a
　generating device according to (1).
　(3)
　the transmission unit transmits the two-dimensional plane information indicating a position of the plurality of two-dimensional plane
　configured to
　generate device according to (2).
　(4)
　a part of the image each other adjacent the 2-dimensional plane overlaps
　configured as
　generating apparatus according to any one of (1) to (3).
　(5)
　Normal line passing through the center of said plurality of two-dimensional plane is a line passing through the center of the midpoint between the cube of each side of the cube
　is configured as
　described in any one of (1) to (4) generating device.
　(6)
　the normal line passing through the center of said plurality of two-dimensional plane is a line passing through the line passing through the center of the midpoint between the cube of each side of the cube, the center of the cube and the center of each face of the cube there
　thus constituted
　generator according to any one of (1) to (4).
　(7)
　the omnidirectional image is made omnidirectional image omnidirectional image and the viewpoint of the right eye viewpoint for left eye,
　the resolution reduction unit, the viewpoint for the left eye that is lower resolution packing the celestial sphere image of the view for the right eye and the omnidirectional image by generating a low-resolution packed image,
　the projection portion is omnidirectional image of the view for the left eye that is mapped to the 3D model and wherein each omnidirectional image of the view for the right eye is generated by projecting the plurality of two-dimensional plane, a plurality of images of the viewpoint and the viewpoint of the right eye for the eye, each of the two-dimensional plane packing to generate a high-resolution packing image
　configured as
　generating apparatus according to any one of (1) to (6).
　(8)
　generating apparatus,
　a resolution reduction step to lower the resolution of the celestial sphere image,
　By projecting the omnidirectional image mapped to the 3D model into a plurality of two-dimensional plane, and a projection step of generating a plurality of image
　generation method comprising.
　(9)
　and at least one image among the plurality of images generated by projecting the omnidirectional image mapped to the 3D model into a plurality of two-dimensional plane, and the omnidirectional image which is lower resolution a receiving unit for receiving a
　said are the image and lower resolution received by the receiving unit based on at least one of the celestial sphere image, the drawing section for generating a display image
　reproducing device provided with.
　(10)
　based on the line-of-sight direction of the viewer, from the plurality of images, the receiving unit selector for selecting the image to be received by
　further comprising a
　configured as
　reproducing apparatus according to (9).
　(11)
　The receiving unit receives the two-dimensional plane information indicating a position of the plurality of two-dimensional plane,
　the selection unit selects the image based on the two-dimensional plane information and the viewing direction
　configured It has been
　reproducing apparatus according to (10).
　(12)
　Mapping the image into the two-dimensional plane, by mapping the omnidirectional image which is lower resolution in the 3D model, the mapping processing unit for generating a 3D model image
　further comprising a
　said drawing unit, the the 3D model image generated by the mapping processing unit generates the display image by projecting a field of view of the viewer
　configured as
　reproducing apparatus according to (9).
　(13)
　a lower resolution has been the celestial sphere image by decoding the low resolution stream generated by encoding, and the low-resolution decoding unit which generates the omnidirectional image which is lower resolution,
　said plurality of at least one of the images by decoding the high resolution stream generated by encoding, and a high resolution decoding unit which generates the image
　further comprising a
　said receiving unit is configured with the low-resolution stream receive a high-resolution stream
　configured as
　reproducing apparatus according to any one of (9) to (12).
　(14)
　a part of the image each other adjacent the 2-dimensional plane overlaps
　configured as
　reproducing apparatus according to any one of (9) to (13).
　(15)
　Normal line passing through the center of said plurality of two-dimensional plane is a line passing through the center of the midpoint between the cube of each side of the cube
　is configured as
　described in any one of (9) to (14) reproducing apparatus.
　(16)
　normal line passing through the center of said plurality of two-dimensional plane is a line passing through the line passing through the center of the midpoint between the cube of each side of the cube, the center of the cube and the center of each face of the cube there
　thus configured
　reproducing apparatus according to any one of (9) to (14).
　(17)
　the lower resolution has been omnidirectional image is an image obtained by packing the celestial sphere image of the view for the right eye and the omnidirectional image resolution reduction has been viewpoint for left eye,
　the plurality of images , for each said 2-dimensional plane, is produced by projecting the omnidirectional image of the view for omnidirectional image and the right eye viewpoint for the left eye that is mapped to the 3D model in each of the two-dimensional plane images are packed images the
　configured such
　reproducing apparatus according to any one of (9) to (11).
　(18)
　the drawing unit, the omnidirectional image which is the image and the lower resolution received by the receiving portion is separated for each viewpoint, each viewpoint, is the image and the low resolution is divided and based on at least one of the celestial sphere image, and generates the display image
　is configured to
　Reproducing apparatus according to (17).
　(19)
　reproducing apparatus,
　at least one image among the plurality of images generated by projecting the omnidirectional image mapped to the 3D model into a plurality of two-dimensional plane, the whole being lower resolution a receiving step of receiving a celestial sphere image,
　and drawing steps based on at least one of the image and lower resolution has been the celestial sphere image received by said receiving step, to generate a display image
　reproduction comprising Method.
DESCRIPTION OF SYMBOLS
[0255]
　12 generator, 23 resolution reduction unit, 24 an encoder, 26-1 to 26-5 perspective projection unit, 27-1 to 27-5 encoder, 29 transmission unit, 40 balls, 41 and 45 a two-dimensional plane, 121 receiving unit , 122 a decoder, 123 receiving unit, 124 a decoder, 125 mapping unit, 126 rendering unit 128 sight line detecting unit, 143 a two-dimensional plane, 151 low-resolution images, 152 high-resolution image, 170 display images, 171 high-resolution image, 172 low-resolution image, 180 display an image, 181 a high resolution image, 182 a low-resolution image, 190 display an image, 191 a high resolution image, 192 low-resolution image

claims
[Requested item 1]
　A resolution reduction unit for low resolution of the omnidirectional image,
　by projecting the omnidirectional image mapped to the 3D model into a plurality of two-dimensional plane, and a projection unit for generating a plurality of images
　generated with a apparatus.
[Requested item 2]
　Wherein the resolution reduction unit for lower resolution has been the celestial sphere image encoding, a low-resolution coding section that generates a low-resolution stream,
　the plurality of images generated by the projection unit respectively coded, high high resolution coding unit for generating the resolution stream and
　the said low-resolution stream generated by the low-resolution coding section, a transmission unit that transmits the high-resolution stream generated by the high-resolution coding section
　further comprises
　generating apparatus according to claim 1.
[Requested item 3]
　The transmitting unit transmits the two-dimensional plane information indicating a position of the plurality of two-dimensional plane
　configured as
　generating apparatus according to claim 2.
[Requested item 4]
　Some overlap of the images to each other adjacent the two-dimensional plane
　configured as
　generating apparatus according to claim 1.
[Requested item 5]
　Normal line passing through the center of said plurality of two-dimensional plane is a line passing through the center of the midpoint between the cube of each side of the cube
　is configured to
　generate device according to claim 1.
[Requested item 6]
　Normal line passing through the center of said plurality of two-dimensional plane is a line passing through the line passing through the center of the midpoint between the cube of each side of the cube, the center of the cube and the center of each face of the cube
　as configured
　generator according to claim 1.
[Requested item 7]
　The omnidirectional image is made omnidirectional image of the view for the right eye and the omnidirectional image of the view for the left eye,
　the resolution reduction unit, omnidirectional viewpoint for the left eye that is lower resolution and packing the celestial sphere image in the image and the viewpoint for the right eye to generate a low-resolution packed image,
　the projection portion is omnidirectional image and the right-eye of the viewpoint for the left eye that is mapped to the 3D model each omnidirectional image of the view is generated by projecting the plurality of two-dimensional plane, a plurality of images of the viewpoint and the viewpoint of the right eye for the left eye, and packed in each of the two-dimensional plane generating a high-resolution packed image Te
　configured as
　generating apparatus according to claim 1.
[Requested item 8]
　Generating device,
　a low resolution steps of the low resolution of the omnidirectional image,
　the projection step by projecting the omnidirectional image mapped to the 3D model into a plurality of two-dimensional plane, for generating a plurality of images capital
　generation method, including.
[Requested item 9]
　Receiving receiving at least one image among the plurality of images generated by projecting the omnidirectional image mapped to the 3D model into a plurality of two-dimensional plane, and said omnidirectional image which is lower resolution parts and,
　based on at least one of the image and lower resolution has been the celestial sphere image received by the receiving unit, and a drawing unit that generates a display image
　reproducing device provided with.
[Requested item 10]
　Based on the line-of-sight direction of the viewer, from the plurality of images, selecting section for selecting the image received by said receiving unit
　further comprises a
　configured as
　reproducing apparatus according to claim 9.
[Requested item 11]
　The receiving unit receives the two-dimensional plane information indicating a position of the plurality of two-dimensional plane,
　the selection unit selects the image based on the viewing direction and the two-dimensional plane information
　is configured
　according reproducing apparatus according to claim 10.
[Requested item 12]
　Mapping the image into the two-dimensional plane, by mapping the omnidirectional image which is lower resolution in the 3D model, the mapping processing unit for generating a 3D model image
　further comprising a
　said drawing unit, the the 3D model image generated by the mapping processing unit generates the display image by projecting a field of view of the viewer
　configured as
　reproducing apparatus according to claim 9.
[Requested item 13]
　Decoding the low-resolution stream generated by encoding the low resolution has been the omnidirectional image, and the low-resolution decoding unit which generates the omnidirectional image which is low resolution,
　the plurality of images at least one image of the decoded high-resolution stream generated by encoding the image and high-resolution decoding section that generates
　further comprising a
　said receiving unit, said high-resolution stream and the low resolution stream receive
　configured as
　reproducing apparatus according to claim 9.
[Requested item 14]
　Some overlap of the images to each other adjacent the two-dimensional plane
　configured as
　reproducing apparatus according to claim 9.
[Requested item 15]
　Normal line passing through the center of said plurality of two-dimensional plane is a line passing through the center of the cube and the midpoint of each side of the cube
　constructed as
　reproducing apparatus according to claim 9.
[Requested item 16]
　Normal line passing through the center of said plurality of two-dimensional plane is a line passing through the line passing through the center of the midpoint between the cube of each side of the cube, the center of the cube and the center of each face of the cube
　as constructed
　reproducing apparatus according to claim 9.
[Requested item 17]
　The lower resolution has been omnidirectional image is an image obtained by packing the celestial sphere image of the view for the right eye and the omnidirectional image resolution reduction has been viewpoint for left eye,
　the plurality of images, the for each two-dimensional plane, an image generated by projecting the omnidirectional image of the view of the celestial sphere image and the right-eye of the viewpoint for the left eye that is mapped to the 3D model in each of the two-dimensional plane It is packed images
　constructed as
　reproducing apparatus according to claim 9.
[Requested item 18]
　The drawing unit, the omnidirectional image which is the image and the lower resolution received by the receiving portion is separated for each viewpoint, the respective viewpoints, divided the image and the lower resolution has been all sky based on at least one of the spherical image, to generate the display image
　configured as
　reproducing apparatus according to claim 17.
[Requested item 19]
　Reproducing apparatus,
　and at least one image among the plurality of images generated by projecting the omnidirectional image mapped to the 3D model into a plurality of two-dimensional plane, the lower resolution has been the celestial sphere image a receiving step of receiving the door,
　based on at least one of the image and lower resolution has been the celestial sphere image received by said receiving step, a drawing step of generating a display image
　reproducing method comprising.