Technical field
[0001]
This technique, generating device, the identification information generating method, reproducing apparatus and relates to an image generation method, in particular generating apparatus that margin there region and margins without region can be easily obtained, the identification information generating method, reproducing apparatus and image relates to the production method.
BACKGROUND
[0002]
　From the captured image captured by the multi-camera, generates a celestial sphere image mapped images of horizontal 360 and vertical 180 degrees around the periphery of the 2D image (plane image), a recording apparatus for encoding and recording is (e.g., see Patent Document 1).
[0003]
　In such a recording apparatus, a method of generating a celestial sphere image, and a method and cubes mapping method using equirectangular is used. If a method of method of generating omnidirectional images using equirectangular, omnidirectional image is an image of equirectangular sphere when the captured image mapped to the surface of the sphere. Also, if the method of generating omnidirectional images are cube mapping method, omnidirectional image is an image of a developed view of a cube when the captured image mapped on the face of the cube (a cube).
[0004]
　On the other hand, as the streaming method of video content, there is MPEG-DASH (Moving Picture Experts Group phase-Dynamic Adaptive Streaming over HTTP). In MPEG-DASH, the management file for managing an encoded stream of video content is transmitted from the distribution server to the client, the client selects the encoded stream to be reproduced based on the management file, requests the distribution server .
CITATION
Patent Document
[0005]
Patent Document 1: JP 2006-14174 JP
Summary of the Invention
Problems that the Invention is to Solve
[0006]
　Meanwhile in the omnidirectional image, it has the following problems.
　(1) image quality is deteriorated when connecting the adjacent faces on the project Ted frame (Projected frame) on a non-continuous and projection structure (projection structure).
　(2) such as a cube projection mapping, the image quality is deteriorated when supplement texture pixel in multi-projection structure (projection structure) surface boundary at the time of use.
　(3) a whole low-resolution omnidirectional image, when only a specific region in the vicinity of the center of the line of sight is such that high resolution image, the superimposed high-resolution layer image to a lower resolution layer image, the image quality of the boundary Getting worse.
　Therefore suppressing the deterioration in image quality by providing the margin area around the omnidirectional image is expected.
[0007]
　As a method for distributing a margin, or deliver a number of pixels is extended as a margin from the margin region without, it is known to deliver a proportion of the margin of the margin there region. However, in this case, the client had in recognizing the areas with and without margin, to be carried out region (region) calculation processing in accordance with the delivery method. As a result, the client it is difficult to obtain a readily margin there region and margins region without.
[0008]
　This technology has been made in view of such circumstances, it is desirable to make it possible to easily obtain a margin there area and the margin region without.
Means for Solving the Problems
[0009]
　According to an embodiment of the present technology, omnidirectional image, the the area of ​​the celestial sphere image, in generating apparatus including an identification information generation unit for generating a margin identification information for identifying comprises a region having the generated margin is there.
[0010]
　The generator, a margin generator for generating a margin area of ​​the omnidirectional image can be further provided.
[0011]
　The margin may be formed outside of the region.
[0012]
　Region information of a region having the margin may be represented by a spherical coordinate system or two-dimensional coordinate system.
[0013]
　The area information may be represented as information Projected frame or packed frame.
[0014]
　The area information of an area having the margin of the Projected frame may be the region is described in the case with the margin.
[0015]
　The area information of an area having the margin of the packed frame may be the region is described when it is region-wise packing.
[0016]
　The area information of an area having the margin of the Projected frame, the Projected frame the margin includes area width and height, and the allocation identification information for identifying the allocation of the margin of the width and the height direction it can be included.
[0017]
　The packed the area information of an area having the margin frame, the Projected frame the margin includes area width and height of, and may include upper left coordinates of the area.
[0018]
　The area information of an area having the margin of the packed frame may be the packed frame is described if they are region-wise packing.
[0019]
　The allocation identification information may be omitted.
[0020]
　The area information of an area having the margin of the Projected frame is omitted, only the region information of a region having the margin of the packed frame can be written.
[0021]
　The margin may be that there is a type not to have the type having a non-formation portion where the margin in the corner is not formed.
[0022]
　The identification information generating unit may further generate the type identification information for identifying the type.
[0023]
　The area information of an area having the margin of the Projected frame, the type identification information can be written when the a type having no non-formation portions.
[0024]
　Said region is a triangle, when the margin along the sides are formed, by molding the triangle rectangle, it is possible to arrange the said margin to said side of the rectangle corresponding to the triangle.
[0025]
　The margin identification information may be described in the box under the Scheme Information Box of ISOBMFF.
[0026]
　The margin identification information may be described in the MPD file MPEG-DASH.
[0027]
　According to an embodiment of the present technology, generating device, omnidirectional image, the the area of ​​the celestial sphere image, the identification information generating step of generating a margin identification information for identifying comprises a region having the generated margin is identification information generation method comprising.
[0028]
　According to an embodiment of the present technology, an acquisition unit omnidirectional image to obtain a margin identification information for identifying comprises a region having a margin, and a generator for generating a Projected frame based on the acquired margin identification information a reproducing apparatus and a rendering unit rendering the Projected frame.
[0029]
　According to an embodiment of the present technology, the reproduction apparatus, generates an acquisition step of omnidirectional images to obtain a margin identification information for identifying comprises a region having a margin, the Projected frame based on the acquired margin identification information a generation step of, an image generating method and a rendering step of rendering the Projected frame.
[0030]
　In one aspect of the present disclosure, the identification information generating unit, the celestial sphere image, the the area of ​​the celestial sphere image to generate a margin identification information for identifying comprises a region having the generated margin.
Effect of the invention
[0031]
　As described above, according to an embodiment of the present technology, it is possible to easily obtain a margin there area and the margin region without. Note that the effect described herein is merely illustrative, and not limitation, also may be additional effects.
BRIEF DESCRIPTION OF THE DRAWINGS
[0032]
[1] is a diagram for explaining a projector Ted frame and packed frame.
Is a diagram illustrating an example of FIG. 2 margin.
Is a diagram illustrating an example of FIG. 3 margin.
Is a diagram showing a pattern of FIG. 4 margin.
Is a diagram illustrating a FIG. 5 margin pattern.
It is a diagram illustrating a configuration example of FIG. 6 Region mapping box.
It is a diagram illustrating the fields of FIG. 7 Region mapping box.
It is a diagram illustrating the fields of FIG. 8 Region mapping box.
9 is a diagram illustrating a margin of a cube matching projection.
10 is a diagram for explaining the field at the time of a cube matching projection.
11 is a diagram illustrating a margin of equirectangular projection.
12 is a diagram for explaining the field during equirectangular projection.
It is a diagram illustrating a configuration example of FIG. 13 Region mapping box.
It is a diagram illustrating a configuration example of FIG. 14 Region mapping box.
It is a diagram illustrating a configuration example of FIG. 15 Region mapping box.
It is a diagram illustrating a configuration example of FIG. 16 Region mapping box.
Is a diagram illustrating the fields of FIG. 17 Region mapping box.
18 is a diagram illustrating a margin of a cube matching projection.
19 is a diagram explaining the field during cube matching projection.
Is a diagram illustrating an example of FIG. 20 margin arrangement.
21 is a diagram showing an example of a margin arrangement.
22 is a diagram showing an example of a margin arrangement.
It is a diagram illustrating an example of MPD file to which the FIG. 23] DASH.
It is a block diagram showing a configuration example of FIG. 24 distribution system.
It is a block diagram showing a configuration example of FIG. 25 generator.
Is a flow chart illustrating the FIG. 26] ISOBMFF generation process.
FIG. 27 is a block diagram showing a configuration example of a playback apparatus.
Is a flow chart illustrating the FIG. 28] ISOBMFF regeneration process.
FIG. 29 is a block diagram showing a configuration example of hardware of a computer.
DESCRIPTION OF THE INVENTION
[0033]
　The following describes exemplary embodiments of the present technology. The description will be made in the following order.
　1. Embodiment
　　principles (1) projection (Fig.
　　1) (2) margin (FIGS. 2 to
　　23) (3) delivery system (Fig.
　　24) (4) generator (25,
　　26) (5) reproducing device (Figure 27, Figure
　28) 2. Computer (Fig.
　29) 3. Otherwise
[0034]
　
　
　In the present technology, omnidirectional image is delivered as a video stream to a client for example from the server, receiving at the client side, it reproduced and viewed. Therefore generation of first omnidirectional image, a description will be given of the principle of a process for delivery.
[0035]
　In this technology, the projector Ted frame celestial sphere image (Projected frame) and packed frame (packed frame) is generated. Figure 1 is a diagram for explaining a projector Ted frame and packed frame. As shown in FIG. 1, omnidirectional image (omnidirectional image) is captured by the camera 1. The omnidirectional image is the direction of the image up, down, left, and right 360 degrees. In the following, when referred to a word in katakana, if rather seems incomprehensible describes in English.
[0036]
　Omnidirectional image is projected (projection) on the projection structure (projection structure) projected frame is obtained. The packed frame is obtained by the Projected frame and arranged to change the position and size for each area on a two-dimensional plane packing (packing). Thus, to packing by changing at least one of the position and size for each area of ​​Region-wise packing (region-wise packing). In packed frame, each region is disposed so as to be rectangular throughout. By using the packed frame, to expand the resolution of the region that is high quality is desired, by decreasing the resolution of the good areas of low quality, it is possible to optimize the transmission capacity.
[0037]
　In the example of FIG. 1, the sphere 11 and a cube 12 is shown as a projection structure. The omnidirectional image subjected to projection by equirectangular the ball 11 (equirectangular projection), projected frame13 is obtained by representing it in two dimensions. In this example, Projected Frame 13 includes a central region A, the region B which is located thereon, and is constituted by a region C located underneath.
[0038]
　The packed Frame15 obtain a Projected Frame 13 by region-wise packing. In this example, the resolution of the area A is as it is, the area B is arranged on the left side thereon, region C is disposed on the right side. Region B and region C resolution is reduced. Hereinafter also referred to simply area or region image.
[0039]
　The omnidirectional image projected onto the cube 12 (performs cube mapping projection), it Projected Frame14 obtain a by a two-dimensional representation. The front of the cube 12 (front face), right side (. Right face), rear (back face), left side (left face), the upper surface of the (top face), and the bottom surface 6 faces the (bottom face) (region) image is disposed composed Projected Frame14 with 4 × 3 pieces of 12 total area. Four regions of the central, sequentially from the left, respectively left side (left), front (front), right side (. Right), area image of the rear (back) is disposed in the area above the front (front) is disposed area image of the top surface (top), it is disposed region image of the bottom surface (bottom) in the area below.
[0040]
　packed frame16 can be obtained by the projected frame14 to region-wise packing. In this example, the expanded-resolution area image the front (front), the resolution of the other region image is as it is. Region image of the left side surface of the upper area images of the front (front) (left) is, on its right side is disposed area image of the top surface (top), in the right area image the front (front), the upper in order from the right side (. right), rear (back), and a region image of the bottom surface (bottom) are disposed.
[0041]
　
　Next margins will be described. Figure 2 and Figure 3 is a diagram showing an example of a margin.
[0042]
　Figure 2 shows an example of a margin in cube mapping projection. In A of FIG. 2, in order from the left to the center line, region image 41 of the respective left side (left), front area images 42, right side of the (front) region image 43 and rear (right) (back) area image 44 is disposed. Also, it is arranged region image 45 of the upper surface (top) over the region image 42 of the front (front), region image 46 of the bottom surface (bottom) is disposed below.
[0043]
　In the example of A of FIG. 2, the margin of the entire circumference are formed on the outside of each side of the region. For example, in the left side surface, the area image 41 including a margin is formed from the inner region 41a to the margins 41b of the outer periphery of the front, forming region image 42 including the margin from the inner area 42a and the margin 42b of the outer periphery It is. In the right side surface, the area image 43 including a margin is formed from the inner region 43a to the margins 43b of the outer periphery thereof in the back region image 44 including a margin is formed from the inner region 44a to the margins 44b of the outer periphery ing. In the upper surface, the area image 45 including a margin is formed from the inner region 45a to the margins 45b of the outer periphery of the bottom surface, area image 46 including a margin is formed from the inner area 46a and the margin 46b of the outer periphery there.
[0044]
　This example, when polygon projection structure used, is used in order to suppress deterioration of image quality when complementing the texture pixels in the surface boundary.
[0045]
　In the example of B of FIG. 2, the margin is formed on the outside of each side of the region. From left to right, for example, in the middle row, region image 61 of the respective left side (left), a region image 64 of the front region image 62 (front), the right side area image 63 and rear (right) (back) arranged It is. Also, it is disposed area image 65 of the upper surface (top) over the region image 62 of the front (front), region image 66 of the bottom surface (bottom) is disposed below.
[0046]
　Of these area images, and margins 61b is formed on the upper side and the lower side of the area image 61, the margin 63b is formed on the upper side and the lower side of the area image 63. In the region image 64, the upper and not lower only, and margins 64b are formed in the right side. In this example, each margin 61b, 63 b, because 64b is formed outside of each area image 61, 63, 64, the margin of the inner region 61a, 63a, 64a is an area image itself.
[0047]
　This example is used to suppress image quality deterioration that occur when connecting the discontinuous and projection structure in adjacent faces on Projected frame. Margin, when configuring the projection structure from Projected frame, a portion to be overlapped with the destination area.
[0048]
　3, in the case of superimposing a high-resolution layer image to the low-resolution layer image (Equirectangular projection) (cube mapping projection), an example of suppressing the deterioration of the image quality of the boundary. For example, with respect to the entire omnidirectional image which is a lower resolution, an example in which superimposing the high-resolution image a specific region only near the center of the line of sight. High-resolution layer image 81 is superimposed on the low-resolution layer image 71. High-resolution layer image 81 is formed of the inner region 81a, from the formed margin 81b to surround it.
[0049]
　For example, inner area 81a and the low-resolution layer image 71 of the high-resolution layer image is rendered as is. And margins 81b of the high-resolution layer image 81, the area of ​​the low-resolution layer image 71 which it overlaps, blending processing (dissolve, etc.) are displayed in place.
[0050]
　Figure 4 is a diagram showing a pattern of a margin. Margin of FIG. 4 is a margin of type 1. In A of FIG. 4, the margin 101b outward along the four sides of the region image 101 (inner region 101a) is formed. In B of FIG. 4, the margin 111b are formed on the outside along the upper side and the left and right sides of the area image 111, the inner is in the region 111a (region 111). In C of Fig. 4, the margin 121b are formed on the outside along the upper side and right side of the area image 121, the inner is in the region 121a (region 121).
[0051]
　In D in FIG. 4, the margin 131b are formed on the outer side along the left side and the right side of the area image 131, the inner is in the region 131a (region 131). In E in FIG. 4, the margin 141b are formed on the outside along the right side of the area image 141, the inner is in the region 141a (region 141). In this pattern 1, the non-formation portion as described later pattern 2 is not provided.
[0052]
　Figure 5 is a diagram showing a pattern of a margin. Margin of FIG. 5 is a margin of type 2. In A of FIG. 5, the margin 161b are formed on the outer side along the four sides of the area image 161, the inner is in the region 161a (region 161). In margin 161b, in its four corners, non-formation portion 161c is a portion not forming the margin 161b (deleted part) is formed.
[0053]
　In B of FIG. 5, the margin 171b is formed along the upper side and the left and right sides of the area image 171, the inner is in the region 171a (region 171). The two corners of the margin 171b, non-formation portion 171c is a portion not forming the margin 171b (deleted part) is formed.
[0054]
　In C of Fig. 5, the margin 181b are formed on the outside along the upper side and right side of the area image 181, the inner is in the area 181a (area 181). In one corner of the margin 181b, non-formation portion 181c is a portion not forming the margin 181b (deleted part) is formed.
[0055]
　Thus, the margin of the type 2 has a non-formation portions in the corner of adjacent sides (no corners). The margin of type 1 contrast, no non-formation portion (having a corner). The margin of type 1, to deliver the margin there is a region in the information of the width and height. The margin of type 2, but is distributed by the information likewise width and height as the margin type 1, of the margin there region, the corners of the margin (non-formation portion) is processed as a region obtained by deleting the margin .
[0056]
　In this technique, using the concept of Projected frame and packed frame, various margin arrangement of Figure 4 and Figure 5, is distributed using RegionMappingBox. Both each region there margin without the margin, are those indicated by the position and size, the client may also retrieve any region easily. Also, omnidirectional image is whether with a region having a margin, are delivered Margin_flag.
[0057]
　RegionMappingBox, for example under the Scheme Information Box defined by ISOBMFF ( 'schi'), but is disposed at SchemeType = 'rmap', can be located elsewhere. Below, it shows an example of a RegionMappingBox.
[0058]
　Figure 6 is a diagram illustrating a configuration example of a region mapping boxes 7 and 8 are diagrams for explaining the fields of the region mapping box.
[0059]
　Figure 6 shows the configuration of RegionMappingBox when showing areas in the projection structure information (area information of Projected frame) in a spherical coordinate system. Center of the area there between the margin region without the projection structure is, center_yaw, indicated generally at Center_pitch. The Region_margin_type, can be switched presence and type of margin margin for each area.
[0060]
　In the example of FIG. 6, projection_format to RegionMappingBox, packing_flag, FOV_flag, margin_flag, num_regions are described. projection_format represents projection type, the value 0 means equirectangular projection, the value 1 means the cube mapping projection. packing_flag is representative whether using region-wise packing, the value 0 means that no use of a region-wise packing, the value 1 means that the use of region-wise packing.
[0061]
　FOV_flag is object_width, represents a reference angle information object_height, the value 0 means on spherical, the value 1 means on the perspective projection plane. As will be described later with reference to FIG. 9, object_width represents a region of the width (angle) in the projection structure, object_height represents the height of the region in the projection structure (angle).
[0062]
　margin_fkag is omnidirectional image represents whether including an area of ​​the margin, the value 0 means only the region of no margin, the value 1 means that there is a region of there margin. num_regions represents the number of regions of the packed frame.
[0063]
　RegionMappingBox are furthermore, center_yaw, center_pitch, object_width, object_height is described. These are areas information in a spherical coordinate system of the region of Projected frame. center_yaw represents the yaw of the area center of projection structure, which is common in the region without the margin there regions. center_pitch represents the pitch of the area center of projection structure, which is common in the region without the margin there regions.
[0064]
　If the value of packing_flag is true (value 1), rect_width, rect_height, rect_left, rect_top is described. These are areas information in a two-dimensional coordinate system in the region of the packed frame. rect_width represents the width of the margin region without the packed frame, rect_height represents the height of the margin region without that the packed frame. rect_left and rect_top each represent the x and y coordinates of the margin region without that the packed frame. These also further described later with reference to FIG.
[0065]
　If the value of margin_flag is true (value 1), region_margin_type is described. region_margin_type represents the type of margin of the area. The value 0 means an area without margin, the value 1 means the margin there region (with corner margin), the value 2 means a margin there region (corner without margins).
[0066]
　Further if (region_margin_type =! 0), i.e. (if the value 1 or value 2) region_margin_type if not 0, i.e. when the area is a region of there margin, object_with_margin_width, object_with_margin_height, width_margin_assignment, height_margin_assingment is described. These are areas information in a spherical coordinate system margin Projected frame.
[0067]
　object_with_margin_width represents the width of the area including a margin in the projection structure (angle). This is the value when the margin is allocated equally. object_with_margin_height represents the height of the region including the margin in the projection structure (angle). This is also a value in the case where the margin is allocated equally.
[0068]
　width_margin_assignment represents the allocation of the width direction of the margin of the area. The value 0 means allocated equally to both sides, or no margin, the value 1, only the right margin, left margin means that deletes the value 2, only the left margin, the right side of the margin means that you have removed. height_margin_assingment represents the assignment of the height direction of the margin of the area. The value 0 means or without margin allocation, equivalent to both sides, the value 1, only the upper margin and the lower margin means that deleted. Value of 2, only the lower margin, the upper margin means that deleted.
[0069]
　Further if Packing_flag is true, i.e., if it is packing_flag = 1 (region-wise_packing is used), rect_with_margin_width, rect_with_margin_height, rect_with_margin_left, rect_with_margin_top is described. These are areas information in a two-dimensional coordinate system margin packed frame. rect_with_margin_width represents the width of the area including the margin in the packed frame, rect_with_margin_height represents the height of the area including the margin in the packed frame. rect_with_margin_left and rect_with_margin_top each represent the x and y coordinates of the upper left corner of the area including the margin in the packed frame.
[0070]
　Above, for each field of RegionMappingBox, are summarized in Figure 7 and Figure 8.
[0071]
　Referring now to FIG. 9 and FIG. 10, using RegionMappingBox, described distribution margin cube mapping projection. Figure 9 is a diagram illustrating a margin of cube matching projection, FIG. 10 is a diagram for explaining the field during cube matching projection.
[0072]
　As shown in A of FIG. 9, left face of the cube 12 is projection structure, back surface, from the side of the right surface and front face, to the extent indicated by the outer edges 211, 212, 213 and 214 is the margin It is. Similarly, to the extent indicated by the outer edges 215, 216 of the top surface and the bottom surface is a margin.
[0073]
　B of FIG. 9 shows the right surface 221 of them. As origin at the center O of the cube 12 is projection structure, xyz coordinates (front x-axis is perpendicular axis plane, y axis is perpendicular axis to right side, the vertical axis on the top surface z it is assumed that axis). Line connecting the center C of the center O and right surface 221 is a line 230, (the case of FIG. 9 B, the same as the line 230) to line 230 projected lines xy coordinate plane and the angle between the x-axis There (in B of FIG. 9, -90 degrees) Center_yaw is. (For B in FIG. 9, 0 °) angle relative to the xy-plane line 230 Center_pitch is.
[0074]
　From the center O and perpendicular 236 relative to the left edge 223 of the right surface 221, and the angle is object_width of a perpendicular line 231 for the right side 222, a vertical line 233 with respect to the upper edge 224, the angle of the perpendicular 234 relative to the lower edge 225 There is a object_height.
[0075]
　Similarly, the margin a perpendicular 236 dropped to the left side 242 of the margin 213 from the center O, the angle of the perpendicular 235 dropped to the right side 241 is Object_with_margin_width, a perpendicular 237 dropped to the side 243 of the upper , the angle between the perpendicular line 238 hung down the sides 244 of the bottom is Object_with_margin_height.
[0076]
　C of FIG. 9 shows a packed frame251. In this configuration example, left from the left in the upper row in turn, front,. Right surfaces of the region image is disposed in the lower part, in order from the left, top, bottom, area image of each side of the back are arranged ing. packed Frame251 is constituted by overall 3 × 2 pieces of area images. The size of each area image is a 400 × 400 pixels, 40 pixels margin on the outside of the four sides is formed. Therefore, the size of the region image including the margin has a 480 × 480 pixels.
[0077]
　Therefore, the rect_with_margin_left the right surface including a margin 960, Rect_with_margin_top is 0, Rect_with_margin_width and rect_with_margin_height are all the 480. rect_left image area 1000, rect_top becomes 400 are both 40 becomes, Rect_width and Rect_height.
[0078]
　As shown in FIG. 9, specific values ​​of the fields of RegionMappingBox by cube mapping projection represented by the spherical coordinate system is shown in Figure 10. projection_format is 1 (cube mapping projection), packing_flag is (using region-wise packing) 1, margin_flag (including areas of there margin) 1, num_regions (the number of areas of the packed frame) is 6, FOV_flag 1 (perspective projection there is a on the surface). right surface of center_yaw, center_pitch, object_width, object_height are respectively -90,0,90,90.
[0079]
　right surface of the rect_width.rect_height, rect_left, rect_top has become, respectively 400,400,1000,40. region_margin_type 1 (margin there area (no corner margin)), the right surface including a margin object_with_margin_width, object_with_margin_height has a respectively 100, 100. width_margin_assignment, height_margin_assignment are respectively 0 (both sides Near East assignment (or without margin)). rect_with_margin_width of right plane including the margin, rect_with_margin_height, rect_with_margin_left, rect_with_margin_top has a respective 480,480,960,0.
[0080]
　The value of each field are summarized in Figure 10. projection_format is 1, packing_flag is 1, margin_flag is, num_regions is 6, FOV_flag is then equal to one. right surface of center_yaw, center_pitch, object_width, object_height are respectively -90,0,90,90. Also, right surface of rect_width, rect_height, rect_left, rect_top has a respective 400,400,1000,40.
[0081]
　region_margin_type the area including the margin of 1,. right surface object_with_margin_width, object_with_margin_height are respectively 100,100. width_margin_assignment, height_margin_assignment is set to 0 none. rect_with_margin_width region including the margin of right face, rect_with_margin_height, rect_with_margin_left.rect_with_margin_top are respectively 480,480,960,0.
[0082]
　Thus, the margin of each area (cube faces) are the type 1, region_margin_type = 1 is delivered. The width of the region including the respective margin, since it is evenly arranged in the vertical direction both, width_margin_assignment, height_margin_assignment are both delivered as 0.
[0083]
　Referring now to FIG. 11 and FIG. 12, using RegionMappingBox, an example of delivering a margin of equirectangular projection connexion be described. Figure 11 is a diagram illustrating a margin of equirectangular projection, FIG. 12 is a diagram for explaining the field during equirectangular projection.
[0084]
　In this example, for convenience, the area information is shown using Projected frame. The A in FIG. 11, there is shown a sphere 11 as projection structure, projected frame301 obtained by the projection is shown in B of FIG. 11. Image area is a margin 303 to the right of the (inner region) 302 is formed, for convenience, the margin 303 is assumed to be uniformly formed on the right and left area image 302. object_with_margin_width is 380, object_with_margin_height is 180. object_width is 360, object_height is 180. width_margin_assignment is a 1 (deletion only. left margin right margin).
[0085]
　C of Figure 11 represents a packed frame311. The right image region (inner region) 312, a margin 313 is formed. rect_width is 1920, rect_height is 1440. rect_with_margin_width is 2000, rect_with_margin_height is 1440. rect_left, rect_top and rect_with_margin_left, rect_with_margin_top are both 0,0.
[0086]
　The value of each field are summarized in Figure 12. projection_format 0 (Equirectangular projection), Packing_flag is 1, margin_flag (including areas of there margin) 1, num_regions is 6, FOV_flag is zero (the sphere). center_yaw, center_pitch, object_width, object_height are respectively 0,0,360,180. In addition, rect_width, rect_height, rect_left, rect_top has become, respectively 1920,1440,0,0.
[0087]
　region_margin_type 1 (margin there area (no corner margin)), object_with_margin_width, object_with_margin_height are respectively 380,180. width_margin_assignment 1 (deleted only. left margin right margin), Height_margin_assignment is 0 (equally allocated to both sides (or without margin)). rect_with_margin_width region including the margin of right face, rect_with_margin_height, rect_with_margin_left.rect_with_margin_top are respectively 2000,1440,0,0.
[0088]
　Thus, the width of the region including the margin, as object_with_margin_width in projection structure, the value when the margin is allocated equally is distributed. Since the margin of the area is a type 1, region_margin_type = 1 is delivered. Further, the margin area is present only right in the width direction, since it is no in the height direction, width_margin_assignment = 1, height_margin_assignment = 0 is delivered.
[0089]
　Next, referring to FIG. 13, a description will be given of modifications of RegionMappingBox in FIG. Figure 13 is a diagram showing a configuration example of a region mapping box.
[0090]
　In the example of FIG. 13 is described in the example of FIG. 6, margin allocation information (width_margin_assignment, height_margin_assignment) margin there region information in the projection structure is omitted. Field semantics other than the omitted fields are the same as in FIG. That is, in the example of FIG. 13, the margin there region information in projection structure is distributed regarded as having a whole circumference equal margin. In this case, the maximum margin width that is applied to the target area is set as a uniform margin width. Or up and down, it may be set as a margin width to be uniform in the lateral direction, respectively. Margin allocation region, region information in packed frame (rect_with_margin_width, rect_with_margin_height, rect_with_margin_left, rect_with_margin_top) delivered by.
[0091]
　In addition the example of FIG. 4 and FIG. 5, for example the margin exists in the entire periphery region, and the region right, even if the vertically nonuniform assignment can be delivered by RegionMappingBox.
[0092]
　Example of FIG. 14 is also a modification of the RegionMappingBox in FIG. Figure 14 is a diagram showing a configuration example of a region mapping box. Example of FIG. 14 is an example that does not deliver margin there area information in projection structure. That, Object_width_margin_width in the example of FIG. 6, object_with_margin_height, width_margin_assignment, height_margin_assignment is omitted. Field semantics other than the omitted fields are the same as in FIG.
[0093]
　Still referring to FIG. 15, a description will be given of modifications of RegionMappingBox in FIG. Figure 15 is a diagram showing a configuration example of a region mapping box. In this example, both the region_margin_type = 1, that is, when the corners of the margin there (if (region_margin_type = 1)), projection structure (object_width_margin_width, object_with_margin_height, width_margin_assignment, height_margin_assignment) and packed frame (rect_with_margin_width, rect_with_margin_height, rect_with_margin_left, rect_with_margin_top) in the margin there area is delivered. The other area there is margin only in the packed frame in the case of to be delivered. Other semantics of each field are the same as in FIG.
[0094]
　Next, an example representing a two-dimensional coordinate system on the area information in Projected frame, will be described with reference to FIGS. 16 and 17. Figure 16 is a diagram showing a configuration example of a region mapping box, FIG. 17 is a diagram illustrating the fields of the region mapping box.
[0095]
　In the example of FIG. 16, the margin without the Projected frame and packed frame, information of the margin there area is shown by a two-dimensional coordinate system by the pixel position and the area size of the upper left of each area. That, Center_yaw in FIG. 6, instead Center_pitch, object_width, the object_height, pf_region_width.pf_region_height, pf_region_left, pf_region_top is described. Further, object_with_margin_width, object_with_margin_height, width_margin_assignment, instead height_margin_assignment, pf_region_with_margin_width, pf_region_with_margin_height, pf_region_with_margin_left, pf_region_with_margin_top is described.
[0096]
　17, the field of FIG. 16 are described collectively. It will be described below fields not included in FIGS. 7 and 8. pf_region_width and pf_region_height each represent the width and height of the margin without region (not including the margin) in projected frame, pf_region_left, pf_region_top each represent x and y coordinates of the margin region without the Projected frame.
[0097]
　pf_region_with_margin_width, pf_region_with_margin_height respectively represent the width and height of the area including the margin of Projected frame. pf_region_with_margin_left, pf_region_with_margin_top represent x and y coordinates of the area including the margin of Projected frame.
[0098]
　Next, with reference to RegionMappingBox, for example to deliver a margin of cube mapping projection, it is described with reference to FIGS. 18 and 19. Figure 18 is a diagram illustrating a margin of cube matching projection, FIG. 19 is a diagram for explaining the field during cube matching projection.
[0099]
　The A in FIG. 18, projected frame401 is shown. The middle row, left from the left in order, front,. Right, area image of each side of the back are disposed, the area image top surface is disposed on the front, a region image of the bottom surface under the front surface It is located. A line extended to the upper side of the top surface to the left in the figure, the intersection of the line extended from the left side of the left side on the figure, are the origin of the xy coordinates.
[0100]
　Each surface of the area image is composed of 400 × 400 pixels. left side, the right side and upper and lower back surface, margin widths of 40 pixels are formed. Further, even right back side, a margin with a width of 40 pixels is formed. This margin has a margin of type 2. Therefore, when focusing on back surface, pf_region_left, pf_region_top becomes respectively 1200,400. pf_region_width, pf_region_height are both the 400.
[0101]
　For margin, pf_region_margin_left, pf_region_margin_top is 1200,360. In addition, pf_region_with_margin_width, pf_region_with_margin_height becomes 440,480.
[0102]
　In B of FIG. 18, packed frame411 generated from Projected Frame401 A in FIG. 18 is shown. Resolution area images of the front (front) is directly used as (400 × 400), the resolution of other regions image width and height are reduced to respectively 1/2 (200 × 200). Because it is reduced to the width of the margin 1/2 (20 pixels), the left side (left), right side (. Right), back to the upper and lower (back), the margin with a width of each of 20 pixels It is located. Further, even right back side, margin 20 pixels width is arranged. This margin has a margin of type 2.
[0103]
　The left upper area images of the front (front), area image of the left side surface having a margin (left) is an area image of the right side surface having a margin on the right side (. Right) is arranged respectively. The right area images of the front (front), in order from the top, the region image of the top surface (top) and bottom (bottom) is arranged. On the region image of the top surface (top) (right area images of the right side surface (right)), area image of the rear (back) surface is disposed with a margin.
[0104]
　19, information of Projected Frame401 and packed Frame411 by cube mapping projection shown in FIG. 18 is described. projection_format is 1 (cube mapping projection), packing_flag is 1 (region-wise packing), margin_flag is (there are areas of there margin) 1, num_regions has a 6. back surface of the pf_region_width, has become pf_region_height, pf_region_left, pf_region_top and each 400,400,1200,400. Further, Rect_width the back surface, rect_height, rect_left, rect_top has a respective 200,200,400,20.
[0105]
　region_margin_type 2 (margin there in the area (no corner of the margin)), of the back surface pf_region_with_margin_width, pf_region_with_margin_height, pf_region_with_margin_left, pf_region_with_margin_top has become, respectively 400,480,1200,360. back surface of the rect_with_margin_width, rect_with_margin_height, rect_with_margin_left.rect_with_margin_top has become, respectively 220,240,400,0.
[0106]
　Thus, in packed frame, as it is the resolution of the front surface, the resolution of the other surfaces is in the half horizontally, vertically. Margin of the back surface, since the corner is no type 2, region_margin_type = 2 is delivered. Margin arrangement of each region, Projected frame, packed frame both regions of width, height, top, is explicitly expressed in left.
[0107]
　Chroma subsampling is 4: 2: 2 and 4: 2: 0, in consideration of the color difference signals thinned, by a number of pixels multiple of the margin width 2, the encoding process, to facilitate the region extraction processing can.
[0108]
　In addition, by setting the number of pixels multiple of the margin width 8 or 16, can increase the affinity between the block size at the time of encoding, the encoding processing, region extraction processing is facilitated.
[0109]
　This technology, cube mapping projection and Equirectangular projection not only, other projection type (e.g. Truncated Square Pyramid, cylinder, etc.) is also applicable to.
[0110]
　In the above, although an example only margin of the rectangular area, other shapes (e.g., triangular or trapezoidal, circular, etc.) similar method also can be applied. 20 to refer to FIG. 22, a description will be given margin triangular areas.
[0111]
　20 optimum FIG 22 is a diagram showing an example of a margin arranged. In A of FIG. 20 is region image 431 is a triangular shape, margin 431b are formed on the outside of the right side in the drawing side of the area inside 431a. In this case, as shown in B of FIG. 20, combinations of regions 442a and region 443a of the other triangle two sides of the region 441a of the triangle corresponding to the region 431a, the region by molding so that the whole is rectangular to form an image 441.
[0112]
　The area image 441 is a rectangular area having the same as the length of the side and bottom of the area image 431 of the triangle, the sides of the same length as the height. And on the right side of one side of the rectangular area image 441, the margin 441b are arranged. This rectangular region image 441 is delivered in place of the region image 431 of the triangle.
[0113]
　In A of FIG. 21 is area image 451 in the shape of a triangle, and the margin 451b is formed on the outside of the bottom of the inner region 451a. In this case, as shown in B of FIG. 21, combinations of regions 462a and region 463a of the other triangle two sides of the region 461a of the triangle corresponding to the region 451a, to form an area image 461 overall is rectangular.
[0114]
　The area image 461 is a rectangular area having the same as the length of the side and bottom of the area image 451 of the triangle, the sides of the same length as the height. And the bottom of the rectangular area images 461, margin 461b are arranged. This rectangular region image 461 is delivered in place of the region image 451 of the triangle.
[0115]
　In A of FIG. 22 is region image 471 is a triangular shape, margin 471b are formed on the outside of the right side and bottom of the inner region 471a. In this case, as shown in B of FIG. 22, combinations of regions 482a and region 483a of the other triangle two sides of the region 481a of the triangle corresponding to the region 471a, to form an area image 481 overall is rectangular.
[0116]
　The area image 481 is a rectangular area having the same as the length of the side and bottom of the area image 471 of the triangle, the sides of the same length as the height. Then the right side and bottom of the rectangular area images 481, margin 481b are arranged. This rectangular region image 481 is delivered in place of the region image 471 of the triangle.
[0117]
　In the above example has been described of ISBMFF, it can be distributed using the MPEG-DASH (ISO / IEC 23009-1). Next, with reference to FIG. 23, a description will be given of application examples of the DASH. Figure 23 is a diagram showing an example of MPD file to which the DASH.
[0118]
　In MPD file in FIG. 23, segment file of the image stream time range corresponding to a period elements, respectively, are grouped into one group, period element includes two adaptation set element (AdaptationSet).
[0119]
　In each AdaptationSet, schemeIdUri = "urn: mpeg: dash: vr: ProjectionType" by value = "cube" of EssentialProperty of omnidirectional image is shown to be due to the cube projection mapping. Further, schemeIdUri = the SupplementalProperty of "urn: mpeg: dash:: vr margin", are delivering whether including margin regions of the celestial sphere image (value is 0 or 1 or). value in this SupplementalProperty has the same semantics as margin_flag already described.
[0120]
　DASH client, for example, themselves when performing processing using the margin area of ​​the celestial sphere image, the margin there of AdaptationSet, select Representation, acquired, can be reproduced. When the margin is not required in the reproduction process, AdaptationSet without margin, the Reresentation selected, acquired, can be reproduced.
[0121]
　Incidentally, DASH client that does not support schemeIdUri of EssentialProperty, it is necessary to ignore AdaptationSet this Property is described (or in some cases, such Representation). Further, DASH client that does not support schemeIdUri of SupplementalProperty, ignoring this Property value, its AdaptationSet (sometimes such Representation) may be utilized.
[0122]
　Of course, it is possible to further describe the other information in the MPD file.
[0123]
　
　Next, a system for distributing omnidirectional image including an area image having a margin as described above will be described with reference to FIG. 24. Figure 24 is a block diagram showing a configuration example of a distribution system.
[0124]
　Distribution system 610 of FIG. 24, the photographing apparatus 611, generator 612, the distribution server 613, and the reproducing apparatus 614 and the head mounted display 615,. Distribution system 610 generates a celestial sphere image from the images picked up by the imaging apparatus 611, and displays the display image of the field of view of the viewer by using the omnidirectional image.
[0125]
　Specifically, the photographing apparatus 611 of a distribution system 610 is composed of six cameras 611A-1 to 611A-6 and a microphone 611B. In the following description, when it is not necessary to distinguish the camera 611A-1 to 611A-6, referred to as a camera 611A them together.
[0126]
　Each camera 611A is photographed moving image, a microphone 611B obtains ambient sounds. Delivery system 610, the audio acquired by the captured image and the microphone 611B are six directions of a moving image taken by the cameras 611A, and supplies to the generating device 612 as a moving image content. The number of cameras capturing device 611 is provided, if a plurality, may be other than six.
[0127]
　Generator 612 by a method using equirectangular generates omnidirectional image from the captured image supplied from the imaging apparatus 611, and encoded in one or more bit rate, Equirectangular of each bit rate to produce a stream. Further, generator 612, by the cube mapping, generates a celestial sphere image from the captured image, is encoded with one or more bit rate to produce a cube stream for each bit rate. Further, generator 612 encodes the voice supplied from the imaging apparatus 611, generates an audio stream.
[0128]
　Generator 612 Equirectangular stream for each bit rate, for each bit rate cubes stream, and the audio stream and ISOBMFF filing. Generator 612 uploads the results generated ISOBMFF file to the distribution server 613.
[0129]
　Here, although the bit rate of equirectangular stream and cubes stream to be a 1 or more, other than the bit rate conditions (e.g., the size of the image, etc.) may be located in one or more.
[0130]
　Further, generator 612 generates the MPD file for managing a segment file of the moving image content is uploaded to the distribution server 613. A segment is for video streams, and filed in hours for about 10 seconds from the seconds audio stream. For example, ISOBMFF including RegionMappingBox is delivered as a segment file.
[0131]
　For example the distribution server 613 for distributing with MEPG-DASH (ISO / IEC 23009-1) stores the uploaded segment files and MPD file from generator 612. The distribution server 613, in response to a request from the playback apparatus 614 as a client, and transmits to the playback apparatus 614 segments file containing.
[0132]
　Reproducing apparatus 614 requests the ISOBMFF file to the distribution server 613, receives a ISOBMFF file transmitted in response to the request. The playback apparatus 614, based on ISOBMFF file, requests the segment files celestial sphere image generated by the generation method of the celestial sphere image corresponding to the mapping can be performed by the reproducing apparatus 614, the request receiving a segment file transmitted in accordance with the. Reproducing apparatus 614 decodes the cube streams included in the received segment file (or a equirectangular stream). Reproducing apparatus 614, by mapping the omnidirectional image obtained as the result of decoding to the 3D model, to generate a 3D model image.
[0133]
　The reproducing apparatus 614 has a built-in camera 614A, photographs the markers 615A attached to the head mounted display 615. The reproduction apparatus 614 based on the captured images of the markers 615A, detects the viewing position in the coordinate system of the 3D model. Further, the reproducing apparatus 614, the detection result of the gyro sensor 615B of the head mounted display 615, and receives from the head-mounted display 615. Reproducing apparatus 614, based on the detection result of the gyro sensor 615B, to determine the viewing direction of the viewer in the coordinate system of the 3D model. Reproducing apparatus 614, based on the viewing position and viewing direction, to determine the viewer's field of view that is located inside the 3D model.
[0134]
　Reproducing apparatus 614, as the focus of the viewing position, by perspective projection of 3D model image the field of view of the viewer, generates an image of the visual field range of the viewer as a display image. Reproducing apparatus 614 supplies the display image on the head mounted display 615.
[0135]
　Head-mounted display 615 is mounted on the head of the viewer, displaying the display image supplied from the reproduction apparatus 614. The head-mounted display 615, the marker 615A is attached to be photographed by the camera 614A. Therefore, the viewer, a head-mounted display 615 in a state of mounting on the head, it is possible to specify the viewing position by moving. Further, the head mounted display 615, a gyro sensor 615B is built, the detection result of the angular velocity by the gyro sensor 615B is transmitted to the playback device 614. Therefore, the viewer, by rotating the head wearing the head-mounted display 615, it is possible to specify the line-of-sight direction.
[0136]
　
　FIG. 25 is a block diagram showing a configuration example of a generator. Generator 612, the stitching processing unit 631, the mapping processing unit 632, region-wise packing (region -wise packing) processing unit 633, an encoder 634, an audio processing unit 635, an encoder 636, the file generation unit 637 and upload unit 638, It constituted by.
[0137]
　Stitching processing unit 631, for each frame, and the same color and brightness of the six directions of the captured image supplied from the camera 611A in FIG. 24, it performs the stitching process of connecting to remove overlap. Stitching processing unit 631 supplies the captured image frame-by-frame after stitching process to the mapping processor 632.
[0138]
　Mapping processor 632, the cube mapping in this example, generates an omnidirectional image from the captured image supplied from the stitching processing unit 631. Specifically, the mapping processing unit 632 are mapped to a cube photographed image after the stitching process as texture, to produce an image of a developed view of the cube as omnidirectional image. Mapping processing unit 632 supplies the omnidirectional image region-wise packing processor 633. Incidentally, stitching processing unit 631 and the mapping processing unit 632 may be integrated.
[0139]
　region-wise packing processing unit 633 performs a region-wise packing process. That is, the packing (packing) to place Projected frame by changing the position and size for each area on a two-dimensional plane to produce a packed frame. region-wise packing processor 633 also generates RegionMappingBox containing Margin_flag, a Region_margin_type.
[0140]
　The encoder 634 encodes the omnidirectional image supplied from the region-wise packing processor 633 with one or more bit rate to generate a cube stream. The encoder 634 supplies the cube stream for each bit rate file generation unit 637.
[0141]
　Audio processing unit 635 obtains a voice supplied from the microphone 611B in FIG. 24, and supplies the encoder 636. Encoder 636 encodes the sound supplied from the audio processing unit 635, to generate an audio stream. The encoder 636 supplies the audio stream to the file generation unit 637.
[0142]
　File generating unit 637, for each bit rate cubes stream, and the audio stream and filed in segments. File generating unit 637 supplies the result generated segment file to the upload unit 638. File generating unit 637 also generates a ISOBMFF file and supplies it to the upload section 638.
[0143]
　Uploader 638, a segment files and ISOBMFF file supplied from the file generation unit 637, is uploaded to the distribution server 613 in FIG. 24.
[0144]
　Next, ISOBMFF generation processing will be described with reference to FIG. 26. Figure 26 is a flowchart for explaining the ISOBMFF generation process. Primarily, the case of processing an image as an example.
[0145]
　Stitching processing unit 631 in step S1, for each frame, and the same color and brightness of the six directions of the captured image supplied from the camera 611A in FIG. 24, performs the stitching process of connecting to remove overlap. Stitching processing unit 631 supplies the captured image frame-by-frame after stitching process to the mapping processor 632.
[0146]
　Mapping processor 632, for example by cube mapping to generate an omnidirectional image from the captured image supplied from the stitching processing unit 631. That is, the mapping processing unit 632 are mapped to a cube photographed image after the stitching process as texture, to produce an image of a developed view of the cube as omnidirectional image. At this time, the mapping processor 632 generates a margin area of ​​the celestial sphere image. Mapping processing unit 632 supplies the omnidirectional image region-wise packing processor 633.
[0147]
　region-wise packing processor 633 in step S2 determines whether omnidirectional image includes a region having a margin. If omnidirectional image includes a region having a margin, region-wise packing unit 633 in step S3 is set to margin_flag = 1 (there are areas of there margin), sets the region_margin_type according to the type of the margin of each area to. That is, as described above, the value 0 in the area without margin, the value 1 in the region having a margin corners there, the value 2 in the region where the corners have a free margin is set, respectively. The region-wise packing unit 633 generates a RegionMappingBox containing them.
[0148]
　On the other hand, if the full spherical image in step S2 is determined not to include a region having a margin, region-wise packing unit 633 in step S4, the margin_flag = 0 (region without margin only), it including generating a RegionMappingBox.
[0149]
　After the processing of step S3 and step S4 is performed, the encoder 634 in step S5 encodes the omnidirectional image. Audio data processed by the audio processing unit 635 is encoded by the encoder 636 is supplied to the file creating unit 637.
[0150]
　File creating unit 637 in step S6, it generates a ISOBMFF file. Files that are created by the file creation unit 637 is uploaded from the upload part 638 to the distribution server 613.
[0151]
　
　Next, an example in which processed image, a configuration example of a playback apparatus 614. Figure 27 is a block diagram showing a configuration example of a playback apparatus. Reproducing apparatus 614 includes a file acquiring unit 701, the stream extraction unit 702, a decoder 703, Projected frame generation unit 704, the mapping processing unit 705, rendering unit 706, receiving section 707, line-of-sight detecting unit 708, and the camera 614A .
[0152]
　File acquisition section 701 acquires the file to be played back from the distribution server 613 in FIG. 24. Stream extraction unit 702, the obtained file by file acquisition unit 701 extracts the video stream. The decoder 703 decodes the video stream extracted by the stream extraction unit 702. Projected frame generation unit 704 generates a Projected frame from the decoded image data by the decoder 703.
[0153]
　Mapping processor 705 maps the omnidirectional image supplied from Projected frame generation unit 704 as a texture to each of the six sides of a cube 12.
[0154]
　Rendering unit 706, the 3D model image supplied from the mapping unit 705, as the focus of the viewing position supplied from the sight line detecting unit 708, by perspective projection in the visual field range of the viewer, the viewing range of the viewer image is generated as a display image. Rendering unit 706 supplies the display image on the head mounted display 615.
[0155]
　Receiving unit 707, a detection result of the gyro sensor 615B in FIG. 24, it receives from the head-mounted display 615, and supplies the line-of-sight detecting unit 708.
[0156]
　Visual axis detection unit 708, based on the detection result of the gyro sensor 615B supplied from the receiving unit 707, determines the viewing direction of the viewer in the coordinate system of the 3D model. Further, the sight line detection unit 708 acquires the captured image of the markers 615A from camera 614A, based on the captured image to detect a viewing position in the coordinate system of the 3D model. Visual axis detection unit 708, based on the viewing position and the viewing direction in the coordinate system of the 3D model to determine the field of view of the viewer in the coordinate system of the 3D model. Visual axis detection unit 708 supplies the drawing unit 706 a viewing position and viewing range of the viewer.
[0157]
　Next, referring to FIG. 28, the operation of the reproducing apparatus 614. Figure 28 is a flowchart for explaining the ISOBMFF regeneration process. Primarily, the case of processing a case and a region image of the cube projection examples.
[0158]
　File acquisition section 701 in step S21 obtains the file to be played back from the distribution server 613 in FIG. 24. Further file acquisition section 701, Margin_flag determines what is. (When including the region of there omnidirectional image margin) Margin_flag be a value 1, the file obtaining unit 701 in step S22, it determines whether playback client uses the margin area. That reproducing apparatus 614 determines whether itself has the ability to use the margin.
[0159]
　If its own use the margin, in step S23, the stream extraction unit 702, the obtained file by file acquisition unit 701 extracts the video stream. The decoder 703 decodes the extracted stream, and supplies the Projected frame generation unit 704.
[0160]
　Projected frame generation unit 704 in step S24, based on the information of RegionMappingBox, Get the margin there region, it generates a Projected frame. In step S25, the omnidirectional image, processing for rendering is executed in the processing of the margin for example, blending, and the like. That mapping processing unit 705, an omnidirectional image is mapped as a texture to each of the six sides of a cube 12. The 3D model image supplied drawing unit 706 from the mapping processing unit 705, as the focus of the viewing position supplied from the sight line detecting unit 708, by perspective projection in the visual field range of the viewer, the viewer's field of view of the image generated as a display image. Rendering unit 706 supplies the display image on the head mounted display 615.
[0161]
　Play client does not use the margin of the region in step S22, i.e., if the reproducing apparatus 614 itself is determined not to have the ability to use the margin, the process proceeds to step S26. In step S26, the stream extraction unit 702, the obtained file by file acquisition unit 701 extracts the video stream. The decoder 703 decodes the extracted stream, and supplies the Projected frame generation unit 704.
[0162]
　Projected frame generation unit 704 in step S27, based on the information of RegionMappingBox, obtains a margin region without. In step S28, the process of rendering the omnidirectional image is performed. That is, the mapping processing unit 705, an omnidirectional image is mapped as a texture to each of the six sides of a cube 12. The 3D model image supplied drawing unit 706 from the mapping processing unit 705, as the focus of the viewing position supplied from the sight line detecting unit 708, by perspective projection in the visual field range of the viewer, the viewer's field of view of the image generated as a display image. Rendering unit 706 supplies the display image on the head mounted display 615.
[0163]
　If margin_flag is determined to be a value 0 in step S21 (if only area without margin), in step S29, the stream extraction unit 702, the obtained file by file acquisition section 701, extracts a video stream to. The decoder 703 decodes the extracted stream, and supplies the Projected frame generation unit 704.
[0164]
　Projected frame generation unit 704 in step S30 obtains the area image based on information of RegionMappingBox.
[0165]
　Process of rendering omnidirectional image is performed in step S31. That is, the mapping processing unit 705, an omnidirectional image is mapped as a texture to each of the six sides of a cube 12. Rendering unit 706, the 3D model image supplied from the mapping unit 705, as the focus of the viewing position supplied from the sight line detecting unit 708, by perspective projection in the visual field range of the viewer, the viewing range of the viewer image is generated as a display image. Rendering unit 706 supplies the display image on the head mounted display 615.
[0166]
　Thus, in the reproduction process, the reproduction by margin_flag client can determine whether it has an area omnidirectional image has a margin. If omnidirectional image has a region having a margin, for example, play client, or rendered using a region having a margin, depending on whether the rendered using area margin is not the appropriate area information obtained by rendering processing can be performed.
[0167]
　Incidentally, ISOBMFF is composed of a plurality of video tracks, there is a case where track celestial sphere image including a track celestial sphere image including only the area margin is not a region having a margin are mixed. In such cases, the clients that do not use the margin, a track celestial sphere image having only a region without margin, the client using a margin, a track celestial sphere image including the region of there margins, respectively selected then, it can be reproduced.
[0168]
　In the above has been described primarily processed image for audio information is also delivered with the image information.
[0169]
　According to this technique, it is possible client using the margin, easily obtain the area including the margin, clients that do not use a margin obtains easily region not including a margin.
[0170]
　Furthermore, by omnidirectional image delivers whether it has a region with a margin as a flag, the clients that do not use the margin, can easily select the omnidirectional image without margins. The client utilizing margin, the omnidirectional image there margin, can be readily selected.
[0171]
　Incidentally present technology, within the scope not departing from the essence, various modifications may be present.
[0172]
　
　FIG. 38 is a block diagram showing a configuration example of hardware of a computer that executes the series of processes described above.
[0173]
　In the computer 900, CPU (Central Processing Unit) 901, ROM (Read Only Memory) 902, RAM (Random Access Memory) 903 are connected to each other via a bus 904.
[0174]
　The bus 904 is further input and output interface 905 is connected. Output interface 905, an input unit 906, output unit 907, storage unit 908, a communication unit 909, and a drive 910 are connected.
[0175]
　Input unit 906 includes a keyboard, a mouse, and a microphone. The output unit 907 includes a display and a speaker. Storage unit 908, a hard disk and a nonvolatile memory. The communication unit 909 includes a network interface. Drive 910 drives a magnetic disk, an optical disk, a magneto-optical disk, or a removable medium 911 such as a semiconductor memory.
[0176]
　In configured computer 900 as described above, CPU 901 is, for example, a program stored in the storage unit 908, output interface 905 and the bus 904 and executes the loaded into RAM 903, the above-described a series of processing is performed.
[0177]
　Program computer 900 (CPU 901) is executed, for example, can be provided by being recorded on the removable medium 911 as a package medium or the like. Further, the program may be provided via a local area network, the Internet, or digital satellite broadcasting, a wired or wireless transmission medium.
[0178]
　In the computer 900, programs, by loading the removable medium 911 into the drive 910, can be installed via the input output interface 905, the storage unit 908. The program via a wired or wireless transmission medium and received by the communication unit 909, can be installed in the storage unit 908. Alternatively, the program may be in the ROM902 and the storage unit 908 installed in advance.
[0179]
　The program may be a program in which processes are performed in time series in the order described herein, the necessary timing such as when parallel or call was made by the computer 900 to execute in the process it may be a program to be carried out.
[0180]
　Further, in this specification, a system includes a plurality of components (devices, modules (components) or the like) means a set of, it does not matter whether or not there is in the same housing all components. Therefore, housed in a separate enclosure, a plurality of devices connected via a network, and one device in which a plurality of modules within a single casing is housed, both of which the system .
[0181]
　Moreover, effects described herein are not intended to be limited to a merely illustrative, there may be other effects.
[0182]
　Further, embodiments of the present technology is not limited to the embodiments described above, but various modifications are possible without departing from the scope of the present disclosure.
[0183]
　
　The present technology may also be configured as follows.
(1)
　omnidirectional image, the the area of the celestial sphere image, comprising an identification information generation unit for generating a margin identification information for identifying comprises a region having the generated margin
　generator.
(2)
　further comprising a margin generator for generating a margin area of the celestial sphere image
　generation apparatus according to (1).
(3)
　the margin is formed outside of the region
　generating apparatus according to (1) or (2).
(4)
　area information of an area having the margin is represented by a spherical coordinate system or two-dimensional coordinate system
　generating apparatus according to (2) or (3).
(5)
　the area information is expressed as information of Projected frame or packed frame
　generating apparatus according to any one of (2) to (4).
(6)
　the area information of an area having the margin of the Projected frame, the area is described in the case with the margin
　generation device according to (5).
(7)
　the area information of an area having the margin of the packed frame, the area is described when it is region-wise packing
　generating apparatus according to (5) or (6).
(8)
　the area information of an area having the margin of the Projected frame, the Projected frame the margin includes area width and height of, and assigned to identify the allocation of the margin of the width and the height direction including the identification information
　generating device according to any one of (5) to (7).
(9)
　the area information of an area having the margin of the packed frame, the Projected frame the margin includes area width and height of, and upper left coordinates of the area
　above (5) to (8) generator according to any one of.
(10)
　the area information of an area having the margin of the packed frame, the packed frame is described if they are region-wise packing
　generating apparatus according to any one of (5) to (9) .
(11)
　The allocation identification information is omitted
　generating device according to (8).
(12)
　the area information of an area having the margin of the Projected frame is omitted, only the region information of a region having the margin of the packed frame is described
　any one of (5) to (11) generating apparatus according to.
(13)
　said margin, there is a type not to have the type having a non-formation portion where the margin in the corner is not formed
　generator according to any one of (1) to (12).
(14)
　the identification information generating unit further generates the type identification information identifying the type
　generating apparatus according to (13).
(15)
　the projected the region information of a region having the margin frame, the type identification information is written when the a type having no unformed portion
　generating apparatus according to (14).
(16)
　said region is triangular, if the margin along the sides are formed, by molding the triangle rectangle, placing the margin to the side of the rectangle corresponding to the triangle
　the ( 1) to generator according to any one of (15).
(17)
　the margin identification information is described in box under Scheme Information Box of ISOBMFF
　generating apparatus according to any one of (1) to (16).
(18)
　the margin identification information is described in MPD file MPEG-DASH
　generating apparatus according to any one of (1) to (17).
(19)
　generating device,
　　omnidirectional image, the the area of the celestial sphere image, including identification information generation step of generating a margin identification information for identifying comprises a region having the generated margin
　identification information generating Method.
(20)
　an obtaining unit omnidirectional image to obtain a margin identification information for identifying comprises a region having a margin,
　and a generator for generating a Projected frame based on the acquired margin identification information,
　the Projected frame a rendering unit for rendering the
　reproduction apparatus comprising a.
(21)
　reproducing apparatus,
　　an acquisition step of omnidirectional images to obtain a margin identification information for identifying comprises a region having a margin,
　　a generation step of generating a Projected frame based on the acquired margin identification information ,
　　a rendering step of rendering the Projected frame
　image generating method comprising.
DESCRIPTION OF SYMBOLS
[0184]
　610 distribution system, 611 an imaging device, 612 generator, 613 distribution server 614 playback apparatus 615 a head-mounted display, 631 stitching processing unit, 632 the mapping unit, 633 region-wise packing processor, 634 an encoder, 637 file generation Department, 638 uploader, 701 file acquisition unit, 704 packing frame generation unit, 705 the mapping processing unit 706 drawing unit

The scope of the claims
[Requested item 1]
　Omnidirectional image, the the area of the celestial sphere image, comprising an identification information generation unit for generating a margin identification information for identifying comprises a region having the generated margin
　generator.
[Requested item 2]
　Further comprising a margin generator for generating a margin area of the celestial sphere image
　generation apparatus according to claim 1.
[Requested item 3]
　The margin is formed outside of the region
　generating apparatus according to claim 2.
[Requested item 4]
　Region information of a region having the margin is represented by a spherical coordinate system or two-dimensional coordinate system
　generating apparatus according to claim 3.
[Requested item 5]
　The area information is expressed as information of Projected frame or packed frame
　generating apparatus according to claim 4.
[Requested item 6]
　The area information of an area having the margin of the Projected frame, the area is described in the case with the margin
　generating device according to claim 5.
[Requested item 7]
　The area information of an area having the margin of the packed frame, the area is described when it is region-wise packing
　generating apparatus according to claim 6.
[Requested item 8]
　The area information of an area having the margin of the Projected frame, the Projected frame the margin includes area width and height, and the allocation identification information for identifying the allocation of the margin of the width and the height direction including
　generating apparatus according to claim 7.
[Requested item 9]
　The area information of an area having the margin of the packed frame, the Projected frame the margin includes area width and height of, and upper left coordinates of the area
　generating apparatus according to claim 8.
[Requested item 10]
　The area information of an area having the margin of the packed frame, the packed frame is described if they are region-wise packing
　generating apparatus according to claim 9.
[Requested item 11]
　The allocation identification information is omitted
　generator according to claim 8.
[Requested item 12]
　The area information of an area having the margin of the Projected frame is omitted, only the region information of a region having the margin of the packed frame is described
　generator according to claim 9.
[Requested item 13]
　The margin, there are a type as not having the type having a non-formation portion where the margin in the corner is not formed
　generator according to claim 8.
[Requested item 14]
　The identification information generating unit further generates the type identification information identifying the type
　generating apparatus according to claim 13.
[Requested item 15]
　The area information of an area having the margin of the Projected frame, the type identification information is written when the a type having no unformed portions
　generator according to claim 14.
[Requested item 16]
　Is the area a triangle, when the margin along the sides are formed, by molding the triangle rectangle, placing the margin to the side of the rectangle corresponding to the triangular
　claim 5 generating device.
[Requested item 17]
　The margin identification information is described in box under Scheme Information Box of ISOBMFF
　generating apparatus according to claim 3.
[Requested item 18]
　The margin identification information is described in MPD file MPEG-DASH
　generating apparatus according to claim 3.
[Requested item 19]
　Generator is
　　omnidirectional image, the the area of the celestial sphere image, including identification information generation step of generating a margin identification information for identifying comprises a region having the generated margin
　identification information generation method.
[Requested item 20]
　An acquisition unit omnidirectional image to obtain a margin identification information for identifying comprises a region having a margin,
　and a generator for generating a Projected frame based on the acquired margin identification information,
　rendering the Projected frame and the rendering unit
　playback device comprising a.
[Requested item 21]
　Reproducing apparatus,
　　an acquisition step of omnidirectional images to obtain a margin identification information for identifying comprises a region having a margin,
　　a generation step of generating a Projected frame based on the acquired margin identification information,
　　the Projected and rendering step of rendering a frame
　image generating method comprising.
Corrected claims (Convention Article 19)
[April 5, 2018 (05.04.2018) The International Bureau acceptance]
[1]
　Omnidirectional image, the the area of the celestial sphere image, comprising an identification information generation unit for generating a margin identification information for identifying comprises a region having the generated margin
　generator.
[2]
　Further comprising a margin generator for generating a margin area of the celestial sphere image
　generation apparatus according to claim 1.
[3]
　The margin is formed outside of the region
　generating apparatus according to claim 2.
[4]
　Region information of a region having the margin is represented by a spherical coordinate system or two-dimensional coordinate system
　generating apparatus according to claim 3.
[5]
　The area information is expressed as information of Projected frame or packed frame
　generating apparatus according to claim 4.
[6]
　The area information of an area having the margin of the Projected frame, the area is described in the case with the margin
　generating device according to claim 5.
[7]
　The area information of an area having the margin of the packed frame, the area is described when it is region-wise packing
　generating apparatus according to claim 6.
[8]
　The area information of an area having the margin of the Projected frame, the Projected frame the margin includes area width and height, and the allocation identification information for identifying the allocation of the margin of the width and the height direction including
　generating apparatus according to claim 7.
[9]
[Corrected] the region information of a region having the margin of the packed frame, the packed frame the margin includes area width and height of, and upper left coordinates of the area
　generation according to claim 8 apparatus.
[10]
　The area information of an area having the margin of the packed frame, the packed frame is described if they are region-wise packing
　generating apparatus according to claim 9.
[11]
　The allocation identification information is omitted
　generator according to claim 8.
[12]
　The area information of an area having the margin of the Projected frame is omitted, only the region information of a region having the margin of the packed frame is described
　generator according to claim 9.
[13]
　The margin, there are a type as not having the type having a non-formation portion where the margin in the corner is not formed
　generator according to claim 8.
[14]
[Corrected] The margin, chroma subsampling is 4: 2: 2 or 4: 2: 0 is the pixel number of the width of the second multiple
　generation device according to claim 6 or claim 7 .
[15]
[Corrected] The margin is eight multiples of the number of pixels wide or 16 pixels number of the width of the multiple of,
　generating apparatus according to claim 6 or claim 7.
[16]
　Is the area a triangle, when the margin along the sides are formed, by molding the triangle rectangle, placing the margin to the side of the rectangle corresponding to the triangular
　claim 5 generating device.
[17]
　The margin identification information is described in box under Scheme Information Box of ISOBMFF
　generating apparatus according to claim 3.
[18]
　The margin identification information is described in MPD file MPEG-DASH
　generating apparatus according to claim 3.
[19]
　Generator is
　　omnidirectional image, the the area of the celestial sphere image, including identification information generation step of generating a margin identification information for identifying comprises a region having the generated margin
　identification information generation method.
[20]
　An acquisition unit omnidirectional image to obtain a margin identification information for identifying comprises a region having a margin,
　and a generator for generating a Projected frame based on the acquired margin identification information,
　rendering the Projected frame and the rendering unit
　playback device comprising a.
[21]
　Reproducing apparatus,
　　an acquisition step of omnidirectional images to obtain a margin identification information for identifying comprises a region having a margin,
　　a generation step of generating a Projected frame based on the acquired margin identification information,
　　the Projected and rendering step of rendering a frame
　image generating method comprising.

Instructions under the Convention Article 19 (1)
　Claim 9 is obtained by correcting the mistake.
　Claim 14, which clarifies the margin, claim 15 is obtained by clarifying the margin.
　This technique, for example, is to be able to easily obtain a margin there area and the margin region without.