Title of invention: Information processing apparatus and method
Technical field
[0001]
　The present disclosure relates to an information processing device and method, and more particularly, to an information processing device and method capable of more easily selecting a sub-picture stream.
Background technology
[0002]
　2. Description of the Related Art Conventionally, MPEG-DASH (Moving Picture Experts Group-Dynamic Adaptive Streaming over HTTP) is a standardized standard for adaptive content distribution technology based on HTTP (Hypertext Transfer Protocol) (see Non-Patent Document 1 and Non-Patent Document 2, for example). ).
[0003]
　Also, as this MPEG-DASH file format, there is ISOBMFF (International Organization for Standardization Base Media File Format) which is a file container specification of the international standard video compression technology "MPEG-4 (Moving Picture Experts Group-4)" ( See, for example, Non-Patent Document 3).
[0004]
　By the way, a stereoscopic image, which is an image obtained by projecting an image of 360 degrees in the horizontal direction and 180 degrees in the vertical direction on a stereoscopic structure, such as a so-called spherical image, is mapped into a planar image. It is considered to use MPEG-DASH for distribution of a projection plane image). For example, MPEG-DASH can be applied by mapping the three-dimensional structure image on a single plane and distributing the three-dimensional structure image as a projection plane image in which the three-dimensional structure image is mapped on the plane. At that time, it has been proposed to divide one projection plane image (also referred to as a whole picture) of a celestial sphere image into a plurality of sub-pictures and store them in a plurality of tracks. When identifying the display area of ​​a sub-picture, first construct an entire picture from the sub-picture based on the sub-picture partition information, and then region-wise packing based on the region-wise packing information. It is necessary to perform a process of rearranging the entire picture thus created (for example, see Non-Patent Document 4).
[0005]
Non-Patent Document 1: "Information technology. Dynamic adaptive streaming over HTTP (DASH). Part 1: Media presentation description and segment formats", ISO/IEC23009-1, 2014/05
Non-Patent Document 2: "Information technology. Dynamic adaptive streaming over HTTP (DASH) .Part 1: Media presentation description and segment formats AMENDMENT2: Spatial relationship description, generalized URL parameters and other extensions", ISO/IEC 23009-1:2014/Amd 2:2015, 2015/07
Non-Patent Document 3 : "Information technology-Coding of audio-visual objects-Part 12: ISO base media file format", ISO/IEC 14496-12, 2005-10-01
Non-Patent Document 4: Ye-Kui Wang, Youngkwon Lim, "MPEG# 120 OMAF meeting agenda and minutes", ISO/IEC JTC1/SC29/WG11 MPEG2017/M41823, October 2017, Macau, China
Summary of the invention
Problems to be Solved by the Invention
[0006]
　However, according to the method currently proposed, when sub-picture conversion is performed, the placement information (region-wise packing information of the whole picture before division) in which the size and position are changed for each picture area is the Sub Picture Composition Box. Signaled to the Region Wise Packing Box below. Therefore, when selecting and playing a sub-picture track, the Sub Picture Composition Box is parsed to identify the display area on the projected picture of the sub-picture track, and the region-wise packing information and sub-picture division information are parsed. It is necessary to distinguish between and, and the processing load may increase as compared to the case of selecting and playing a track that is not a sub-picture track.
[0007]
　The present disclosure has been made in view of such a situation, and makes it possible to more easily select a sub-picture stream.
Means for solving the problem
[0008]
　An information processing apparatus according to an aspect of the present technology manages image coded data for each sub-picture obtained by dividing an entire picture into a plurality of sub-pictures and encoding the information, and information regarding a region in the entire picture corresponding to the sub-picture. Is an information processing device including a file generation unit that generates a control file that is used to control distribution of the image coded data and that includes as a piece of information different from the arrangement information for each picture area.
[0009]
　An information processing method according to one aspect of the present technology is to manage image coded data for each sub-picture obtained by dividing an entire picture into a plurality of sub-pictures and encoding the information, and information on a region in the entire picture corresponding to the sub-picture. Is an information processing method for generating a control file used for distribution control of the image coded data, the information including: as information different from the arrangement information for each picture area.
[0010]
　An information processing device according to another aspect of the present technology manages image coded data for each sub-picture in which a whole picture is divided into a plurality of sub-pictures and coded, and relates to a region in the whole picture corresponding to the sub-picture. A file acquisition unit for acquiring a control file used for distribution control of the image coded data, which includes information different from the arrangement information for each picture area, and included in the control file acquired by the file acquisition unit. And an image processing unit that selects a stream of the image coded data based on information regarding the region to be encoded.
[0011]
　An information processing method according to another aspect of the present technology manages image coded data for each sub-picture obtained by dividing an entire picture into a plurality of sub-pictures and encoding the sub-picture, and relates to a region in the entire picture corresponding to the sub-picture. Information is included as information different from the arrangement information for each picture area, and a control file used for distribution control of the image coded data is acquired, based on the information about the area included in the acquired control file, It is an information processing method for selecting a stream of the image coded data.
[0012]
　In an information processing device and method according to an aspect of the present technology, an entire picture is divided into a plurality of sub-pictures, and encoded image data for each sub-picture is managed. A control file used for distribution control of image coded data is generated, which includes information related to areas as information different from the arrangement information for each picture area.
[0013]
　In an information processing device and method according to another aspect of the present technology, the whole picture is managed by dividing the sub-picture into a plurality of sub-pictures and managing the coded image data for each sub-picture. A control file used for controlling distribution of image coded data, which includes information regarding the area in the picture area as information different from the arrangement information for each picture area, is acquired, and based on the information regarding the area included in the acquired control file. Thus, the stream of image coded data is selected.
Effect of the invention
[0014]
　According to the present disclosure, information can be processed. In particular, the sub-picture stream can be selected more easily.
Brief description of the drawings
[0015]
FIG. 1 is a diagram showing an example of a Box hierarchical structure of a sub-picture track of ISOBMFF.
FIG. 2 is a diagram showing an example of a Box hierarchical structure of a track which is not an ISOBMFF sub-picture track.
FIG. 3 is a diagram showing an example of the syntax of a Sub Picture Composition Box.
FIG. 4 is a diagram showing an example of syntax of a Sub Picture Region Box.
FIG. 5 is a diagram showing an example of the semantics of fields defined in the Sub Picture Region Box.
FIG. 6 is a diagram showing an example of syntax of Region Wise Packing Box.
FIG. 7 is a diagram showing an example of syntax of Region Wise Packing Struct.
FIG. 8 is a diagram showing an example of the semantics of fields defined in Region Wise Packing Struct.
FIG. 9 is a diagram showing an example of the syntax of RectRegionPacking.
FIG. 10 is a diagram showing an example of the semantics of fields defined in RectRegionPacking.
FIG. 11 is a block diagram showing a main configuration example of a file generation device.
FIG. 12 is a block diagram showing a main configuration example of a client device.
FIG. 13 is a diagram showing an example of display area information.
FIG. 14 is a flowchart illustrating an example of the flow of upload processing.
FIG. 15 is a flowchart illustrating an example of the flow of content reproduction processing.
FIG. 16 is a diagram showing an example of syntax of a 2D Coverage Information Box.
FIG. 17 is a diagram showing an example of the semantics of fields defined in the 2D Coverage Information Box.
FIG. 18 is a diagram showing an example of display area information.
FIG. 19 is a diagram showing an example of a sub-picture including discontinuous areas.
FIG. 20 is a diagram showing an example of the syntax of a 2D Coverage Information Box.
FIG. 21 is a diagram showing an example of semantics of fields added in this case.
FIG. 22 is a diagram showing an example of syntax of Region Wise Packing Struct.
FIG. 23 is a diagram showing an example of semantics of fields added in this case.
FIG. 24 is a diagram showing an example of the syntax of RectProjectedRegion.
FIG. 25 is a diagram showing an example of the semantics of fields defined in RectProjectedRegion.
FIG. 26 is a diagram showing an example of syntax of Region Wise Packing Struct.
FIG. 27 is a diagram showing an example of syntax of RectRegionPacking.
FIG. 28 is a diagram showing an example of syntax of a Coverage Information Box.
FIG. 29 is a diagram showing an example of the semantics of fields defined in the Coverage Information Box.
FIG. 30 is a diagram showing an example of the syntax of a Spherical offset projection SEI message.
FIG. 31 is a diagram showing an example of semantics of fields defined in a Spherical offset projection SEI message.
FIG. 32 is a diagram showing an example of the syntax of 2D Coverage Information Sample Entry.
FIG. 33 is a diagram showing an example of syntax of a 2D Coverage Information Sample.
FIG. 34 is a diagram showing an example of a Sample Table Box.
FIG. 35 is a diagram showing an example of the syntax of 2D Coverage Information Sample Group Entry.
FIG. 36 is a flowchart illustrating an example of the flow of upload processing.
FIG. 37 is a flowchart illustrating an example of the flow of content reproduction processing.
FIG. 38 is a diagram showing an example of attribute values ​​of a 2D coverage information descriptor.
FIG. 39 is a diagram showing an example of attribute values ​​of a 2D coverage information descriptor.
FIG. 40 is a diagram showing an example of a data type.
FIG. 41 is a diagram showing an example of attribute values ​​of Region Wise Packing descriptor.
FIG. 42 is a diagram showing an example of attribute values ​​of Region Wise Packing descriptor.
FIG. 43 is a diagram showing an example of attribute values ​​of Content coverage descriptor.
FIG. 44 is a diagram continued from FIG. 43 showing an example of attribute values ​​of the Content coverage descriptor.
FIG. 45 is a diagram showing an example of a data type.
FIG. 46 is a diagram continued from FIG. 45, showing an example of the data type.
FIG. 47 is a diagram continued from FIG. 46, showing an example of the data type.
FIG. 48 is a diagram showing an example of sub-picture conversion.
FIG. 49 is a diagram showing an example of sub-picture conversion.
FIG. 50 is a diagram showing an example of sub-picture conversion.
FIG. 51 is a flowchart illustrating an example of the flow of upload processing.
FIG. 52 is a flowchart illustrating an example of the flow of content reproduction processing.
FIG. 53 is a diagram showing an example of the syntax of the Original Stereo Video Box.
FIG. 54 is a diagram showing an example of the semantics of fields defined in the Original Stereo Video Box.
FIG. 55 is a diagram showing an example of a display size signal.
FIG. 56 is a diagram showing an example of syntax of a Pixel Aspect Ratio Box.
FIG. 57 is a diagram showing an example of the semantics of fields defined in the Pixel Aspect Ratio Box.
FIG. 58 is a diagram showing an example of a signal of a pixel aspect ratio during display.
FIG. 59 is a diagram showing an example of the syntax of the Original Scheme Information Box.
FIG. 60 is a diagram showing an example of syntax of a 2D Coverage Information Box.
FIG. 61 is a diagram showing an example of the semantics of fields defined in the 2D Coverage Information Box.
FIG. 62 is a diagram showing an example of a stereo_presentation_suitable signal.
FIG. 63 is a diagram showing an example of the syntax of a Track Stereo Video Box.
FIG. 64 is a diagram showing an example of syntax of a 2D Coverage Information Box.
FIG. 65 is a diagram showing an example of the semantics of fields defined in the 2D Coverage Information Box.
FIG. 66 is a diagram showing an example of a view_idc signal.
FIG. 67 is a flowchart illustrating an example of the flow of upload processing.
FIG. 68 is a flowchart illustrating an example of the flow of content reproduction processing.
FIG. 69 is a diagram showing an example of attribute values ​​of a 2D coverage information descriptor.
FIG. 70 is a diagram illustrating an example of attribute values ​​of a 2D coverage information descriptor, following FIG. 69.
FIG. 71 is a diagram showing an example of a data type.
FIG. 72 is a block diagram showing a main configuration example of a computer.
FIG. 73 is a diagram showing an example of syntax of Sub Picture Composition Box.
FIG. 74 is a diagram showing an example of the syntax of Supplemental Property.
MODE FOR CARRYING OUT THE INVENTION
[0016]
　Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be given in the following order.
1. Signal of information about sub-picture
2. (Signal of the display area of the sub-picture, extension ISOBMFF) First Embodiment
3. (Signal of the display area of the sub-picture, extension MPD) Second Embodiment
4. (Signal of the stereo information of the whole picture, extension ISOBMFF) Third Embodiment
5. 4. Fourth embodiment (stereo information signal of whole picture, extension of MPD)
6. Note
[0017]
　<1. Signal of information about sub-picture>
　　
　The scope disclosed by the present technology is not only the content described in the embodiments but also known at the time of application. The contents described in the following non-patent documents are also included.
[0018]
　Non-Patent Document 1: (above)
　Non-Patent Document 2: (above)
　Non-Patent Document 3: (above)
　Non-Patent Document 4: (above)
[0019]
　That is, the contents described in the above non-patent documents also serve as the basis for determining the support requirement. For example, technical terms such as Parsing, Syntax, and Semantics are also within the scope of the present technology even if there is no direct description in the embodiments, and It shall meet the scope support requirements.
[0020]
　　
　Conventionally, as described in, for example, Non-Patent Document 1 and Non-Patent Document 2, MPEG-DASH (Moving Picture Experts Group- Dynamic Adaptive Streaming over HTTP).
[0021]
　With this MPEG-DASH, for example, using HTTP, which is the same communication protocol as when downloading a web page of the Internet from a website, to realize video playback at an optimal bit rate according to the fluctuation of the network bandwidth. You can
[0022]
　This standard makes it easier to develop the infrastructure for video distribution services and the technology for video playback clients. In particular, for businesses that handle distribution services, the compatibility between the video distribution service and the video playback client is improved, and there is an advantage that existing content assets can be easily utilized, and it is expected to promote the growth of the market. To be done.
[0023]
　MPEG-DASH is mainly composed of two technical projects. Operation of a standard that defines a manifest file specification called MPD (Media Presentation Description) that describes metadata that manages video and audio files, and a file format called segment format for actually transmitting video content It is a standard.
[0024]
　As this file format, for example, as described in Non-Patent Document 3, ISOBMFF (International Organization for Standardization Base Media File) which is a file container specification of the international standard technology "MPEG-4 (Moving Picture Experts Group-4)" for moving image compression. Format). Functional extensions to meet the requirements of MPEG-DASH have been added to ISOBMFF as extended specifications of ISO/IEC (International Organization for Standardization/International Electrotechnical Commission) 14496-12.
[0025]
　　
　By the way, like so-called spherical image, a stereoscopic structure image obtained by projecting an image of 360 degrees in the horizontal direction and 180 degrees in the vertical direction on a stereoscopic structure is displayed. , There is a projected plane image mapped to the plane image. For example, by rendering a peripheral image (a celestial sphere image) viewed from a viewpoint into a stereoscopic structure image centered on the viewpoint, the image around the viewpoint can be expressed more naturally, or An image in a desired line-of-sight direction can be easily generated from the three-dimensional structure image.
[0026]
　In recent years, it has been considered to use MPEG-DASH for distribution of this projection plane image (such as spherical image). For example, as described in Non-Patent Document 4, MPEG-DASH can be applied by mapping a three-dimensional structure image on a single plane and distributing the three-dimensional structure image as a projection plane image mapped on the plane. ..
[0027]
　As a method of projecting onto a three-dimensional structure and mapping onto a plane (also called a projection format), there are, for example, ERP (Equirectangular projection) and CMP (Cubemap projection). For example, in the case of ERP, a three-dimensional structure image in which an image of 360 degrees in the horizontal direction and 180 degrees in the vertical direction is projected on a spherical three-dimensional structure is And are orthogonally mapped. In the case of CMP, for example, an image of 360 degrees in the horizontal direction and 180 degrees in the vertical direction is projected on each surface of the cube. Is mapped to.
[0028]
　The projection plane image on which the spherical image is projected and mapped in this way is also referred to as a projected picture. That is, the projected picture is a two-dimensional image (two-dimensional picture) that represents a celestial sphere image, which is determined for each projection format.
[0029]
　In the MPEG-I Part2 Omnidirectional Media Format (ISO/IEC 23090-2) FDIS (Final Draft International Standards) (hereinafter, also referred to as OMAF) described in Non-Patent Document 4, a projection plane image of one spherical image (whole image) A technique for dividing a picture (also called a picture) into a plurality of sub-pictures and storing the sub-pictures in a plurality of tracks has been discussed.
[0030]
　For example, there is a use case in which a sub-picture track (sub-picture track) corresponding to a visual field is configured for each specific visual field area, and the client selects and reproduces the sub-picture track according to its own visual field area. is there.
[0031]
　　
　A Box hierarchical structure 11 of FIG. 1 shows an example of a Box hierarchical structure of an ISOBMFF file when converting a spherical image into a sub-picture track.
[0032]
　In this case, as shown in the Box hierarchical structure 11, information about the entire picture is stored under the Track Group Box. For example, the Sub Picture Composition Box (spco) stores information used for grouping sub-picture tracks, such as whether a picture is sub-pictureized. In addition, Boxes such as Sub Picture Region Box (sprg), Region Wise Packing Box (rwpk), and Stereo Video Box (stvi) are formed below it.
[0033]
　The Sub Picture Region Box stores sub-picture division information indicating how to divide the sub-picture. The Region Wise Packing Box stores region-wise packing information of the whole picture before division. Further, the Stereo Video Box stores information (stereo information) regarding stereo display (stereoscopic display) of the entire picture. The stereo information is information indicating the type of stereoscopic display image, such as side by side or top & bottom.
[0034]
　In addition, Restricted Sample Entry (resv) (a type of Sample Entry) under Sample Description Box (stsd) under Sample Table Box (stbl) under Media Information Box (minf) under Media Box (mdia) Boxes such as Projected Omnidirectional Video Box (povd) and Stereo Video Box (stvi) are formed under Scheme Information Box (schi) under Restricted Scheme Information Box (rinf) below.
[0035]
　The Projected Omnidirectional Video Box stores metadata related to spherical images. StereoVideoBox stores stereo information about the sub-picture corresponding to the Box.
[0036]
　The Box hierarchical structure 12 of FIG. 2 shows an example of the Box hierarchical structure of the ISOBMFF file when the spherical image is not converted into a sub-picture track.
[0037]
　In this case, as shown in the Box hierarchical structure 12, the Track Group Box is not formed, and the Region Wise Packing Box is formed below the Projected Omnidirectional Video Box.
[0038]
　That is, when sub-picture conversion is performed, the Region Wise Packing Box, which indicates the arrangement information in which the size and position are changed for each picture area, is signaled only to the Sub Picture Composition Box, and the region-wise packing of the entire picture before division is performed. Have information. On the other hand, when sub-picture conversion is not performed, the Region Wise Packing Box has the region-wise packing information of the picture signaled to the Projected Omnidirectional Video Box and stored in the track. Hereinafter, a track having a Sub Picture Composition Box will be referred to as a sub-picture track.
[0039]
　　
　Therefore, depending on whether the track is a sub-picture track or a normal track that is not sub-pictureized, the client projects the projection plane of the image of that track. The process for identifying the display area on the image (projected picture) is different. For example, when selecting and playing a sub-picture track, the Sub Picture Composition Box is parsed to identify the display area on the projected picture of the sub-picture track, and the region-wise packing information and sub-picture division information are Need to identify. On the other hand, this processing is not necessary when selecting and playing back a track that is not a sub-picture track.
[0040]
　The syntax 21 of FIG. 3 shows an example of the syntax of the Sub Picture Composition Box. As shown in syntax 21, a Sub Picture Region Box and a Region Wise Packing Box are set in this Sub Picture Composition Box.
[0041]
　The syntax 22 of FIG. 4 shows an example of the syntax of the Sub Picture Region Box. As shown in syntax 22, fields such as track_x, track_y, track_width, track_height, composition_width, and composition_height are defined in this Sub Picture Region Box.
[0042]
　The semantics 23 of FIG. 5 shows an example of the semantics of the field defined in the Sub Picture Region Box. As shown in the semantics 23, track_x indicates the horizontal position of the sub-picture stored in the track on the whole picture. track_y indicates the vertical position of the sub-picture stored in track on the whole picture. track_width indicates the width of the sub-picture stored in track. track_height indicates the height of the sub-picture stored in track. composition_width indicates the width of the entire picture. composition_height indicates the height of the entire picture.
[0043]
　The syntax 24 of FIG. 6 shows an example of the syntax of Region Wise Packing Box. As shown in syntax 24, Region Wise Packing Struct is set in this Region Wise Packing Box.
[0044]
　The syntax 25 of FIG. 7 shows an example of the syntax of Region Wise Packing Struct. As shown in syntax 25, in this Region Wise Packing Struct, fields such as constituent_picture_matching_flag, num_regions, proj_picture_width, proj_picture_height, packed_picture_width, packed_picture_height, guard_band_flag[i], packing_type[i], GuardBand(i) are set.
[0045]
　The semantics 26 of FIG. 8 shows an example of the semantics of the field defined in Region Wise Packing Struct. As shown in the semantics 26, the constituent_picture_matching_flag is a flag indicating whether the same region-wise packing is applied to the left-eye view (Left view) and the right-eye view (Right view) when the picture is stereo. It is information. For example, when the value of this field is 0, it indicates that it is mono (single-viewpoint view) or that different packings are applied to the Left view and the Right view. Further, when the value of this field is 1, it indicates that the same packing is applied to the Left view and the Right view.
[0046]
　Also, num_regions indicates the number of packed regions. proj_picture_width indicates the width of the projected picture. proj_picture_height indicates the height of the projected picture. packed_picture_width indicates the width of a packed picture (picture that has been region-wise packed). packed_picture_height indicates the height of the packed picture.
[0047]
　Also, guard_band_flag[i] is flag information indicating whether or not a guard band exists. For example, a value of 0 in this field indicates that there is no guard band in the packed region, and a value of 1 in this field indicates that there is a guard band in the packed region. packing_type[i] indicates the shape of the packed region. For example, when the value of this field is 0, it indicates that the packed region is rectangular. GuardBand(i) is guard band information around the area.
[0048]
　Further, as shown in syntax 25, in Region Wise Packing Struct, RectRegionPacking is further set. The syntax 27 of FIG. 9 shows an example of the syntax of this RectRegionPacking. As shown in syntax 27, in this RectRegionPacking, proj_reg_width[i], proj_reg_height[i], proj_reg_top[i], proj_reg_left[i], transform_type[i], packed_reg_width[i], packed_reg_height[i], packed_reg_top[ Fields such as i] and packed_reg_left[i] are set.
[0049]
　The semantics 28 of FIG. 10 shows an example of the semantics of the field defined in RectRegionPacking. As shown in the semantics 28, proj_reg_width[i] indicates the width of the projected region of the region-wise packing application source. proj_reg_height[i] indicates the height of the projected region where the region-wise packing is applied. proj_reg_top[i] indicates the vertical position of the projected region where the region-wise packing is applied. proj_reg_left[i] indicates the horizontal position of the projected region where the region-wise packing is applied. transform_type[i] indicates rotation or mirroring of the packed region. packed_reg_width[i] indicates the width of the packed region rearranged in the region-wise packing. packed_reg_height[i] indicates the height of the packed region rearranged in the region-wise packing. packed_reg_top[i] indicates the vertical position of the packed region rearranged in the region-wise packing. packed_reg_left[i] indicates the horizontal position of the packed region rearranged in the region-wise packing.
[0050]
　That is, for example, when the client selects a sub-picture track according to the user's field of view, it is necessary to parse this information, and compared to when selecting and playing a track that is not a sub-picture track, The processing load might increase.
[0051]
　　 When
　the whole picture of a stereo spherical image is converted into a sub-picture, the stereo information of the whole picture (what kind of stereoscopic display image the whole picture is, etc.) is shown. The Stereo Video Box is signaled to the Sub Picture Composition Box and shows the stereo information of the sub-picture (what kind of stereoscopic display image the sub-picture is, etc.) The Stereo Video Box is the Sample Entry of the track. Signaled under the Scheme Information Box of. On the other hand, when sub-picture conversion is not performed, the Stereo Video Box is signaled only under the Scheme Information Box and has stereo information of the picture stored in the track.
[0052]
　Therefore, the process for the client to identify the stereo information of the track is different depending on whether the track is a sub-picture track or a normal track without sub-picture conversion. For example, when the whole picture is a stereo image (stereoscopic image), the divided sub-picture track includes L view and R view, but the frame packing arrangement may not be top & bottom or side by side. is there.
[0053]
　Therefore, when identifying whether such a sub-picture can be displayed in stereo, it is necessary to parse the Sub Picture Composition Box and identify the region-wise packing information, sub-picture division information, and stereo information. Becomes On the other hand, this processing is not necessary when selecting and playing back a track that is not a sub-picture track.
[0054]
　That is, for example, when the client selects a sub-picture track according to its stereo display capability, the processing load may increase as compared with the case where a track that is not a sub-picture track is selected and played. ..
[0055]
　In the above, selection of the sub-picture track in the ISOBMFF file has been described, but in the MPD file, the sub-picture is managed as an adaptation set (Adaptation Set). The selection of an Adaptation Set that refers to a sub-picture in this MPD file may increase the processing load for the same reason. That is, the load of stream selection may increase regardless of whether it is an ISOBMFF file or an MPD file.
[0056]
　　
　Therefore, when converting an entire picture into a sub-picture, information about the display area of ​​the sub-picture is signaled (provided to the content reproducing side). .. The display area refers to an area in the whole picture. That is, the information about the display area of ​​the sub-picture is information about the area in the entire picture corresponding to the sub-picture, that is, information indicating which part of the whole picture the sub-picture is. This information indicates, for example, the position, size, shape, etc. of the area corresponding to the sub-picture. The method of expressing the area is arbitrary, and for example, the range of the area may be indicated by coordinates or the like.
[0057]
　By doing so, the client that reproduces the content can grasp where the sub-picture is displayed in the omnidirectional image based on this information.
[0058]
　At that time, information about the display area of ​​this sub-picture is signaled as information for each sub-picture. By doing so, the client can easily obtain this information. Therefore, the client can easily select a desired sub-picture stream. For example, when selecting a stream according to the user's visual field, the client can easily select an appropriate stream corresponding to the direction or range of the visual field.
[0059]
　　
　Also, stereo information, which is information regarding stereo display of the whole picture to be sub-pictured, is signaled. By doing so, the client that reproduces the content can determine whether or not the entire picture is a stereo image (stereoscopic image) based on this information, and if it is a stereo image, its type. It can be easily grasped. This allows the client to determine what kind of image the sub-picture contains (eg what type of stereo image (or what part of a mono image (single-view image)) it corresponds to). It can be easily grasped.
[0060]
　Therefore, the client can easily select a desired stream. For example, when selecting a stream according to its own capability, the client can easily select an appropriate stream according to its own capability.
[0061]
　　
　Next, the configuration of a device that performs a signal related to sub-picture will be described. FIG. 11 is a block diagram showing an example of the configuration of a file generation device that is an aspect of an information processing device to which the present technology is applied. The file generation device 100 shown in FIG. 11 is a device that generates an ISOBMFF file (segment file) or an MPD file. For example, the file generation device 100 implements the techniques described in Non-Patent Documents 1 to 4 and is used for an ISOBMFF file containing a stream and a distribution control of the stream by a method compliant with MPEG-DASH. MPD files that are control files are generated, and these files are uploaded (transmitted) to a server that distributes these files via a network.
[0062]
　Note that FIG. 11 shows main components such as a processing unit and a data flow, and the components shown in FIG. 11 are not limited to all. That is, in the file generation device 100, there may be a processing unit not shown as a block in FIG. 11 or a process or data flow not shown as an arrow or the like in FIG.
[0063]
　As shown in FIG. 11, the file generation device 100 includes a control unit 101, a memory 102, and a file generation unit 103.
[0064]
　The control unit 101 controls the operation of the entire file generation device 100. For example, the control unit 101 controls the file generation unit 103 to generate an ISOBMFF file or MPD file or upload the generated ISOBMFF file or MPD file. The control unit 101 uses the memory 102 to perform such control-related processing. For example, the control unit 101 loads the desired program or the like into the memory 102 and executes the program to perform the above-described control-related processing.
[0065]
　Under the control of the control unit 101, the file generation unit 103 performs processing related to generation and upload (transmission) of an ISOBMFF file or MPD file. As shown in FIG. 11, the file generating unit 103 includes a data input unit 111, a data encoding/generating unit 112, an MPD file generating unit 113, a recording unit 114, and an uploading unit 115.
[0066]
　The data input unit 111 performs processing related to acceptance of data input. For example, the data input unit 111 receives input of data such as images necessary for generating textures and meshes, and metadata necessary for generating MPD files. The data input unit 111 also supplies the received data to the data encoding/generating unit 112 and the MPD file generating unit 113.
[0067]
　The data encoding/generating unit 112 performs processing relating to data encoding and file generation. For example, the data encoding/generating unit 112 generates a stream such as a texture or a mesh based on the data such as the image supplied from the data input unit 111. The data encoding/generating unit 112 also generates an ISOBMFF file that stores the generated stream. The data encoding/generating unit 112 also supplies the generated ISOBMFF file to the recording unit 114.
[0068]
　As shown in FIG. 11, the data encoding/generating unit 112 includes a preprocessing unit 121, an encoding unit 122, and a segment file generating unit 123.
[0069]
　The pre-processing unit 121 performs processing on data such as an image before encoding. For example, the preprocessing unit 121 generates a texture or mesh stream based on data such as an image supplied from the data input unit 111. Further, for example, the preprocessing unit 121 supplies the generated stream to the encoding unit 122.
[0070]
　The encoding unit 122 performs processing related to stream encoding. For example, the encoding unit 122 encodes the stream supplied from the preprocessing unit 121. In addition, for example, the encoding unit 122 supplies the encoded data obtained by the encoding to the segment file generating unit 123.
[0071]
　The segment file generation unit 123 performs processing relating to generation of a segment file. For example, the segment file generation unit 123 converts the encoded data supplied from the encoding unit 122 into files on a segment-by-segment basis (generates a segment file) based on the metadata and the like supplied from the data input unit 111. Further, for example, the segment file generation unit 123 supplies the ISOBMFF file generated as described above to the recording unit 114 as a process related to generation of the segment file. For example, the segment file generation unit 123 generates an ISOBMFF file as a segment file and supplies the generated ISOBMFF file to the recording unit 114.
[0072]
　The MPD file generation unit 113 performs processing related to generation of MPD files. For example, the MPD file generation unit 113 generates an MPD file based on the metadata and the like supplied from the data input unit 111. Further, for example, the MPD file generation unit 113 supplies the generated MPD file to the recording unit 114. Note that the MPD file generation unit 113 may acquire the metadata and the like required to generate the MPD file from the segment file generation unit 123.
[0073]
　The recording unit 114 has, for example, an arbitrary recording medium such as a hard disk or a semiconductor memory, and performs processing relating to data recording and the like. For example, the recording unit 114 records the MPD file supplied from the MPD file generation unit 113. Further, for example, the recording unit 114 records the segment file (for example, ISOBMFF file) supplied from the segment file generation unit 123.
[0074]
　The upload unit 115 performs processing related to file upload (transmission). For example, the upload unit 115 reads the MPD file recorded in the recording unit 114. Further, for example, the upload unit 115 uploads (transmits) the read MPD file to a server (not shown) that distributes the MPD file to the client or the like via a network or the like.
[0075]
　Also, for example, the upload unit 115 reads a segment file (for example, an ISOBMFF file) recorded in the recording unit 114. Further, for example, the upload unit 115 uploads (transmits) the read segment files to a server (not shown) that distributes the segment files to clients and the like via a network or the like.
[0076]
　That is, the upload unit 115 functions as a communication unit that transmits the MPD file and the segment file (for example, ISOBMFF file) to the server. The destination of the MPD file and the destination of the segment file (for example, ISOBMFF file) by the upload unit 115 may be the same or different. In addition, here, an example in which the file generation device 100 functions as a device that uploads an MPD file or a segment file (for example, an ISOBMFF file) to a server that distributes those files to a client will be described. You may make it function as the server. In that case, the upload unit 115 of the file generation device 100 may deliver the MPD file or the segment file (eg, ISOBMFF file) to the client via the network.
[0077]
　　
　FIG. 12 is a block diagram showing an example of the configuration of a client device which is one mode of an information processing device to which the present technology is applied. The client device 200 shown in FIG. 12 is a device that acquires an MPD file or a segment file (for example, an ISOBMFF file), and reproduces the content based on those files. For example, the client device 200 implements the techniques described in Non-Patent Documents 1 to 4 and acquires a segment file from the server (or the above-described file generation device 100) by a method conforming to MPEG-DASH. , Play the stream (content) included in the segment file. At this time, the client device 200 may acquire the MPD file from the server (or the above-described file generation device 100), select a desired segment file using the MPD file, and acquire the desired segment file from the server.
[0078]
　Note that FIG. 12 illustrates main components such as a processing unit and a data flow, and the components illustrated in FIG. 12 are not necessarily all. That is, in the client device 200, a processing unit not shown as a block in FIG. 12 may exist, or a process or data flow not shown as an arrow or the like in FIG. 12 may exist.
[0079]
　As shown in FIG. 12, the client device 200 has a control unit 201, a memory 202, and a reproduction processing unit 203.
[0080]
　The control unit 201 controls the overall operation of the client device 200. For example, the control unit 201 controls the reproduction processing unit 203 to acquire an MPD file or a segment file (for example, an ISOBMFF file) from the server, or reproduce a stream (content) included in the segment file. The control unit 201 uses the memory 202 to perform processing related to such control. For example, the control unit 201 performs the control-related processing as described above by loading a desired program or the like into the memory 202 and executing the program.
[0081]
　Under the control of the control unit 201, the reproduction processing unit 203 performs processing regarding reproduction of the stream (content) included in the segment file. As shown in FIG. 12, the reproduction processing unit 203 includes a measurement unit 211, an MPD file acquisition unit 212, an MPD file processing unit 213, a segment file acquisition unit 214, a display control unit 215, a data analysis/decoding unit 216, and a display. It has a section 217.
[0082]
　The measurement unit 211 performs processing regarding measurement. For example, the measuring unit 211 measures the transmission band of the network between the client device 200 and the server. Further, for example, the measurement unit 211 supplies the measurement result to the MPD file processing unit 213.
[0083]
　The MPD file acquisition unit 212 performs processing related to acquisition of MPD files. For example, the MPD file acquisition unit 212 acquires an MPD file corresponding to desired content (content to be reproduced) from the server via the network. Further, for example, the MPD file acquisition unit 212 supplies the acquired MPD file to the MPD file processing unit 213.
[0084]
　The MPD file processing unit 213 performs processing based on the MPD file. For example, the MPD file processing unit 213 selects a stream to be acquired based on the MPD file supplied from the MPD file acquisition unit 212. Further, for example, the MPD file processing unit 213 supplies the selection result to the segment file acquisition unit 214. When selecting the stream to be acquired, the measurement result supplied from the measurement unit 211 and the information regarding the viewpoint position and the line-of-sight direction of the user supplied from the display control unit 215 are also appropriately used.
[0085]
　The segment file acquisition unit 214 performs processing regarding acquisition of a segment file (for example, an ISOBMFF file). For example, the segment file acquisition unit 214 acquires a segment file in which a stream required for reproducing desired content is stored, from a server via a network. Further, for example, the segment file acquisition unit 214 supplies the acquired segment file to the data analysis/decoding unit 216.
[0086]
　The server from which the segment file acquisition unit 214 acquires the segment file (for example, ISOBMFF file) may be the same as or different from the server from which the MPD file acquisition unit 212 acquires the MPD file. Further, the segment file acquisition unit 214 may acquire the segment file based on the stream selection result supplied from the MPD file processing unit 213. That is, the segment file acquisition unit 214 may acquire the segment file in which the stream selected based on the MPD file or the like is stored from the server.
[0087]
　The display control unit 215 performs processing relating to control of reproduction (display) of content. For example, the display control unit 215 acquires the detection result of the viewpoint position and the line-of-sight direction of the user who views the content. Further, for example, the display control unit 215 supplies the acquired detection result (information regarding the viewpoint position and the line-of-sight direction of the user) to the MPD file processing unit 213 and the data analysis/decoding unit 216.
[0088]
　The data analysis/decryption unit 216 performs processing related to data analysis, decryption, and the like. For example, the data analysis/decoding unit 216 processes the ISOBMFF file supplied from the segment file acquisition unit 214 and generates a display image of content. The data analysis/decoding unit 216 also supplies the display image data to the display unit 217.
[0089]
　As shown in FIG. 12, the data analysis/decoding unit 216 includes a segment file processing unit 221, a decoding unit 222, and a display information generation unit 223.
[0090]
　The segment file processing unit 221 processes a segment file (for example, an ISOBMFF file). For example, the segment file processing unit 221 extracts encoded data of a desired stream from the ISOBMFF file supplied from the segment file acquisition unit 214. Further, for example, the segment file processing unit 221 supplies the extracted encoded data to the decoding unit 222.
[0091]
　The segment file processing unit 221 selects a stream based on the information about the user's viewpoint position and line-of-sight direction supplied from the display control unit 215, the transmission band measured by the measurement unit 211, and the code of the stream. The converted data may be extracted from the segment file.
[0092]
　The decoding unit 222 performs processing related to decoding. For example, the decoding unit 222 decodes the encoded data supplied from the segment file processing unit 221. Further, for example, the decoding unit 222 supplies the stream obtained by the decoding to the display information generation unit 223.
[0093]
　The display information generation unit 223 performs processing relating to generation of display image data. For example, the display information generation unit 223 responds to the user's viewpoint position or line-of-sight direction based on the information regarding the user's viewpoint position or line-of-sight direction supplied from the display control unit 215 and the stream supplied from the decoding unit 222. Generate display image data. Further, for example, the display information generation unit 223 supplies the generated display image data to the display unit 217.
[0094]
　The display unit 217 has an arbitrary display device such as a display using a liquid crystal display panel or a projector, and performs processing relating to image display using the display device. For example, the display unit 217 performs content reproduction such as image display based on the data supplied from the display information generation unit 223.
[0095]
　<2. First Embodiment>
　　 The
　above-mentioned signal of information about the sub-picture display area may be transmitted in an ISOBMFF file which is a segment file.
[0096]
　That is, a file that includes information about an area in the entire picture that corresponds to the stored sub-picture is included as information that is different from the placement information for each picture area, and that further includes image coded data in which the sub-picture is coded. You may do it.
[0097]
　For example, in the file generation apparatus 100 that is an information processing apparatus, the segment file generation unit 123 includes information regarding the area in the entire picture corresponding to the stored sub-picture as information that is different from the arrangement information for each picture area, and Alternatively, the sub-picture may function as a file generation unit that generates a file including encoded image data. That is, the information processing device (for example, the file generation device 100) may include the file generation unit (for example, the segment file generation unit 123).
[0098]
　By doing so, as described above, the client can more easily select the stream based on this information.
[0099]
　In the ISOBMFF file, the stream is managed as a track. That is, when using the ISOBMFF file, the stream is selected by selecting the track.
[0100]
　Also, the above-mentioned picture (entire picture) should be all or part of the omnidirectional image (projection plane image obtained by projecting and mapping the image of 360 degrees in the horizontal direction and 180 degrees in the vertical direction). May be. The omnidirectional image is an omnidirectional image centered on the viewpoint (that is, an image of the periphery viewed from the viewpoint). This spherical image can be rendered into a three-dimensional structure to form an image with 360 degrees in the horizontal direction and 180 degrees in the vertical direction. As described above, by mapping the three-dimensional structure image to a single plane to form a projection plane image, it becomes possible to perform stream distribution control to which MPEG-DASH is applied. That is, in the case where the file generation device 100 sets all or part of such a projection plane image as a whole picture and converts it into a sub-picture, the present technology can be applied as described above. Even when a part of the projection plane image is the whole picture, the information about the display area in the entire projection plane image of the sub-picture is signaled.
[0101]
　For example, as shown in FIG. 13, an image of 360 degrees in the horizontal direction and 180 degrees in the vertical direction is projected onto a three-dimensional structure (cube) by Cubemap projection to generate a three-dimensional structure image 301. Also, the three-dimensional structure image 301 is mapped to a single plane by a predetermined method, and a projected plane image (projected picture) 302 is generated. The file generation device 100 converts such a projected plane image 302 into sub-pictures to generate sub-pictures (sub-picture 303 to sub-picture 308), and generates an ISOBMFF file which stores each in different tracks.
[0102]
　At that time, the file generation device 100, in the ISOBMFF file, provides information (display area information) indicating which sub-picture corresponds to which part of the whole picture (projection plane image 302) as indicated by an arrow 311. Signal.
[0103]
　By doing so, even when the omnidirectional video is distributed, the client can more easily select the stream based on this information as described above.
[0104]
　The information (display area information) about this area may be included in the ISOBMFF file as information for each sub-picture. By doing so, the client can easily grasp which part of the whole picture the sub-picture corresponds to by simply referring to the information of the sub-picture track.
[0105]
　　
　An example of the flow of uploading process executed by the file generating apparatus 100 of FIG. 11 in that case will be described with reference to the flowchart of FIG.
[0106]
　When the upload process is started, the data input unit 111 of the file generation device 100 acquires an image and metadata in step S101.
[0107]
　In step S102, the segment file generation unit 123 generates an ISOBMFF file including the display area information in the projected picture as the information for each sub-picture.
[0108]
　In step S103, the recording unit 114 records the ISOBMFF file generated by the process of step S102.
[0109]
　In step S104, the upload unit 115 reads the ISOBMFF file recorded in step S103 from the recording unit 114 and uploads it to the server.
[0110]
　When the process of step S104 ends, the upload process ends.
[0111]
　By performing the upload process as described above, the file generation device 100 can generate an ISOBMFF file including display area information in the projected picture as information for each sub-picture.
[0112]
　Therefore, the client can more easily select and reproduce an appropriate stream according to the field of view of the user based on the information.
[0113]
　　 Also, stream selection and reproduction may be performed using information about the sub-picture display area signaled to the ISOBMFF file.
[0114]
　That is, a file including information about the area in the entire picture corresponding to the stored sub-picture is included as information different from the arrangement information for each picture area, and further, a file including the encoded image data in which the sub-picture is encoded is acquired. However, the stream of the image coded data may be selected based on the information regarding the area included in the acquired file.
[0115]
　For example, in the client device 200, which is an information processing device, the segment file acquisition unit 214 includes information about an area in the entire picture corresponding to the stored sub-picture as information different from the arrangement information for each picture area, and further, The sub-picture functions as a file acquisition unit that acquires a file including coded image coded data, and the data analysis/decoding unit 216 outputs information about the area included in the file acquired by the file acquisition unit. Based on this, the image processing unit may function as an image processing unit that selects a stream of encoded image data. That is, the information processing device (for example, the client device 200) may include the file acquisition unit (for example, the segment file acquisition unit 214) and the image processing unit (for example, the data analysis/decoding unit 216).
[0116]
　By doing so, the client device 200 can more easily select a stream.
[0117]
　The above-mentioned picture (entire picture) should be all or part of the omnidirectional image (projection plane image obtained by projecting and mapping the image of 360 degrees in the horizontal direction and 180 degrees in the vertical direction). May be. That is, the present technology can be applied as described above even when the client apparatus 200 acquires a stream obtained by converting all or part of a projected plane image into a whole picture into a sub-picture and reproducing the stream. ..
[0118]
　Further, information about this area (display area information) may be included in the ISOBMFF file as information for each sub-picture. By doing so, the client apparatus 200 can easily grasp which part of the whole picture the sub-picture corresponds to by simply referring to the information of the sub-picture track.
[0119]
　　
　An example of the flow of content reproduction processing executed by the client device 200 in that case will be described with reference to the flowchart of FIG.
[0120]
　When the content reproduction process is started, the segment file acquisition unit 214 of the client device 200 acquires an ISOBMFF file including display area information in the projected picture as information for each sub-picture in step S121.
[0121]
　In step S122, the display control unit 215 acquires the measurement result of the viewpoint position (and the line-of-sight direction) of the user.
[0122]
　In step S123, the measuring unit 211 measures the transmission bandwidth of the network between the server and the client device 200.
[0123]
　In step S124, the segment file processing unit 221 selects a sub-picture track corresponding (corresponding to) the visual field of the user of the client device 200 based on the display area information of the projected picture of the sub-picture.
[0124]
　In step S125, the segment file processing unit 221 extracts the encoded data of the stream of the track selected in step S124 from the ISOBMFF file acquired in step S121.
[0125]
　In step S126, the decoding unit 222 decodes the encoded data of the stream extracted in step S125.
[0126]
　In step S127, the display information generation unit 223 reproduces the stream (content) obtained by decoding in step S126. More specifically, the display information generation unit 223 generates display image data from the stream, supplies it to the display unit 217, and displays it.
[0127]
　When the process of step S127 ends, the content reproduction process ends.
[0128]
　By performing the content reproduction process as described above, the client apparatus 200 can more easily select the stream by using the information about the display area of ​​the sub-picture included in the ISOBMFF file. For example, the client device 200 can easily select an appropriate stream according to the field of view of the user based on the information.
[0129]
　　
　As described above, the segment file generation unit 123 of the file generation device 100 indicates which part of the projected picture the sub-picture corresponds to in the projected picture in the OMAF ISOBMFF file. The display area information of picture is newly defined and signaled to track. That is, the segment file generation unit 123 defines the display area information of the sub-picture as information for each sub-picture.
[0130]
　For example, the segment file generation unit 123 defines a 2D Coverage Information Box as the display area information of the sub-picture, and signals it as a box different from the Region Wise Packing Box. For example, the segment file generation unit 123 defines the 2D Coverage Information Box as a Scheme Information Box. For example, the segment file generation unit 123 may define the 2D Coverage Information Box as a Projected Omnidirectional Video Box under the Scheme Information Box. Further, the segment file generating unit 123 may define the 2D Coverage Information Box as another Box.
[0131]
　In other words, the display area information of the sub-picture (information about the area in the whole picture corresponding to the sub-picture stored in the track) is different from the Region Wise Packing Box in the Scheme Information Box of the ISOBMFF file or its Scheme Information Box. It may be stored in Box in the lower hierarchy.
[0132]
　By doing so, the client device 200 can easily select and play the sub-picture track without parsing the Sub Picture Composition Box.
[0133]
　In addition, this 2D Coverage Information Box displays the display area information even if the picture (picture) stored in track is not a sub-picture, or if the Region Wise Packing Box does not exist (when the picture is not Region Wise Packing). Can be used for signals.
[0134]
　The syntax 331 of FIG. 16 shows an example of the syntax of this 2D Coverage Information Box. As shown in syntax 331, fields such as proj_picture_width, proj_picture_height, proj_reg_width, proj_reg_height, proj_reg_top, proj_reg_left are set in the 2D Coverage Information Box.
[0135]
　The semantics 332 of FIG. 17 shows an example of the semantics of the field defined in this 2D Coverage Information Box. As shown in the semantics 332, proj_picture_width indicates the width of the projected picture. proj_picture_height indicates the height of the projected picture. proj_reg_width indicates the width of the area on the projected picture corresponding to the picture of track. proj_reg_height indicates the height of the area on the projected picture corresponding to the picture of track. proj_reg_top indicates the vertical direction coordinate of the area on the projected picture corresponding to the picture of track. proj_reg_left indicates the area horizontal direction coordinate on the projected picture corresponding to the track picture.
[0136]
　That is, various information as shown in FIG. 18 is defined in the 2D Coverage Information Box.
[0137]
　Note that each of these fields may be indicated by the actual number of pixels, or proj_reg_width, proj_reg_height, proj_reg_top, and proj_reg_left may be indicated by relative values ​​to proj_picture_width and proj_picture_height. When presented in actual pixels, it is useful in choosing a track depending on the resolution of the client's display.
[0138]
　The client device 200 can easily identify the display area of ​​the sub-picture track by referring to the 2D Coverage Information Box of the sub-picture track having such a configuration without parsing the Sub Picture Composition Box. Thereby, the client device 200 can more easily select the sub-picture track according to the visual field of the user, for example. Note that the client device 200 can also select a track that is not a sub-picture track by the same process.
[0139]
　In addition, in the Sub Picture Composition Box shown in the syntax 21 of FIG. 3, an additional identification_to_proj_pic_flag field is defined as shown in the syntax 1001 of FIG. 73 to indicate whether the entire picture is the same as the projected picture. When it is the same as the projected picture, the Sub Picture Region Box shown in the syntax 22 of FIG. 4 may indicate the display area information of the sub-picture track. In the value of the identical_to_proj_pic_flag field, for example, 0 indicates that the whole picture is different from the projected picture, and 1 indicates that the whole picture is the same as the projected picture.
[0140]
　At this time, when the identification_to_proj_pic_flag field is 1, the whole picture is not subjected to region-wise packing processing, and the track_x, track_y, track_width, track_height, composition_width, and composition_height field semantics of the Sub Picture Region Box shown in FIG. Are the same as the semantics of the proj_reg_left, proj_reg_top, proj_reg_width, proj_reg_height, proj_picture_width, proj_picture_height fields of the 2D Coverage Information Box shown in the semantics 332 of FIG. 17, respectively.
[0141]
　The identification_to_proj_pic_flag field may be additionally defined in the Sub Picture Region Box, or may be defined in another Box. Further, whether or not the entire picture is the same as the projected picture may be indicated by the presence/absence of a specific Box.
[0142]
　Further, it is possible to use 1 bit of 24-bit flags commonly held by the Sub Picture Composition Box and other Boxes that extend the Full Box to indicate whether or not the entire picture is the same as the projected picture.
[0143]
　　 In
　syntax 331 of FIG. 16, sub-picture includes discontinuous area on projected picture as shown in FIG. I can't. In the case of the example in FIG. 19, the projected picture 351 is converted into a sub-picture, and sub-picture 352 to sub-picture 355 are formed. In this case, the sub-picture 352 includes the Left plane and the Right plane in the stereoscopic structure image (the Left plane and the Right plane are adjacent to each other). The Left plane and the Right plane are discontinuous in the projected picture 351. Also, the sub-picture 353 includes the Top plane and the Bottom plane in the three-dimensional structure image (the Top plane and the Bottom plane are adjacent to each other). The Top surface and the Bottom surface are discontinuous in the projected picture 351.
[0144]
　In the syntax 331 of FIG. 16, only one grouped area of ​​the projected picture can be designated, and thus a plurality of such discontinuous areas cannot be designated.
[0145]
　Therefore, a plurality of areas may be designated in the 2D Coverage Information Box, and a plurality of discontinuous areas in the projected picture may be designated.
[0146]
　The syntax 371 of FIG. 20 shows an example of the syntax of the 2D Coverage Information Box in this case. In this case, the num_regions field is added to the field being defined, as shown in syntax 371. The semantics 372 of FIG. 21 shows an example of the semantics of the field added in the 2D Coverage Information Box in this case. As shown in the semantics 372, num_regions indicates the number of regions on the projected picture included in the sub-picture.
[0147]
　That is, in this case, in the 2D Coverage Information Box, the fields shown in FIG. 17 are defined for each area of ​​the projected picture (independently of each other) using the num_regions field. Therefore, it is possible to specify a plurality of areas of the projected picture. This enables a discontinuous display area signal of a projected picture.
[0148]
　When the 2D Coverage Information Box is signaled to the Sub Picture Composition Box, the display area of ​​the entire picture (projected picture) may be signaled.
[0149]
　If the 2D Coverage Information Box does not exist in the Projected Omnidirectional Video Box of the track, it may indicate that the track stores the 360° spherical image. Similarly, if the 2D Coverage Information Box does not exist in the Sub Picture Composition Box, the entire picture configured by the sub-picture track may be a 360° spherical image.
[0150]
　　
　The Region Wise Packing Struct in the Region Wise Packing Box specified by OMAF is expanded to signal which part of the projected picture the sub-picture of the track corresponds to. You may do it. The signal location of Region Wise Packing Box is under Projected Omnidirectional Video Box of Sample Entry of sub-picture track. The Region Wise Packing Box may be signaled to other places.
[0151]
　For example, a flag indicating that the display area information of the sub-picture is newly signaled and a Rect Projected Region structure that signals the display area information of the sub-picture are defined, and signaled by Region Wise Packing Struct. The Region Wise Packing Struct can be used as a signal of the display area information even when the picture stored in the track is not a sub-picture.
[0152]
　The syntax 373 of FIG. 22 shows an example of the syntax of Region Wise Packing Struct in that case. In this case, as shown in syntax 373, the 2D_coverage_flag field is added to the fields defined in Region Wise Packing Struct. The semantics 374 of FIG. 23 shows an example of the semantics of the field additionally defined in Region Wise Packing Struct in this case. As shown by the semantics 374, 2D_coverage_flag is flag information indicating whether or not to signal only the display area on the projected picture. For example, a value of 0 in this field indicates that region-wise packing information is signaled. When the value of this field is 1, it indicates that the display area on the projected picture is signaled.
[0153]
　In addition, in the Region Wise Packing Struct in this case, RectProjetedRegion is further defined. The syntax 375 of FIG. 24 shows an example of the syntax of the RectProjetedRegion. As shown in syntax 375, in this RectProjetedRegion, fields such as proj_reg_width[i], proj_reg_height[i], proj_reg_top[i], proj_reg_left[i] are defined.
[0154]
　The semantics 376 of FIG. 25 shows an example of the semantics of the field defined in this RectProjetedRegion. As shown in the semantics 376, proj_reg_width indicates the width of the area on the projected picture corresponding to the picture of track. proj_reg_height indicates the height of the area on the projected picture corresponding to the picture of track. proj_reg_top indicates the vertical direction coordinate of the area on the projected picture corresponding to the picture of track. proj_reg_left indicates the area horizontal direction coordinate on the projected picture corresponding to the track picture.
[0155]
　Note that each of the above fields may be indicated by the actual number of pixels, or may be indicated by a relative value for proj_picture_width and proj_picture_height where proj_reg_width, proj_reg_height, proj_reg_top and proj_reg_left are signaled by Region Wise Packing Struct. Good.
[0156]
　Further, Rect Wise Packing Struct may be extended so that only the display area information in the projected picture is signaled when 2D_coverage_flag==1.
[0157]
　The syntax 377 of FIG. 26 shows an example of the syntax of Rect Wise Packing Struct in that case. The syntax 378 in FIG. 27 is a diagram showing an example of the syntax of Rect Region Packing set in the Rect Wise Packing Struct in this case.
[0158]
　　 The Coverage Information Box,
　which shows the display area on the spherical surface of the track defined by OMAF, is extended and the display area on the projected picture can be signaled by the newly defined 2D Content Coverage Struct. May be.
[0159]
　That is, the display area information of the sub-picture (information about the area in the whole picture corresponding to the sub-picture stored in the track) is stored in the Coverage Information Box indicating the display area on the spherical surface of the track in the ISOBMFF file. You can
[0160]
　The syntax 379 of FIG. 28 shows an example of the syntax of the expanded Coverage Information Box. As shown in syntax 379, 2D_coverage_flag, ContentCoverageStruct(), and 2DContentCoverageStruct() are defined in the Coverage Information Box in this case.
[0161]
　The semantics 380 of FIG. 29 shows an example of the semantics of these fields. As shown in the semantics 380, 2D_coverage_flag is flag information that signals the type of display area information. When this value is 0, it indicates that the spherical display area information is signaled, and when this value is 1, it indicates that the display area on the projected picture is signaled. ContentCoverageStruct() signals the spherical display area of ​​track. 2DContentCoverageStruct() signals the display area on the projected picture of track. The fields in the 2D Content Coverage Struct are the same as those in the 2D Coverage Information Box in the case of FIG.
[0162]
　The Content Coverage Struct may be expanded to signal the display area on the projected picture in addition to the spherical display area.
[0163]
　　 In the
　above, signaling when the sub-picture division method does not dynamically change in the stream has been described. On the other hand, when the division method dynamically changes, the display area information of the sub-picture in the projected picture dynamically changes in the stream. In that case, the above example cannot cope.
[0164]
　Therefore, an example of additional signaling for signaling the dynamically changing display area information of the sub-picture will be described below. The signaled information is the same as the information signaled by the 2D Coverage Information Box described above (for example, FIG. 16 and the like).
[0165]
　　 In
　HEVC and AVC, a 2D Coverage Information SEI message is newly defined, in which sub-picture display area information that dynamically changes in the stream is signaled in access unit units. You may do it.
[0166]
　That is, the display area information of the sub-picture (information about the area in the entire picture corresponding to the sub-picture stored in the track) may be stored in the Supplemental Enhancement information message of the ISOBMFF file.
[0167]
　The syntax 381 of FIG. 30 shows an example of the syntax of the 2D Coverage Information SEI message in that case. As shown in syntax 381, in the 2D Coverage Information SEI message, 2D_coverage_information_cancel_flag, 2D_coverage_information_persistence_flag, 2D_coverage_information reserved_zero_6bits, proj_picture_width, proj_picture_he_, [_]_i__height, reg_height, num_regions, proj_reg_width_i[_]_height, Is set.
[0168]
　The semantics 382 of FIG. 31 shows an example of the semantics of the field defined in the 2D Coverage Information SEI message. As shown in the semantics 382, ​​2D_coverage_information_cancel_flag is flag information regarding cancellation of 2D_coverage_information. If this value is 1, the persistent application of SEI preceding in output order is canceled. Also, if this value is 0, 2D coverage information is signaled.
[0169]
　2D_coverage_information_persitence_flag is flag information related to the application range of SEI. When this value is 0, the SEI information is applied only to the picture including the SEI. When this value is 1, the application of SEI continues until a new coded video sequence is started or the end of the stream is reached.
[0170]
　2D_coverage_information_reserved_zero_6bits is filled with 0. proj_picture_width indicates the width of the projected picture. proj_picture_height indicates the height of the projected picture. num_regions indicates the number of regions on the projected picture. proj_reg_width indicates the width of the area on the projected picture corresponding to the stream. proj_reg_height indicates the height of the area on the projected picture corresponding to the stream. proj_reg_top indicates the vertical coordinate of the area on the projected picture corresponding to the stream. proj_reg_left indicates the horizontal coordinate of the area on the projected picture corresponding to the stream.
[0171]
　The above fields may be indicated by the actual number of pixels, or proj_reg_width, proj_reg_height, proj_reg_top and proj_reg_left may be indicated by relative values ​​with respect to proj_picture_width and proj_picture_height.
[0172]
　　 In
　addition, by using the mechanism of timed metadata, which is a stream that stores time-varying metadata, 2D Coverage Information timed metadata is newly defined, and within it, it dynamically changes within the referenced stream. The display area information of the sub-picture may be signaled. For example, '2dco' is used as the track reference type for the track associated with the 2D Coverage Information timed metadata.
[0173]
　That is, the display area information of the sub-picture (information about the area in the entire picture corresponding to the sub-picture stored in the track) may be stored in the timed metadata of the ISOBMFF file.
[0174]
　By using the timed metadata, the client can pre-identify the dynamically changing display area without decoding the sub-picture stream and use it as a criterion for selecting which stream to select.
[0175]
　The syntax 383 of FIG. 32 shows an example of the syntax of 2D Coverage Information Sample Entry. The syntax 384 of FIG. 33 shows an example of the syntax of the 2D Coverage Information Sample.
[0176]
　In 2D Coverage Information Sample Entry, proj_picture_width and proj_picture_height, which are generally unchanged in the stream, are signaled. If these change in the stream, they may be signaled in 2DCoverageInformationSample.
[0177]
　Note that the semantics of each field in the 2D Coverage Information Sample Entry and the 2D Coverage Information Sample are the same as in FIGS. 17 and 21.
[0178]
　　
　Using the tool called Sample Group, which is a mechanism that links meta information in sample units, defined in ISOBMFF, the display area information of the sub-picture that dynamically changes in the stream is displayed in sample units. You may make it signal.
[0179]
　As shown in FIG. 34, the Sample Group in which the meta information is described is signaled as a group entry (Group Entry) in the sample group description box (Sample Group Description Box) of the sample table box (Sample Table Box), and the sample It is linked to sample through the To Group Box.
[0180]
　As shown in FIG. 34, the grouping type (grouping_type) of the Sample To Group Box indicates the grouping_type of the associated Sample Group Description Box. A sample count (sample_count) and a group description index (group_description_index) are signaled for one entry, the group_description_index indicates the index (index) of the associated group entry (Group Entry), and the sample_count indicates the Group Entry. Indicates the number of samples belonging to.
[0181]
　For example, a 2D Coverage Information Sample Group Entry may be newly defined, and the display area information of the sub-picture that dynamically changes in the stream may be stored therein.
[0182]
　That is, the display area information of the sub-picture (information about the area in the entire picture corresponding to the sub-picture stored in the track) may be stored in Sample Group Entry of the ISOBMFF file.
[0183]
　The syntax 391 of FIG. 35 shows an example of the syntax of the 2D Coverage Information Sample Group Entry. As described above, this Sample Group Entry is signaled to the Sample Group Description Box, and the Sample To Group Box associates the sample with the Sample Group Entry. grouping_type will be '2cgp'.
[0184]
　The semantics of each field in this 2D Coverage Information Sample Group Entry are the same as those in FIGS. 16 and 21.
[0185]
　Note that the above three examples (Supplemental Enhancement Information (SEI) message, Timed metadata, Sample Group) are used as signals of dynamically changing display area information even when the picture stored in track is not a sub-picture. Can be used.
[0186]
　If the display area of ​​the sub-picture on the projected picture changes dynamically as described above, the information of the 2D Coverage Information Box signaled to the Projected Omnidirectional Video Box should be the initial value of the display area of ​​the stream. You can
[0187]
　Further, a flag indicating that the display area of ​​the projected picture of the sub-picture dynamically changes in the stream may be signaled in the 2D Coverage Information Box or other Box. With this information, the client can easily identify that the stream has a dynamically changing display area.
[0188]
　<3. Second Embodiment>
　　 The signal of information on the display area of
　sub-picture described above may be performed in the MPD file. That is, in order to enable the client to select and reproduce the Adaptation Set that refers to the sub-picture, for example, according to the user's view, the display area information on the projected picture of the sub-picture is newly added in the MPD file. It may be defined and signaled to the Adaptation Set.
[0189]
　In other words, the image coded data for each sub-picture, which is coded by dividing the entire picture into a plurality of sub-pictures, is managed, and the information about the area in the entire picture corresponding to the sub-picture is set as the arrangement information for each picture area. It is also possible to generate a control file which is used as a distribution control of image coded data and which is included as different information.
[0190]
　For example, in the file generation device 100 which is an information processing device, the MPD file generation unit 113 manages image coded data for each sub-picture in which the whole picture is divided into a plurality of sub-pictures and coded, and the sub-pictures are managed. It may be configured to function as a file generation unit that generates a control file used for distribution control of image coded data, which includes information about an area in the entire picture corresponding to the above as information different from the arrangement information for each picture area. .. That is, the information processing device (for example, the file generation device 100) may include the file generation unit (for example, the MPD file generation unit 113).
[0191]
　By doing so, as described above, the client can more easily select the stream based on this information.
[0192]
　In the MPD file, the metadata for each stream is managed as an adaptation set (Adaptation Set) or a representation (Representation). That is, when using an MPD file, a stream is selected by selecting an adaptation set (Adaptation Set) or a representation (Representation).
[0193]
　Also, the above-mentioned picture (entire picture) should be all or part of the omnidirectional image (projection plane image obtained by projecting and mapping the image of 360 degrees in the horizontal direction and 180 degrees in the vertical direction). May be. That is, in the case where the file generation device 100 sets all or part of such a projection plane image as a whole picture and converts it into a sub-picture, the present technology can be applied as described above.
[0194]
　By doing so, even when the omnidirectional video is distributed, the client can more easily select the stream based on this information as described above.
[0195]
　The information (display area information) about this area may be included in the MPD file as information for each sub-picture. By doing so, the client can easily understand which part of the whole picture the sub-picture corresponds to, only by referring to the information of the sub-picture referred to by the Adaptation Set.
[0196]
　　
　An example of the flow of uploading process executed by the file generating apparatus 100 of FIG. 11 in that case will be described with reference to the flowchart of FIG. 36.
[0197]
　When the upload process is started, the data input unit 111 of the file generation device 100 acquires an image and metadata in step S201.
[0198]
　In step S202, the segment file generation unit 123 generates a segment file of the image.
[0199]
　In step S203, the MPD file generation unit 113 generates an MPD file including the display area information of the projected picture as the information of each sub-picture.
[0200]
　In step S204, the recording unit 114 records the segment file generated by the process of step S202. The recording unit 114 also records the MPD file generated by the process of step S203.
claims
[Claim 1]
　The image coded data for each sub-picture, in which the whole picture is divided into a plurality of sub-pictures and encoded, is managed, and the information about the area in the whole picture corresponding to the sub-picture is different from the arrangement information for each picture area. An information
　processing apparatus , comprising: a file generation unit that generates a control file, which is included as information and is used to control distribution of the encoded image data .
[Claim 2]

　The information processing apparatus according to claim 1 　, wherein the whole picture is a spherical image .
[Claim 3]
　The information
　processing apparatus according to claim 1 , wherein the information about the area is included in the control file as information for each sub-picture .
[Claim 4]
　The control file is an MPD (Media Presentation Description) file,
　the image coded data for each sub-picture is managed for each adaptation set, and
　the arrangement information for each picture area is stored in a Region-wise packing descripitor. ,
　information on the region is defined in Supplemental or Tools and Drivers Property or Essential Property of the MPD file
　information processing apparatus according to claim 3.
[Claim 5]

　The information processing apparatus according to claim 1, 　wherein the control file further includes stereo information that is information regarding stereo display of an adaptation set .
[Claim 6]

　The information processing apparatus according to claim 5, 　wherein the whole picture is a spherical image .
[Claim 7]

　The information processing apparatus according to claim 1, 　wherein the control file further includes view information indicating a view type of the sub-picture .
[Claim 8]
　The information
　processing apparatus according to claim 7, wherein the view information is information for each area included in the sub-picture .
[Claim 9]

　The information processing apparatus according to claim 8, 　wherein the control file further includes information indicating whether the view information exists for each area .
[Claim 10]

　The information processing apparatus according to claim 7, 　wherein the control file further includes information indicating whether or not the adaptation set can be displayed in stereo .
[Claim 11]
　The image coded data for each sub-picture, in which the whole picture is divided into a plurality of sub-pictures and encoded, is managed, and the information about the area in the whole picture corresponding to the sub-picture is different from the arrangement information for each picture area. An information
　processing method for generating a control file, which is included as information and is used for distribution control of the encoded image data .
[Claim 12]
　The image coded data for each sub-picture, in which the whole picture is divided into a plurality of sub-pictures and encoded, is managed, and the information about the area in the whole picture corresponding to the sub-picture is different from the arrangement information for each picture area. The image code
　based on the information about the area included in the control file acquired by the file acquisition unit, which is included as information, and which acquires the control file used to control the distribution of the encoded image data, An
　information processing apparatus including an image processing unit that selects a stream of encoded data.
[Claim 13]

　The information processing apparatus according to claim 12, 　wherein the whole picture is a spherical image .
[Claim 14]
　The information
　processing apparatus according to claim 12 , wherein the information regarding the area is included in the control file as information for each sub-picture .
[Claim 15]
　The control file is an MPD (Media Presentation Description) file,
　the image coded data for each sub-picture is managed for each adaptation set, and
　the arrangement information for each picture area is stored in a Region-wise packing descripitor. ,
　information on the region is defined in Supplemental or Tools and Drivers Property or Essential Property of the MPD file
　information processing apparatus according to claim 14.
[Claim 16]

　The information processing apparatus according to claim 12, 　wherein the control file further includes stereo information that is information regarding a stereo display of an adaptation set .
[Claim 17]

　The information processing apparatus according to claim 16, 　wherein the entire picture is a spherical image .
[Claim 18]

　The information processing apparatus according to claim 12, 　wherein the control file further includes view information indicating a view type of the sub-picture .
[Claim 19]

　The information processing apparatus according to claim 18, 　wherein the control file further includes information indicating whether or not the adaptation set can be displayed in stereo .
[Claim 20]
　The image coded data for each sub-picture, in which the whole picture is divided into a plurality of sub-pictures and encoded, is managed, and the information about the area in the whole picture corresponding to the sub-picture is different from the arrangement information for each picture area. including the information, the image encoding takes control file used for distribution control of the data,
　based on information relating to the area included in the acquired control files, selects a stream of the encoded image data
　Info Processing method.