Technical field
[0001]
　The present technology relates to a signal processing device and method, and a program, and particularly relates to a signal processing device and method, and a program that can easily determine a localization position of a sound image.
Background technology
[0002]
　In recent years, object-based audio technology has attracted attention.
[0003]
　In the object-based audio, object audio data is composed of a waveform signal for the audio object and meta information indicating localization information of the audio object represented by a relative position from a listening position which is a predetermined reference.
[0004]
　Then, the waveform signal of the audio object is rendered into a signal having a desired number of channels by, for example, VBAP (Vector Based Amplitude Panning) based on the meta information and reproduced (see, for example, Non-Patent Document 1 and Non-Patent Document 2). ).
[0005]
　With object-based audio, it is possible to arrange audio objects in various directions in a three-dimensional space in the production of audio contents.
[0006]
　For example, in Dolby Atoms Panner plus-in for Pro Tools (see Non-Patent Document 3, for example), it is possible to specify the position of an audio object on a 3D graphic user interface. In this technique, by specifying the position on the image of the virtual space displayed on the user interface as the position of the audio object, the sound image of the sound of the audio object can be localized in any direction in the three-dimensional space. ..
[0007]
　On the other hand, the localization of the sound image for the conventional 2-channel stereo is adjusted by a method called panning. For example, by changing the proportional distribution ratio of the left and right channels to a predetermined audio track by a UI (User Interface), it is determined at which position in the left and right direction the sound image is localized.
Prior art documents
Non-patent literature
[0008]
Non-Patent Document 1: ISO/IEC 23008-3 Information technology-High efficiency coding and media delivery in heterogeneous environments-Part 3: 3D audio
Non-Patent Document 2: Ville Pulkki, "Virtual Sound Source Positioning Using Vector Base Amplitude Panning", Journal of AES, vol.45, no.6, pp.456-466, 1997
Non-Patent Document 3: Dolby Laboratories, Inc., “Authoring for Dolby Atmos(R) Cinema Sound Manual”, [online], [2017] October 31st search], Internet
Summary of the invention
Problems to be Solved by the Invention
[0009]
　However, it is difficult to easily determine the localization position of the sound image with the above-mentioned technique.
[0010]
　That is, in both the case of object-based audio and the case of 2-channel stereo, the creator of audio content cannot intuitively specify the localization position of the sound image with respect to the actual listening position of the sound of the content.
[0011]
　For example, in Dolby Atoms Panner plus-in for Pro Tools, it is possible to specify any position on the 3D space as the localization position of the sound image, I can't know if it is in position.
[0012]
　Similarly, even in the case of 2-channel stereo, it is difficult to intuitively grasp the relationship between the proportional division ratio and the localization position of the sound image when designating the proportional division ratio.
[0013]
　For this reason, the creator must repeatedly adjust the localization position of the sound image and listen to the sound at that localization position to determine the final localization position, and reduce the number of times of such localization position adjustment. Needed a sense of experience.
[0014]
　Especially, we want to match the localization position of the sound to the image, for example, by locating the person's voice at the position of the person's mouth on the screen, as if the person's mouth were speaking. In this case, it was difficult to accurately and intuitively specify the localization position on the user interface.
[0015]
　The present technology is made in view of such a situation, and makes it possible to easily determine the localization position of the sound image.
Means for solving the problem
[0016]
　A signal processing device according to an aspect of the present technology is an acquisition unit that acquires information regarding a localization position of a sound image of an audio object in the listening space specified in a state where the listening space viewed from the listening position is displayed, And a generation unit that generates a bitstream based on information about the localization position.
[0017]
　A signal processing method or program according to one aspect of the present technology obtains information about a localization position of a sound image of an audio object in the listening space specified in a state where the listening space viewed from the listening position is displayed, and the localization is performed. Generating a bitstream based on the position information.
[0018]
　In one aspect of the present technology, information regarding a localization position of a sound image of an audio object in the listening space specified in a state where the listening space viewed from the listening position is displayed is acquired, and based on the information regarding the localization position. To generate a bitstream.
Effect of the invention
[0019]
　According to one aspect of the present technology, it is possible to easily determine the localization position of the sound image.
[0020]
　Note that the effects described here are not necessarily limited, and may be any effects described in the present disclosure.
Brief description of the drawings
[0021]
FIG. 1 is a diagram illustrating determination of an edited image and a sound image localization position.
FIG. 2 is a diagram illustrating calculation of a gain value.
FIG. 3 is a diagram showing a configuration example of a signal processing device.
FIG. 4 is a flowchart illustrating a localization position determination process.
FIG. 5 is a diagram showing an example of setting parameters.
FIG. 6 is a diagram showing a display example of a POV image and an overhead view image.
FIG. 7 is a diagram illustrating adjustment of the arrangement position of the localization position mark.
[Fig. 8] Fig. 8 is a diagram for explaining adjustment of the arrangement position of the localization position mark.
FIG. 9 is a diagram showing a display example of a speaker.
FIG. 10 is a diagram illustrating interpolation of position information.
FIG. 11 is a flowchart illustrating a localization position determination process.
FIG. 12 is a diagram showing a configuration example of a computer.
MODE FOR CARRYING OUT THE INVENTION
[0022]
　Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.
[0023]

The
　present technology is a GUI that simulates a listening space in which content is reproduced by a point of view shot (hereinafter simply referred to as POV) from a listening position ( By specifying the localization position of the sound image on the Graphical User Interface), it is possible to easily determine the localization position of the sound image.
[0024]
　This makes it possible to realize a user interface that allows the localization position of the sound to be easily determined in, for example, an audio content production tool. Especially in the case of object-based audio, it becomes possible to realize a user interface capable of easily determining position information of an audio object.
[0025]
　First, a case will be described in which the content is a video that is a still image or a moving image, and content that includes two left and right channel sounds accompanying the video.
[0026]
　In this case, for example, in content production, it is possible to easily determine the sound localization in accordance with the video by using a visual and intuitive user interface.
[0027]
　Here, as a specific example, it is assumed that there are audio data tracks of contents, that is, audio data tracks of a total of four musical instruments of a drum, an electric guitar, and two acoustic guitars as audio tracks. Also, it is assumed that, as the image of the content, there is an image in which those musical instruments and the performers of the musical instruments are shown as subjects.
[0028]
　Furthermore, the left channel speaker is in the direction in which the horizontal angle is 30 degrees from the listening position of the sound of the content by the listener, and the right channel speaker is in the direction in which the horizontal angle is -30 degrees from the listening position. Suppose
[0029]
　The horizontal angle referred to here is an angle indicating the horizontal position, that is, the horizontal position as seen by the listener at the listening position. For example, the horizontal angle indicating the position in the front direction of the listener in the horizontal direction is 0 degree. Further, it is assumed that the horizontal angle indicating the position to the left of the listener is a positive angle, and the horizontal angle indicating the position to the right of the listener is a negative angle.
[0030]
　Now, let us consider determining the localization position of the sound image of the sound of the content for the output of the left and right channels.
[0031]
　In such a case, in the present technology, for example, the edited image P11 shown in FIG. 1 is displayed on the display screen of the content production tool.
[0032]
　The edited image P11 is an image (video) viewed by the listener while listening to the sound of the content, and for example, an image including the video of the content is displayed as the edited image P11.
[0033]
　In this example, in the edited image P11, the player of the musical instrument is displayed as a subject on the video of the content.
[0034]
　That is, here, the edited image P11 shows the drum player PL11, the electric guitar player PL12, the first acoustic guitar player PL13, and the second acoustic guitar player PL14. Has been done.
[0035]
　The edited image P11 also displays musical instruments such as drums, electric guitars, and acoustic guitars used in the performances by the performers PL11 to PL14. It can be said that these musical instruments are audio objects that are sound sources based on audio tracks.
[0036]
　In the following, when distinguishing two acoustic guitars, the one used by the performer PL13 is also referred to as the acoustic guitar 1, and the one used by the performer PL14 is also referred to as the acoustic guitar 2.
[0037]
　Such an edited image P11 also functions as a user interface, that is, as an input interface. Is also displayed.
[0038]
　Here, each of the localization position marks MK11 to MK14 indicates the respective sound image localization positions of the sounds of the audio tracks of the drum, the electric guitar, the acoustic guitar 1, and the acoustic guitar 2.
[0039]
　In particular, the localization position mark MK12 of the audio track of the electric guitar selected for the localization position adjustment is highlighted, and it is displayed in a different format from the localization position marks of other audio tracks that are not selected. It is displayed.
[0040]
　The content creator moves the localization position mark MK12 of the selected audio track to an arbitrary position on the edited image P11 so that the sound image of the sound of the audio track is localized at the position of the localization position mark MK12. can do. In other words, an arbitrary position on the video of the content, that is, on the listening space can be designated as the localization position of the sound image of the sound of the audio track.
[0041]
　In this example, the localization positions marks MK11 to MK14 of the sounds of the audio tracks corresponding to those instruments are arranged at the positions of the musical instruments of the performers PL11 to PL14, and the sound image of the sounds of each instrument is recorded by the performer. It is designed to localize to the position of the instrument.
[0042]
　In the content production tool, when the localization position for the sound of each audio track is specified by specifying the display position of the localization position mark, the left and right sides of the audio track (audio data) are displayed based on the display position of the localization position mark. The gain value of each channel is calculated.
[0043]
　That is, based on the coordinates indicating the position of the localization position mark on the edited image P11, the proportional division ratio to the left and right channels of the audio track is determined, and the gain value of each of the left and right channels is obtained from the determination result. In this case, since the left and right channels are proportionally distributed, only the horizontal direction (horizontal direction) on the edited image P11 is considered, and the position of the localization position mark in the vertical direction is not considered.
[0044]
　Specifically, for example, as shown in FIG. 2, the gain value is obtained based on the horizontal angle indicating the horizontal position of each localization position mark viewed from the listening position. In FIG. 2, parts corresponding to those in FIG. 1 are designated by the same reference numerals, and the description thereof will be omitted as appropriate. Further, in FIG. 2, the localization position mark is not shown in order to make the diagram easy to see.
[0045]
　In this example, the position in front of the listening position O is the edited image P11, that is, the center position O'of the screen on which the edited image P11 is displayed, and the length in the left-right direction of the screen, that is, the left and right of the edited image P11. The image width in the direction is L.
[0046]
　Further, the positions of the performers PL11 to PL14 on the edited image P11, that is, the positions of the musical instruments used by the performers to perform are positions PJ1 to PJ4. Particularly, in this example, since the localization position marks are arranged at the positions of the musical instruments of the performers, the positions of the localization position marks MK11 to MK14 are positions PJ1 to PJ4.
[0047]
　Further, in the figure on the screen on which the edited image P11 is displayed, the position of the left end is position PJ5, and on the screen, the position of the right end is position PJ6. These positions PJ5 and PJ6 are also positions where the left and right speakers are arranged.
[0048]
　Now, in the figure, it is assumed that the coordinates indicating the positions PJ1 to PJ4 viewed from the center position O′ in the left-right direction are X 1 to X 4 . In particular, here, it is assumed that the direction of the position PJ5 is positive when viewed from the center position O′, and the direction of the position PJ6 is negative when viewed from the center position O′.
[0049]
　Therefore, for example, the distance from the center position O′ to the position PJ1 becomes the coordinate X 1 indicating the position PJ1 .
[0050]
　Further, it is assumed that the angles indicating the horizontal positions of the positions PJ1 to PJ4 viewed from the listening position O, that is, the horizontal positions in the drawing are horizontal angles θ 1 to θ 4 .
[0051]
　For example, the horizontal angle θ 1 is an angle formed by a straight line connecting the listening position O and the central position O′ and a straight line connecting the listening position O and the position PJ1. In particular, here, when viewed from the listening position O, the left direction is the direction of the positive horizontal angle, and when viewed from the listening position O, the right direction is the direction of the negative negative horizontal angle. To do.
[0052]
　As described above, the horizontal angle indicating the position of the left channel speaker is 30 degrees, and the horizontal angle indicating the position of the right channel speaker is -30 degrees, so the horizontal angle of position PJ5 is 30 degrees. , The horizontal angle of position PJ6 is -30 degrees.
[0053]
　Since the left and right channel speakers are arranged at the left and right ends of the screen, the viewing angle of the edited image P11, that is, the viewing angle of the video of the content is ±30 degrees.
[0054]
　In such a case, the proportional distribution ratio of each audio track (audio data), that is, the gain value of each of the left and right channels is determined by the horizontal angle of the localization position of the sound image when viewed from the listening position O.
[0055]
　For example, the horizontal angle θ 1 indicating the position PJ1 on the audio track of the drum can be calculated from the coordinate X 1 indicating the position PJ1 viewed from the center position O′ and the image width L by the calculation shown in the following formula (1). it can.
[0056]
[Number 1]

[0057]
　Therefore, the horizontal angle theta 1 gain value GainL of the left and right channels for a position PJ1 localizing a sound image of the sound based on the audio data of the drum (audio track) indicated by 1 and the gain value GainR 1 has the following formula (2 ) And equation (3). The gain value GainL 1 is the gain value of the left channel, and the gain value GainR 1 is the gain value of the right channel.
[0058]
[Number 2]

[0059]
[Number 3]

[0060]
　When reproducing the content, the gain value GainL 1 is multiplied by the audio data of the drum, and the sound is output from the speaker of the left channel based on the audio data obtained as a result. Further, the gain value GainR 1 is multiplied by the audio data of the drum, and the sound is output from the speaker of the right channel based on the audio data obtained as a result.
[0061]
　Then, the sound image of the sound of the drum is localized at the position PJ1, that is, the position of the drum (player PL11) in the video of the content.
[0062]
　Not only the drum audio track but also other electric guitars, acoustic guitars 1, and acoustic guitars 2 are calculated in the same manner as the above equations (1) to (3), and the gain values ​​of the left and right channels are calculated. It is calculated.
[0063]
　That is, the coordinate X 2 on the basis of the image width L, the gain value GainL of the left and right channels of the audio data of the electric guitar 2 and the gain value GainR 2 is obtained.
[0064]
　Further, the gain value GainL 3 and the gain value GainR 3 of the left and right channels of the audio data of the acoustic guitar 1 are obtained based on the coordinate X 3 and the image width L, and the acoustic guitar is obtained based on the coordinate X 4 and the image width L. gain value of the left and right channels of the second audio data GainL 4 and the gain value GainR 4 is obtained.
[0065]
　If it is assumed that the left and right channel speakers are located outside the edge of the screen, that is, if the distance L spk between the left and right speakers is greater than the image width L, then in equation (1), The calculation may be performed by replacing the image width L with the distance L spk .
[0066]
　As described above, in the left and right two-channel content production, the sound image localization position of the sound matched with the image of the content can be easily determined by the intuitive user interface.
[0067]

　Next, a signal processing device to which the present technology described above is applied will be described.
[0068]
　FIG. 3 is a diagram showing a configuration example of an embodiment of a signal processing device to which the present technology is applied.
[0069]
　The signal processing device 11 shown in FIG. 3 has an input unit 21, a recording unit 22, a control unit 23, a display unit 24, a communication unit 25, and a speaker unit 26.
[0070]
　The input unit 21 includes switches and buttons, a mouse, a keyboard, a touch panel provided so as to be superimposed on the display unit 24, and supplies a signal according to an input operation of a user who is a content creator to the control unit 23.
[0071]
　The recording unit 22 is composed of, for example, a non-volatile memory such as a hard disk, and records the audio data and the like supplied from the control unit 23 and supplies the recorded data to the control unit 23. The recording unit 22 may be a removable recording medium that can be attached to and detached from the signal processing device 11.
[0072]
　The control unit 23 controls the overall operation of the signal processing device 11. The control unit 23 has a localization position determination unit 41, a gain calculation unit 42, and a display control unit 43.
[0073]
　The localization position determination unit 41 determines the localization position of each audio track, that is, the sound image of the sound of each audio data, based on the signal supplied from the input unit 21.
[0074]
　In other words, the localization position determination unit 41 acquires information regarding the localization position of the sound image of the sound of the audio object such as a musical instrument viewed from the listening position in the listening space displayed on the display unit 24, and determines the localization position. It can be said that it functions as an acquisition unit.
[0075]
　Here, the information regarding the localization position of the sound image is, for example, position information indicating the localization position of the sound image of the sound of the audio object viewed from the listening position, information for obtaining the position information, and the like.
[0076]
　The gain calculator 42 calculates the gain value of each channel for audio data for each audio object, that is, for each audio track, based on the localization position determined by the localization position determination unit 41. The display control unit 43 controls the display unit 24 to control the display of images and the like on the display unit 24.
[0077]
　The control unit 23 also generates an output bitstream including at least audio data of the content based on the information regarding the localization position acquired by the localization position determination unit 41 and the gain value calculated by the gain calculation unit 42. It also functions as a generator that outputs.
[0078]
　The display unit 24 is composed of, for example, a liquid crystal display panel or the like, and displays various images such as POV images under the control of the display control unit 43.
[0079]
　The communication unit 25 communicates with an external device via a wired or wireless communication network such as the Internet. For example, the communication unit 25 receives data transmitted from an external device and supplies the data to the control unit 23, or transmits the data supplied from the control unit 23 to the external device.
[0080]
　The speaker unit 26 is, for example, a speaker of each channel of a speaker system having a predetermined channel configuration, and reproduces (outputs) the sound of the content based on the audio data supplied from the control unit 23.
[0081]

　Next, the operation of the signal processing device 11 will be described.
[0082]
　That is, hereinafter, the localization position determination process performed by the signal processing device 11 will be described with reference to the flowchart of FIG.
[0083]
　In step S11, the display control unit 43 causes the display unit 24 to display the edited image.
[0084]
　For example, when a signal for instructing the activation of the content production tool is supplied from the input unit 21 to the control unit 23 in response to an operation by the content producer, the control unit 23 activates the content production tool. At this time, the control unit 23 reads the image data of the video of the content specified by the content creator and the audio data accompanying the video from the recording unit 22 as necessary.
[0085]
　Then, the display control unit 43 supplies image data for displaying the display screen (window) of the content production tool including the edited image to the display unit 24 in response to the activation of the content production tool, and displays the display screen. .. Here, the edited image is, for example, an image in which a localization position mark indicating a sound image localization position of a sound based on each audio track is superimposed on the video of the content.
[0086]
　The display unit 24 displays the display screen of the content production tool based on the image data supplied from the display control unit 43. Thereby, for example, a screen including the edited image P11 shown in FIG. 1 is displayed on the display unit 24 as a display screen of the content production tool.
[0087]
　When the display screen of the content production tool including the edited image is displayed, the content creator operates the input unit 21 to adjust the localization position of the sound image from the audio tracks (audio data) of the content. Select. Then, the input unit 21 supplies the control unit 23 with a signal according to the selection operation of the content creator.
[0088]
　The selection of the audio track may be performed by designating the desired audio track at the desired reproduction time on the timeline of the audio track displayed separately from the edited image on the display screen, or displayed. The localization position mark may be directly specified.
[0089]
　In step S12, the localization position determination unit 41 selects an audio track for which the localization position of the sound image is adjusted based on the signal supplied from the input unit 21.
[0090]
　When the localization position determining unit 41 selects an audio track for which the localization position of the sound image is to be adjusted, the display control unit 43 controls the display unit 24 according to the selection result and corresponds to the selected audio track. The localization position mark is displayed in a display format different from that of other localization position marks.
[0091]
　When the localization position mark corresponding to the selected audio track is displayed in a display format different from the other localization position marks, the content creator operates the input unit 21 to move the target localization position mark to an arbitrary position. By doing so, the localization position of the sound image is specified.
[0092]
　For example, in the example shown in FIG. 1, the content creator specifies the sound image localization position of the sound of the electric guitar by moving the position of the localization position mark MK12 to an arbitrary position.
[0093]
　Then, a signal according to the input operation of the content creator is supplied from the input unit 21 to the control unit 23, so that the display control unit 43 controls the display unit 24 according to the signal supplied from the input unit 21. , Move the display position of the localization position mark.
[0094]
　Further, in step S13, the localization position determining unit 41 determines the localization position of the sound image of the sound of the audio track to be adjusted, based on the signal supplied from the input unit 21.
[0095]
　That is, the localization position determination unit 41 acquires, from the input unit 21, information (signal) indicating the position of the localization position mark in the edited image, which is output according to the input operation of the content creator. Then, the localization position determination unit 41 determines, as the localization position of the sound image, the position indicated by the target localization position mark on the edited image, that is, on the video of the content, based on the acquired information.
[0096]
　In addition, the localization position determination unit 41 generates position information indicating the localization position according to the determination of the localization position of the sound image.
[0097]
　For example, in the example shown in FIG. 2, it is assumed that the localization position mark MK12 is moved to the position PJ2. In such a case, the localization position determination unit 41 performs the same calculation as in the above-described formula (1) based on the acquired coordinate X 2 to obtain position information indicating the localization position of the sound image of the audio track of the electric guitar, In other words, the horizontal angle θ 2 is calculated as position information indicating the position of the performer PL12 (electric guitar) as an audio object .
[0098]
　In step S14, the gain calculator 42 calculates the gain values ​​of the left and right channels of the audio track selected in step S12 based on the horizontal angle as the position information obtained as the result of the localization position determination in step S13. ..
[0099]
　For example, in step S14, the same calculation as the above-described equations (2) and (3) is performed to calculate the gain values ​​of the left and right channels.
[0100]
　In step S15, the control unit 23 determines whether to end the adjustment of the localization position of the sound image. For example, when the content creator operates the input unit 21 to instruct the output of the content, that is, the end of the content production, it is determined in step S15 that the adjustment of the localization position of the sound image is completed.
[0101]
　When it is determined in step S15 that the adjustment of the localization position of the sound image is not yet finished, the process returns to step S12, and the above-described process is repeated. That is, the localization position of the sound image is adjusted for the newly selected audio track.
[0102]
　On the other hand, when it is determined in step S15 that the adjustment of the localization position of the sound image is to be ended, the process proceeds to step S16.
[0103]
　In step S16, the control unit 23 outputs the output bitstream based on the position information of each object, in other words, the output bitstream based on the gain value obtained in the process of step S14, and the localization position determination process ends.
[0104]
　For example, in step S16, the control unit 23 multiplies the audio data by the gain value obtained in the process of step S14 to generate audio data of left and right channels for each audio track of the content. In addition, the control unit 23 adds the obtained audio data of the same channel to obtain the final audio data of each of the left and right channels, and outputs an output bit stream including the audio data thus obtained. Here, the output bit stream may include image data of video of the content.
[0105]
　Further, the output destination of the output bitstream can be any output destination such as the recording unit 22, the speaker unit 26, or an external device.
[0106]
　For example, an output bitstream including audio data and image data of the content may be supplied to and recorded in the recording unit 22, a removable recording medium, or the like, or audio data as an output bitstream may be supplied to the speaker unit 26 to output the content. The sound may be played. Further, for example, an output bitstream including audio data and image data of content may be supplied to the communication unit 25, and the communication unit 25 may transmit the output bitstream to an external device.
[0107]
　At this time, for example, the audio data and the image data of the content included in the output bitstream may or may not be encoded by a predetermined encoding method. Further, of course, an output bit stream including each audio track (audio data), the gain value obtained in step S14, and the image data of the video of the content may be generated.
[0108]
　As described above, the signal processing device 11 displays the edited image, moves the localization position mark according to the operation of the user (content creator), and displays the position indicated by the localization position mark, that is, the localization position mark. The localization position of the sound image is determined based on the display position.
[0109]
　By doing so, the content creator can easily determine (specify) the appropriate localization position of the sound image by simply moving the localization position mark to the desired position while viewing the edited image. it can.
[0110]

　By the way, in the first embodiment, an example in which the audio (sound) of the content is output from left and right two channels has been described. However, the present technology is not limited to this, and is also applicable to object-based audio that localizes a sound image at an arbitrary position in a three-dimensional space.
[0111]
　Hereinafter, a case will be described where the present technology is applied to object-based audio (hereinafter, simply referred to as object-based audio) that targets sound image localization in a three-dimensional space.
[0112]
　Here, it is assumed that the sound of the audio object is included as the sound of the content, and the audio object includes the drum, the electric guitar, the acoustic guitar 1, and the acoustic guitar 2 as in the above-described example. Further, it is assumed that the content includes audio data of each audio object and video image data corresponding to the audio data. The video of the content may be a still image or a moving image.
[0113]
　With object-based audio, sound images can be localized in all directions in a three-dimensional space, so it is assumed that sound images will be localized at a position outside a certain range of the image, that is, a position that cannot be seen in the image even when accompanied by the image. To be done. In other words, it is difficult to accurately determine the sound image localization position according to the image because of the high degree of freedom in the localization of the sound image. It is necessary to specify the localization position.
[0114]
　Therefore, in the present technology, for object-based audio content, the content production tool first sets the content reproduction environment.
[0115]
　Here, the reproduction environment is, for example, a three-dimensional space such as a room in which the content is reproduced, that is, a listening space, which is assumed by the content creator. When setting the playback environment, the size of the room (listening space), the listening position that is the position of the viewer who views the content, that is, the listener of the sound of the content, the shape of the screen on which the video of the content is displayed, and the layout of the screen The position is specified by parameters.
[0116]
　For example, the parameter shown in FIG. 5 is specified by the content creator as a parameter (hereinafter, also referred to as a setting parameter) for specifying the reproduction environment, which is specified when the reproduction environment is set.
[0117]
　In the example shown in FIG. 5, “depth”, “width”, and “height” that determine the size of the room, which is the listening space, are shown as setting parameters. Here, the depth of the room is “6.0 m”. The width of the room is "8.0m" and the height of the room is "3.0m".
[0118]
　In addition, a "listening position", which is the position of the listener in the room (listening space), is shown as a setting parameter, and the listening position is the "center of the room".
[0119]
　Further, the "size" and the "aspect ratio" that determine the shape of the screen (display device) on which the image of the content is displayed in the room (listening space), that is, the shape of the display screen are shown as setting parameters.
[0120]
　The setting parameter “size” indicates the size of the screen, and “aspect ratio” indicates the aspect ratio of the screen (display screen). Here, the size of the screen is "120 inches" and the aspect ratio of the screen is "16:9".
[0121]
　In addition, in FIG. 5, “front/rear”, “left/right”, and “up/down” that determine the position of the screen are shown as setting parameters related to the screen.
[0122]
　Here, the setting parameter “front and back” is the distance in the front-back direction from the listener to the screen when the listener at the listening position in the listening space (room) sees the reference direction, and in this example, The value of the setting parameter “front and back” is set to “2 m in front of the listening position”. That is, the screen is placed 2 m in front of the listener.
[0123]
　Further, the setting parameter "left and right" is the position in the left and right direction of the screen as seen by the listener who is facing the reference direction at the listening position in the listening space (room). The setting (value) is "center". That is, the screen is arranged so that the position of the center of the screen in the left-right direction is directly in front of the listener.
[0124]
　The setting parameter “vertical” is the vertical position of the screen as seen by the listener who is facing the reference direction in the listening position in the listening space (room). In this example, the setting parameter “vertical” is set ( The value) is "the center of the screen is the height of the listener's ears." That is, the screen is arranged so that the vertical position of the center of the screen is at the height of the listener's ears.
[0125]
　In the content production tool, the POV image and the like are displayed on the display screen according to the above setting parameters. That is, a POV image simulating a listening space is displayed in 3D graphic on the display screen by setting parameters.
[0126]
　For example, when the setting parameters shown in FIG. 5 are designated, the screen shown in FIG. 6 is displayed as the display screen of the content production tool. Note that, in FIG. 6, portions corresponding to those in FIG. 1 are denoted by the same reference numerals, and description thereof will be appropriately omitted.
[0127]
　In FIG. 6, a window WD11 is displayed as the display screen of the content production tool. In this window WD11, a POV image P21, which is an image of the listening space as seen from the viewpoint of the listener, and the listening space are viewed overhead. An overhead view image P22, which is an image, is displayed.
[0128]
　In the POV image P21, the wall of the room, which is the listening space, is viewed from the listening position, and the screen SC11 on which the image of the content is superimposed is displayed at the position in front of the listener in the room. . In the POV image P21, the listening space seen from the actual listening position is reproduced almost as it is.
[0129]
　In particular, this screen SC11 is a screen having an aspect ratio of 16:9 and a size of 120 inches as specified by the setting parameters in FIG. Further, the screen SC11 is arranged at a position in the listening space defined by the setting parameters "front and rear", "right and left", and "up and down" shown in FIG.
[0130]
　On the screen SC11, performers PL11 to PL14, which are subjects in the video of the content, are displayed.
[0131]
　The POV image P21 also displays localization position marks MK11 to MK14, and in this example, these localization position marks are located on the screen SC11.
[0132]
　Note that, in FIG. 6, an example in which the POV image P21 is displayed when the line-of-sight direction of the listener is a predetermined reference direction, that is, the front direction of the listening space (hereinafter, also referred to as a reference direction) Showing. However, the content creator can change the line-of-sight direction of the listener to an arbitrary direction by operating the input unit 21. When the listener's line-of-sight direction is changed, an image of the listening space in the changed line-of-sight direction is displayed in the window WD11 as a POV image.
[0133]
　Further, more specifically, the viewpoint position of the POV image can be not only the listening position but also a position near the listening position. For example, when the viewpoint position of the POV image is set to a position near the listening position, the listening position is always displayed on the front side of the POV image.
[0134]
　As a result, even if the viewpoint position is different from the listening position, the content creator who is viewing the POV image can easily understand which position the displayed POV image is the viewpoint position. can do.
[0135]
　On the other hand, the bird's-eye view image P22 is an image of the entire room, which is the listening space, that is, an image of the bird's-eye view of the listening space.
[0136]
　In particular, in the drawing of the listening space, the length in the direction indicated by the arrow RZ11 is the length of the depth of the listening space indicated by the setting parameter "depth" shown in FIG. Similarly, the length of the listening space in the direction indicated by the arrow RZ12 is the width of the listening space indicated by the setting parameter “width” shown in FIG. 5, and the direction indicated by the arrow RZ13 in the listening space. Is the height of the listening space indicated by the setting parameter "height" shown in FIG.
[0137]
　Furthermore, the point O displayed on the bird's-eye view image P22 indicates the position indicated by the setting parameter “listening position” shown in FIG. 5, that is, the listening position. Hereinafter, the point O will be particularly referred to as a listening position O.
[0138]
　As described above, by displaying the image of the entire listening space in which the listening position O, the screen SC11, and the localization position mark MK11 to the localization position mark MK14 are displayed as the bird's-eye view image P22, the content creator can listen to the listening position O and the screen SC11. , It is possible to properly grasp the positional relationship between the performer and the musical instrument (audio object).
[0139]
　The content creator operates the input unit 21 while looking at the POV image P21 and the bird's-eye view image P22 thus displayed, and moves the localization position marks MK11 to MK14 for each audio track to desired positions. By doing so, the localization position of the sound image is specified.
[0140]
　By doing so, as in the case of FIG. 1, the content creator can easily determine (designate) an appropriate localization position of the sound image.
[0141]
　The POV image P21 and the bird's-eye view image P22 shown in FIG. 6 also function as an input interface similarly to the case of the edited image P11 shown in FIG. 1, and designate any position of the POV image P21 and the bird's-eye view image P22. Thus, the sound image localization position of the sound of each audio track can be designated.
[0142]
　For example, when the content creator operates the input unit 21 or the like to specify a desired position on the POV image P21, a localization position mark is displayed at that position.
[0143]
　In the example shown in FIG. 6, as in the case of FIG. 1, the localization position marks MK11 to MK14 are displayed at positions on the screen SC11, that is, positions on the image of the content. Therefore, it can be seen that the sound image of the sound of each audio track is localized at the position of each subject (audio object) in the video corresponding to the sound. That is, it can be seen that sound image localization that matches the video of the content is realized.
[0144]
　In the signal processing device 11, for example, the position of the localization position mark is managed by the coordinates of the coordinate system with the listening position O as the origin (reference).
[0145]
　For example, if the coordinate system with the listening position O as the origin is polar coordinates, the position of the localization position mark is the horizontal direction viewed from the listening position O, that is, the horizontal angle indicating the horizontal position and the vertical angle viewed from the listening position O. Direction, that is, a vertical angle indicating a vertical position, and a radius indicating a distance from the listening position O to the localization position mark.
[0146]
　In the following, the position of the localization position mark will be described as being represented by a horizontal angle, a vertical angle, and a radius, that is, represented by polar coordinates. May be represented by coordinates such as a three-dimensional orthogonal coordinate system.
[0147]
　When the localization position mark is thus represented by polar coordinates, the display position of the localization position mark in the listening space can be adjusted as follows, for example.
[0148]
　That is, when the content creator operates the input unit 21 or the like to specify a desired position on the POV image P21 by clicking or the like, the localization position mark is displayed at that position. Specifically, for example, a localization position mark is displayed at a position designated by the content creator on a spherical surface having a radius of 1 centered on the listening position O.
[0149]
　At this time, for example, as shown in FIG. 7, a straight line L11 extending from the listening position O in the direction of the line of sight of the listener is displayed, and the localization position mark MK11 to be processed is displayed on the straight line L11. Note that, in FIG. 7, portions corresponding to those in FIG. 6 are denoted by the same reference numerals, and description thereof will be appropriately omitted.
[0150]
　In the example shown in FIG. 7, the localization position mark MK11 corresponding to the audio track of the drum is the processing target, that is, the localization position of the sound image is adjusted, and the localization position mark MK11 extends in the line of sight of the listener. It is displayed above.
[0151]
　The content creator can move the localization position mark MK11 to an arbitrary position on the straight line L11, for example, by performing a wheel operation or the like on the mouse as the input unit 21. In other words, the content creator can adjust the distance from the listening position O to the localization position mark MK11, that is, the radius of polar coordinates indicating the position of the localization position mark MK11.
[0152]
　The content creator can also adjust the direction of the straight line L11 to an arbitrary direction by operating the input unit 21.
[0153]
　By such an operation, the content creator can move the localization position mark MK11 to an arbitrary position in the listening space.
[0154]
　Therefore, for example, the content creator sets the position of the localization position mark closer to the viewer than the position of the screen of the content corresponding to the audio object, that is, the position of the screen SC11 corresponding to the audio object. Can also be moved.
[0155]
　For example, in the example shown in FIG. 7, the localization position mark MK11 of the audio track of the drum is located on the far side of the screen SC11 when viewed from the listener, and the localization position mark MK12 of the audio track of the electric guitar is detected by the listener. It is located on the front side of the screen SC11 when viewed.
[0156]
　The localization position mark MK13 of the audio track of the acoustic guitar 1 and the localization position mark MK14 of the audio track of the acoustic guitar 2 are located on the screen SC11.
[0157]
　As described above, in the content production tool to which the present technology is applied, the sound image is localized at an arbitrary position in the depth direction, such as the front side or the back side of the listener with respect to the position of the screen SC11 as a reference. You can control the sense of distance.
[0158]
　For example, in object-based audio, position coordinates in polar coordinates with the listener's position (listening position) as the origin are treated as meta information of the audio object.
[0159]
　In the example described with reference to FIGS. 6 and 7, it can be said that each audio track is audio data of an audio object and each localization position mark is the position of the audio object. Therefore, the position information indicating the position of the localization position mark can be used as the position information as the meta information of the audio object.
[0160]
　Then, when the content is played, if the audio object (audio track) is rendered based on the position information that is the meta information of the audio object, the audio is moved to the position indicated by the position information, that is, the position indicated by the localization position mark. The sound image of the sound of the object can be localized.
[0161]
　In rendering, for example, a VBAP method is used to calculate a gain value proportionally distributed to each speaker channel of the speaker system used for reproduction based on the position information. That is, the gain calculator 42 calculates the gain value of each channel of the audio data.
[0162]
　Then, the audio data multiplied by each of the calculated gain values ​​of the respective channels is set as the audio data of those channels. When there are a plurality of audio objects, the audio data of the same channel obtained for those audio objects are added together to obtain the final audio data.
[0163]
　The sound image of the sound of the audio object is localized at the position indicated as the meta information, that is, the position indicated by the localization position mark, by the speaker outputting the sound based on the audio data of each channel thus obtained. Like
[0164]
　Therefore, particularly when the position on the screen SC11 is designated as the position of the localization position mark, the sound image is localized at the position on the video of the content when the content is actually reproduced.
[0165]
　As shown in FIG. 7, as the position of the localization position mark, any position such as a position different from the position on the screen SC11 can be designated. Therefore, the radius indicating the distance from the listener to the audio object, which constitutes the position information as the meta information, can be used as information for controlling the sense of distance when reproducing the sound of the content.
[0166]
　For example, when reproducing the content in the signal processing device 11, it is assumed that the radius included in the position information as the meta information of the audio data of the drum is twice the reference value (for example, 1).
[0167]
　In such a case, for example, if the control unit 23 multiplies the audio data of the drum by the gain value “0.5” and adjusts the gain, the sound of the drum becomes small, and the sound of the drum becomes a reference distance. It is possible to realize a sense of distance control that makes the player feel as if he or she is listening from a position farther than the position.
[0168]
　Note that the sense of distance control by gain adjustment is merely an example of the sense of distance control using the radius included in the position information, and the sense of distance control may be realized by any other method. By performing such sense of distance control, for example, the sound image of the sound of the audio object can be localized at a desired position such as the front side or the back side of the reproduction screen.
[0169]
　In addition, for example, in the MPEG (Moving Picture Experts Group)-H 3D Audio standard, the reproduction screen size on the content production side can be sent as meta information to the user side, that is, the content reproduction side.
[0170]
　In this case, when the position and size of the playback screen on the content production side are different from those on the playback screen on the content playback side, the position information of the audio object is corrected on the content playback side to generate the sound image of the sound of the audio object. It can be positioned at an appropriate position on the playback screen. Therefore, also in the present technology, for example, the setting parameter indicating the position, size, arrangement position, etc. of the screen shown in FIG. 5 may be used as the meta information of the audio object.
[0171]
　Further, in the description given with reference to FIG. 7, an example has been described in which the position of the localization position mark is set to the position on the front side or the back side of the screen SC11 in front of the listener, or the position on the screen SC11. However, the position of the localization position mark is not limited to the front of the listener, but may be any position outside the screen SC11 such as the side, the rear, the upper, and the lower of the listener.
[0172]
　For example, if the position of the localization position mark is set to a position outside the frame of the screen SC11 when viewed from the listener, when the content is actually played, the sound image of the sound of the audio object is outside the range where the image of the content exists. It comes to localize to the position.
[0173]
　Further, the case where the screen SC11 on which the image of the content is displayed is in the reference direction when viewed from the listening position O has been described as an example. However, the screen SC11 is not limited to the reference direction, and may be arranged in any direction such as backward, upward, downward, left side, or right side when viewed from the listener looking at the reference direction, and within the listening space. Multiple screens may be arranged in the.
[0174]
　As described above, the content production tool can change the line-of-sight direction of the POV image P21 to an arbitrary direction. In other words, the listener can look around the listening position O as a center.
[0175]
　Therefore, the content creator operates the input unit 21 to designate any direction such as the side or the back when the reference direction is the front direction as the line-of-sight direction of the POV image P21, and the arbitrary position in each direction. A localization position mark can be placed on.
[0176]
　Therefore, for example, as shown in FIG. 8, it is possible to change the line-of-sight direction of the POV image P21 to a direction outside the right end of the screen SC11, and to arrange a localization position mark MK21 of a new audio track in that direction. .. Note that, in FIG. 8, portions corresponding to those in FIG. 6 or 7 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.
[0177]
　In the example of FIG. 8, vocal audio data as an audio object is added as a new audio track, and a localization position mark MK21 indicating a sound image localization position of a sound based on the added audio track is displayed. ..
[0178]
　Here, the localization position mark MK21 is arranged at a position outside the screen SC11 when viewed from the listener. Therefore, when the content is reproduced, the listener perceives the vocal sound as if it were heard from a position that cannot be seen in the image of the content.
[0179]
　In addition, when it is assumed that the screen SC11 is arranged at a side or a rear position as seen from the listener who is looking at the reference direction, the screen SC11 is arranged at a side or a rear position thereof, The POV image in which the video of the content is displayed is displayed on the screen SC11. In this case, if each localization position mark is arranged on the screen SC11, the sound image of the sound of each audio object (musical instrument) is localized at the position of the video when the content is reproduced.
[0180]
　As described above, in the content production tool, it is possible to easily realize the sound image localization in accordance with the image of the content simply by disposing the localization position mark on the screen SC11.
[0181]
　Further, as shown in FIG. 9, the layout display of the speakers used for reproducing the content may be displayed on the POV image P21 and the bird's eye view image P22. Note that in FIG. 9, portions corresponding to those in FIG. 6 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.
[0182]
　In the example shown in FIG. 9, on the POV image P21, a plurality of speakers including the front left speaker SP11 of the listener, the front right speaker SP12 of the listener, and the front upper speaker SP13 of the listener are displayed. .. Similarly, a plurality of speakers including the speakers SP11 to SP13 are displayed on the overhead view image P22.
[0183]
　These speakers are the speakers of each channel that make up the speaker system used when reproducing the content, which is assumed by the content creator.
[0184]
　The content creator operates the input unit 21 to specify the channel configuration of the speaker system, such as 7.1 channel or 22.2 channel, so that each speaker of the speaker system having the specified channel configuration is displayed on the POV image P21 and the bird's eye view image. It can be displayed on P22. That is, the speaker layout having the designated channel configuration can be displayed in a superimposed manner in the listening space.
[0185]
　Object-based audio can support various speaker layouts by performing rendering based on the position information of each audio object using the VBAP method.
[0186]
　In the content production tool, by displaying the speakers on the POV image P21 and the bird's eye view image P22, the content creator can display the speaker, the localization position mark, that is, the audio object, and the display position of the image of the content, that is, the screen SC11. , The positional relationship with the listening position O can be easily grasped visually.
[0187]
　Therefore, the content creator uses the speaker displayed in the POV image P21 or the bird's-eye view image P22 as auxiliary information when adjusting the position of the audio object, that is, the position of the localization position mark, and the localization position at a more appropriate position. Marks can be placed.
[0188]
　For example, when a content creator creates commercial content, the content creator often uses a speaker layout in which speakers such as 22.2 channels are densely arranged as a reference. In this case, for example, the content creator may select 22.2 channels as the channel configuration and display the speaker of each channel on the POV image P21 or the bird's eye view image P22.
[0189]
　On the other hand, for example, when the content creator is a general user, the content creator often uses a speaker layout in which speakers are roughly arranged, such as 7.1 channel. In this case, for example, the content creator may select 7.1 channel as the channel configuration and display the speaker of each channel on the POV image P21 or the bird's eye view image P22.
[0190]
　When a speaker layout in which speakers are roughly arranged, such as 7.1 channels, is used, depending on the position where the sound image of the sound of the audio object is localized, there is no speaker near that position, and the localization of the sound image is blurred. There is. In order to clearly localize the sound image, the localization position mark position is preferably arranged near the speaker.
[0191]
　As described above, in the content creation tool, it is possible to select an arbitrary channel configuration of the speaker system and display each speaker of the speaker system of the selected channel configuration in the POV image P21 or the bird's eye view image P22. There is.
[0192]
　Therefore, the content creator uses the speaker displayed in the POV image P21 or the bird's-eye view image P22 according to the speaker layout that he or she envisions as auxiliary information, and sets the localization position mark at a more appropriate position such as a position near the speaker. You will be able to arrange. That is, the content creator can visually understand the influence of the speaker layout on the sound image localization of the audio object, and appropriately adjust the placement position of the localization position mark while considering the positional relationship with the image and the speaker. ..
[0193]
　Furthermore, the content production tool can specify a localization position mark for each audio track at each reproduction time of the audio track (audio data).
[0194]
　For example, as shown in FIG. 10, it is assumed that the position of the localization position mark MK12 has changed in accordance with the movement of the player PL12 of the electric guitar between the predetermined reproduction time t1 and the subsequent reproduction time t2. Note that in FIG. 10, portions corresponding to those in FIG. 6 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.
[0195]
　In FIG. 10, the player PL12' and the localization position mark MK12' represent the player PL12 and the localization position mark MK12 at the reproduction time t2.
[0196]
　For example, assume that the player PL12 of the electric guitar is located at the position indicated by arrow Q11 at the predetermined reproduction time t1 on the video of the content, and the content creator places the localization position mark MK12 at the same position as the player PL12.
[0197]
　Also, at the reproduction time t2 after the reproduction time t1, the player PL12 of the electric guitar moves to the position indicated by the arrow Q12 on the video of the content, and at the reproduction time t2, the content creator is at the same position as the player PL12'. It is assumed that the localization position mark MK12' is placed at.
[0198]
　Here, for other reproduction times between the reproduction time t1 and the reproduction time t2, the content creator does not particularly specify the position of the localization position mark MK12.
[0199]
　In such a case, the localization position determination unit 41 performs interpolation processing to determine the position of the localization position mark MK12 at another reproduction time between the reproduction time t1 and the reproduction time t2.
[0200]
　During the interpolation processing, for example, based on the position information indicating the position of the localization position mark MK12 at the reproduction time t1 and the position information indicating the position of the localization position mark MK12′ at the reproduction time t2, the horizontal angle and the vertical angle as the position information are obtained. , And radius for each of the three components, the value of each component of the position information indicating the position of the localization position mark MK12 at the target reproduction time is obtained by linear interpolation.
[0201]
　As described above, even when the position information is represented by the coordinates of the three-dimensional rectangular coordinate system, the coordinates such as the x coordinate, the y coordinate, and the z coordinate are the same as when the position information is represented by the polar coordinates. Linear interpolation is performed for each component.
[0202]
　In this way, when the position information of the localization position mark MK12 at another reproduction time between the reproduction time t1 and the reproduction time t2 is obtained by the interpolation process, the position of the player PL12 of the electric guitar on the video is reproduced during the content reproduction. As the sound moves, the localization position of the sound image of the electric guitar, that is, the sound image of the sound of the audio object, also moves. As a result, it is possible to obtain natural content in which the sound image position moves smoothly without any discomfort.
[0203]

　Next, as described with reference to FIGS. 6 to 10, the operation of the signal processing device 11 when the present technology is applied to object-based audio will be described. That is, hereinafter, the localization position determination processing by the signal processing device 11 will be described with reference to the flowchart of FIG.
[0204]
　In step S41, the control unit 23 sets the reproduction environment.
[0205]
　For example, when the content creation tool is activated, the content creator operates the input unit 21 to specify the setting parameters shown in FIG. Then, the control unit 23 determines the setting parameter based on the signal supplied from the input unit 21 according to the operation of the content creator.
[0206]
　Thereby, for example, the size of the listening space, the listening position in the listening space, the size and aspect ratio of the screen on which the image of the content is displayed, the arrangement position of the screen in the listening space, and the like are determined.
[0207]
　In step S42, the display control unit 43 controls the display unit 24 based on the setting parameter determined in step S41 and the image data of the video of the content, and causes the display unit 24 to display the display screen including the POV image.
[0208]
　Thereby, for example, the window WD11 including the POV image P21 and the bird's-eye view image P22 shown in FIG. 6 is displayed.
[0209]
　At this time, the display control unit 43 draws a wall of the listening space (room) in the POV image P21 and the bird's-eye view image P22 according to the setting parameter set in step S41, or sets the position in the position determined by the setting parameter according to the setting parameter. Display screen SC11 of a certain size. Further, the display control unit 43 causes the image of the content to be displayed at the position of the screen SC11.
[0210]
　Furthermore, in the content creation tool, the speakers that make up the speaker system are displayed on the POV image and the bird's-eye view image, more specifically, whether or not to display an image imitating the speaker, and the channel configuration of the speaker system when the speaker is displayed are selected. be able to. The content creator operates the input unit 21 as necessary to instruct whether to display a speaker or select a channel configuration of the speaker system.
[0211]
　In step S43, the control unit 23 determines whether to display the speaker in the POV image and the bird's-eye view image based on the signal and the like supplied from the input unit 21 according to the operation of the content creator.
[0212]
　When it is determined in step S43 that the speaker is not displayed, the process of step S44 is not performed, and then the process proceeds to step S45.
[0213]
　On the other hand, if it is determined in step S43 that the speaker is to be displayed, then the process proceeds to step S44.
[0214]
　In step S44, the display control unit 43 controls the display unit 24 to display each speaker of the speaker system having the channel configuration selected by the content creator on the POV image and the bird's-eye view image with the speaker layout of the channel configuration. Let Thereby, for example, the speaker SP11 and the speaker SP12 shown in FIG. 9 are displayed in the POV image P21 and the bird's-eye view image P22.
[0215]
　If the speaker is displayed by the process of step S44, or if it is determined that the speaker is not displayed in step S43, the localization position determination unit 41 generates the sound image based on the signal supplied from the input unit 21 in step S45. Select the audio track for which you want to adjust the localization position.
[0216]
　For example, in step S45, the same processing as step S12 in FIG. 4 is performed, and a predetermined reproduction time on a desired audio track is selected as a sound image localization adjustment target.
[0217]
　When the sound image localization adjustment target is selected, the content creator subsequently operates the input unit 21 to move the location position of the localization position mark in the listening space to an arbitrary position, and to correspond to the localization position mark. Specifies the sound image localization position of the sound on the audio track.
[0218]
　At this time, the display control unit 43 controls the display unit 24 based on the signal supplied from the input unit 21 according to the input operation of the content creator, and moves the display position of the localization position mark.
[0219]
　In step S46, the localization position determination unit 41 determines the localization position of the sound image of the sound of the audio track to be adjusted, based on the signal supplied from the input unit 21.
[0220]
　That is, the localization position determination unit 41 acquires information (signal) indicating the position of the localization position mark viewed from the listening position in the listening space from the input unit 21, and the position indicated by the acquired information is the localization position of the sound image. To do.
[0221]
　In step S47, the localization position determination unit 41 generates position information indicating the localization position of the sound image of the sound of the audio track to be adjusted, based on the determination result of step S46. For example, the position information is information represented by polar coordinates with the listening position as a reference.
[0222]
　The position information generated in this way is position information indicating the position of the audio object corresponding to the audio track to be adjusted. That is, the position information obtained in step S47 is meta information of the audio object.
[0223]
　The position information as the meta information may be polar coordinates, that is, a horizontal angle, a vertical angle, and a radius as described above, or may be orthogonal coordinates. In addition, the setting parameter indicating the position, size, arrangement position, etc. of the screen, which is set in step S41, may be the meta information of the audio object.
[0224]
　In step S48, the control unit 23 determines whether to end the adjustment of the localization position of the sound image. For example, in step S48, the same determination processing as in step S15 of FIG. 4 is performed.
[0225]
　When it is determined in step S48 that the adjustment of the localization position of the sound image is not yet finished, the process returns to step S45 and the above-described process is repeated. That is, the localization position of the sound image is adjusted for the newly selected audio track. In this case, when the setting of whether or not to display the speaker is changed, the speaker is displayed or the speaker is not displayed according to the change.
[0226]
　On the other hand, if it is determined in step S48 that the adjustment of the localization position of the sound image is to be ended, the process proceeds to step S49.
[0227]
　In step S49, the localization position determination unit 41 appropriately performs interpolation processing on each audio track, and obtains the localization position of the sound image at the reproduction time for the reproduction time at which the localization position of the sound image is not designated.
[0228]
　For example, as described with reference to FIG. 10, the position of the localization position mark at the reproduction time t1 and the reproduction time t2 is specified by the content creator for a predetermined audio track, but other positions between those reproduction times are specified. Regarding the reproduction time, it is assumed that the position of the localization position mark is not specified. In this case, the position information is generated for the reproduction times t1 and t2 by the process of step S47, but the position information is not generated for the other reproduction times between the reproduction times t1 and t2. It is in a state.
[0229]
　Therefore, the localization position determination unit 41 performs interpolation processing such as linear interpolation on a predetermined audio track based on the position information at the reproduction time t1 and the position information at the reproduction time t2, and obtains the position information at another reproduction time. To generate. By performing such interpolation processing for each audio track, position information can be obtained for all reproduction times of all audio tracks. In the localization position determination process described with reference to FIG. 4, the same interpolation process as in step S49 may be performed to obtain the position information of unspecified reproduction time.
[0230]
　In step S50, the control unit 23 outputs the output bitstream based on the position information of each audio object, that is, the output bitstream based on the position information obtained in the processes of step S47 and step S49, and the localization position determination process ends. To do.
[0231]
　For example, in step S50, the control unit 23 performs rendering by the VBAP method based on the position information obtained as the meta information of the audio object and each audio track to generate audio data of each channel having a predetermined channel configuration.
[0232]
　Then, the control unit 23 outputs an output bitstream including the obtained audio data. Here, the output bit stream may include image data of video of the content.
[0233]
　Similar to the case of the localization position determination processing described with reference to FIG. 4, the output destination of the output bitstream can be an arbitrary output destination such as the recording unit 22, the speaker unit 26, or an external device.
[0234]
　That is, for example, an output bit stream including audio data and image data of content may be supplied to and recorded in the recording unit 22 or a removable recording medium, or audio data as an output bit stream may be supplied to the speaker unit 26. The sound of the content may be played.
[0235]
　In addition, the rendering process is not performed, and the position information obtained in step S47 or step S49 is used as meta information indicating the position of the audio object, and output including at least audio data of the content audio data, image data, and meta information. A bitstream may be generated.
[0236]
　At this time, the audio data, the image data, and the meta information are appropriately encoded by the control unit 23 by a predetermined encoding method, and the encoded bit stream including the encoded audio data, the image data, and the meta information is output bits. It may be generated as a stream.
[0237]
　In particular, this output bitstream may be supplied to the recording unit 22 or the like for recording, or may be supplied to the communication unit 25 so that the communication unit 25 transmits the output bitstream to an external device. You may
[0238]
　As described above, the signal processing device 11 displays the POV image, moves the localization position mark according to the operation of the content creator, and determines the localization position of the sound image based on the display position of the localization position mark. To do.
[0239]
　By doing this, the content creator can easily determine (designate) the appropriate localization position of the sound image by simply moving the localization position mark to the desired position while viewing the POV image. it can.
[0240]
　As described above, according to the present technology, for audio content of two left and right channels, and particularly for object-based audio content that targets sound image localization in three-dimensional space, sound image localization is performed at a specific position on a video in a content production tool. Such panning and position information of the audio object can be easily set.
[0241]

　By the way, the series of processes described above can be executed by hardware or software. When the series of processes is executed by software, the programs forming the software are installed in the computer. Here, the computer includes a computer incorporated in dedicated hardware and, for example, a general-purpose personal computer capable of executing various functions by installing various programs.
[0242]
　FIG. 12 is a block diagram showing a configuration example of hardware of a computer that executes the series of processes described above by a program.
[0243]
　In a computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are connected to each other by a bus 504.
[0244]
　An input/output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input/output interface 505.
[0245]
　The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, and the like. The output unit 507 includes a display, a speaker and the like. The recording unit 508 includes a hard disk, a non-volatile memory, or the like. The communication unit 509 includes a network interface or the like. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
[0246]
　In the computer configured as described above, the CPU 501 loads the program recorded in the recording unit 508 into the RAM 503 via the input/output interface 505 and the bus 504 and executes the program, thereby performing the above-described series of operations. Is processed.
[0247]
　The program executed by the computer (CPU 501) can be provided by being recorded in a removable recording medium 511 such as a package medium, for example. In addition, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.
[0248]
　In the computer, the program can be installed in the recording unit 508 via the input/output interface 505 by mounting the removable recording medium 511 in the drive 510. Further, the program can be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. In addition, the program can be installed in the ROM 502 or the recording unit 508 in advance.
[0249]
　It should be noted that the program executed by the computer may be a program in which processing is performed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.
[0250]
　The embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present technology.
[0251]
　For example, the present technology may have a configuration of cloud computing in which one device shares and jointly processes one function via a network.
[0252]
　In addition, each step described in the above-described flowcharts can be executed by one device or shared by a plurality of devices.
[0253]
　Further, when one step includes a plurality of processes, the plurality of processes included in the one step can be executed by one device or shared by a plurality of devices.
[0254]
　Furthermore, the present technology may have the following configurations.
[0255]
(1)
　An acquisition unit that acquires information about a localization position of a sound image of an audio object in the listening space specified in a state where the listening space viewed from the listening position is displayed, and
　a bit based on the information about the localization position A
　signal processing device comprising: a generation unit that generates a stream .
(2) 　The signal processing device according
　to
(1), wherein the generation unit generates the bitstream by using information about the localization position as meta information of the audio object .
(3) 　The signal processing device according to (2)
　, wherein the bitstream includes audio data of the audio object and the meta information
.
(4) 　The signal processing device according to any one of (1) to (3)
　, wherein the information regarding the localization position is position information indicating the localization position in the listening space
.
(5) 　The signal processing device according to (4),
　wherein the position information includes information indicating a distance from the listening position to the localization position
.
(6) 　The signal processing device according to (4) or (5),
　wherein the localization position is a position on a screen that displays an image arranged in the listening space
.
(7) The
　acquisition unit, based on the position information at a first time and the position information at a second time, a third time between the first time and the second time.
　7. The signal processing device according to any one of (4) to (6), wherein the position information in (1) is obtained by interpolation processing .
(8) 　The signal processing device according to any one of (1) to (7),
　further including a display control unit that controls display of an image of the listening space viewed from the listening position or a position near the listening position.
..
(9) 　The signal processing device
　according to
(8), wherein the display control unit causes the speakers of the speaker system having a predetermined channel configuration to be displayed on the image in a speaker layout having the predetermined channel configuration .
(10) 　The signal processing device
　according to
(8) or (9), wherein the display control unit displays a localization position mark indicating the localization position on the image .
(11) 　The signal processing device
　according to
(10), wherein the display control unit moves the display position of the localization position mark according to an input operation .
(12) The
　display control unit displays, on the image, a screen that is arranged in the listening space and displays a video including a subject corresponding to the audio object.
　The signal processing device according to any one of (8) to (11).
(13) 　The signal processing device according to any one of (8) to (12),
　wherein the image is a POV image
.
(14) The
　signal processing device
　acquires information regarding a localization position of a sound image of an audio object in the listening space designated in a state where the listening space viewed from the listening position is displayed,
　and based on the information regarding the localization position.
　Signal processing method for generating a bitstream by using the following method.
(15)
　Obtaining information about the localization position of the sound image of the audio object in the listening space specified in the state where the listening space viewed from the listening position is displayed,
　and generating a bitstream based on the information about the localization position.
　A program that causes a computer to execute a process including a step of performing.
Explanation of symbols
[0256]
　11 signal processing device, 21 input unit, 23 control unit, 24 display unit, 25 communication unit, 26 speaker unit, 41 localization position determination unit, 42 gain calculation unit, 43 display control unit
claims
[Claim 1]
　An acquisition unit that acquires information about the localization position of the sound image of the audio object in the listening space specified in the state where the listening space viewed from the listening position is displayed, and a
　bitstream is generated based on the information about the localization position. And a
　signal processing device that includes
[Claim 2]

　The signal processing device according 　to claim 1, wherein the generation unit generates the bitstream by using information about the localization position as meta information of the audio object .
[Claim 3]

　The signal processing device according to claim 2 　, wherein the bitstream includes audio data of the audio object and the meta information .
[Claim 4]

　The signal processing device according to claim 1 　, wherein the information regarding the localization position is position information indicating the localization position in the listening space .
[Claim 5]

　The signal processing device according to claim 4, 　wherein the position information includes information indicating a distance from the listening position to the localization position .
[Claim 6]

　The signal processing device according to claim 4, 　wherein the localization position is a position on a screen that displays an image arranged in the listening space .
[Claim 7]
　The acquisition unit, based on the position information at the first time and the position information at the second time, the position at the third time between the first time and the second time.
　The signal processing device according to claim 4, wherein the information is obtained by interpolation processing .
[Claim 8]

　The signal processing device according to claim 1, 　further comprising a display control unit that controls display of an image of the listening space viewed from the listening position or a position near the listening position .
[Claim 9]

　The signal processing device 　according to claim 8, wherein the display control unit causes the speakers of the speaker system having a predetermined channel configuration to be displayed on the image in a speaker layout having the predetermined channel configuration .
[Claim 10]

　The signal processing device 　according to claim 8, wherein the display control unit displays a localization position mark indicating the localization position on the image .
[Claim 11]

　The signal processing device 　according to claim 10, wherein the display control unit moves the display position of the localization position mark according to an input operation .
[Claim 12]

　The signal processing device 　according to claim 8, wherein the display control unit displays a screen, which is arranged in the listening space, on which an image including a subject corresponding to the audio object is displayed, on the image .
[Claim 13]

　The signal processing device according to claim 8, 　wherein the image is a POV image .
[Claim 14]
　A signal processing device
　acquires information about a localization position of a sound image of an audio object in the listening space specified in a state where the listening space viewed from the listening position is displayed, and
　a bit stream based on the information about the localization position. A
　signal processing method for generating .
[Claim 15]

　A
　step of 　acquiring information about a localization position of a sound image of an audio object in the listening space specified in a state where the listening space viewed from the listening position is displayed, and generating a bitstream based on the information about the localization position. A program that causes a computer to execute the included processing.