Specification
DESCRIPTION BROADCASTING SYSTEM, SENDING APPARATUS AND SENDING METHOD, RECEIVING APPARATUS AND RECEIVING METHOD, AND PROGRAM Technical Field
[0001]
The present invention relates to a broadcasting system, a sending apparatus and a sending method, a receiving apparatus and a receiving method, and a program, and, more particularly, it relates to, for example, a broadcasting system, a sending apparatus and a sending method, a receiving apparatus and a receiving method, and a program that make it possible to perform an appropriate process on a receiving side of the broadcasting system. Background Art
[0002]
For example, in the current broadcasting system for analog broadcasting or digital broadcasting, images and sounds are edited as material data in a broadcast station being on a sending side. The edited images and sounds are broadcasted as programs.
[0003]
On the other hand, for example, in each home being on a receiving side, output of the programs broadcasted from the broadcast station, i.e., display of the images and output of the sounds as the programs, are performed by a TV (a
television set) or the like. [0004]
Note that, when attention is focused on one program, one type of image and sound that is obtained as a result of editing performed in the broadcast station is broadcasted as one program in the current broadcasting system. Accordingly, users being on the receiving side can enjoy only the one type of image and sound. [0005]
However, requests for user participation, e.g., regarding one program such as a drama program, a request that a user desires to watch the development of the drama program from the point of view of a user's favorite character and a request that a user desires to cause the story development to differ depending on a point of view, potentially exist. [0006]
Hence, the present applicant has previously proposed a digital broadcasting receiving apparatus allowing a user to arbitrarily select and monitor a plurality of materials that are prepared in advance from different points of view (for example, see Patent Document 1). [0007]
[Patent Document 1] Japanese Unexamined Patent Application Publication No. 2006-217662
Disclosure of Invention Technical Problem [0008]
Meanwhile, in the broadcast station, for example, images and sounds are edited as described above. However, the editing is not performed with consideration of a display screen, in which the images are to be displayed, of a TV provided on the receiving side. [0009]
Accordingly, when the display screen of a TV provided on the receiving side is large, the face of a person is displayed, on the TV, in a size that is larger than a size of the face in a case in which the face is actually looked at, so that the sense of realism may be reduced. [0010]
Furthermore, on the TV provided on the receiving side, for example, as described above, the edited images and sounds, which were edited in the broadcast station, are displayed. [0011]
Accordingly, for example, when editing in which a large number of telops are combined with an image is performed in the broadcast station, even a user (an audience) that feels that emphasis effects using the telops are unnecessary must watch the image with which the large number of telops are
combined. [0012]
The present invention has been made in view of such circumstances, and makes it possible to perform an appropriate process on a receiving side of a broadcasting system, thereby making it possible, for example, to cause a user to feel a sense of realism or to display an image suitable for the preferences of a user. Technical Solution [0013]
A broadcasting system according to a first aspect of the present invention is a broadcasting system including a sending apparatus that sends data and a receiving apparatus that receives the data. The sending apparatus includes sending means for sending material data, which is acquired by a sensor, and additional information, which is used to process the material data with the receiving apparatus. The receiving apparatus includes receiving means for receiving the material data and the additional information; generating means for generating output data on the basis of the material data and the additional information; and output means for producing output on the basis of the output data. [0014]
In the first aspect as described above, the sending apparatus sends material data, which is acquired by the
sensor, and additional information, which is used to process the material data with the receiving apparatus. The receiving apparatus receives the material data and the additional information, and generates output data on the basis of the material data and the additional information. Then, output is produced on the basis of the output data. [0015]
A sending apparatus or a program according to a second aspect of the present invention is a sending apparatus that, together with a receiving apparatus which receives data, configures a broadcasting system and that sends the data, or a program for causing a computer to function as the sending apparatus. The sending apparatus includes sending means for sending material data, which is acquired by a sensor, and additional information, which is used to process the material data with the receiving apparatus. [0016]
A sending method according to the second aspect of the present invention is a sending method for a sending apparatus that, together with a receiving apparatus which receives data, configures a broadcasting system and that sends the data. The sending method includes a step of sending material data, which is acquired by a sensor, and additional information, which is used to process the material data with the receiving apparatus, with the sending
apparatus. [0017]
In the second aspect as described above, material data, which is acquired by the sensor, and additional information, which is used to process the material data with the receiving apparatus, are sent. [0018]
A receiving apparatus or a program according to a third aspect of the present invention is a receiving apparatus that, together with a sending apparatus which sends data, configures a broadcasting system and that receives the data, or a program for causing a computer to function as the receiving apparatus. The receiving apparatus includes, in a case in which the sending apparatus sends material data, which is acquired by a sensor, and additional information, which is used to process the material data with the receiving apparatus, receiving means for receiving the material data and the additional information; generating means for generating output data on the basis of the material data and the additional information; and output means for producing output on the basis of the output data. [0019]
A receiving method according to the third aspect of the present invention is a receiving method for a receiving apparatus that, together with a sending apparatus which
sends data, configures a broadcasting system and that receives the data. The receiving method includes the steps of: in a case in which the sending apparatus sends material data, which is acquired by a sensor, and additional information, which is used to process the material data with the receiving apparatus, with the receiving apparatus, receiving the material data and the additional information; generating output data on the basis of the material data and the additional information; and producing output on the basis of the output data with output means for producing output. [0020]
In the third aspect as described above, the material data and the additional information are received, and output data is generated on the basis of the material data and the additional information. Then, output is produced on the basis of the output data by the output means for producing output. [0021]
Note that the programs can be provided by being transmitted via a transmission medium or by being recorded on a recording medium. [0022]
Furthermore, the sending apparatus or the receiving apparatus may be an independent apparatus, or may be an
internal block configuring one apparatus. [0023]
Further, sending (broadcasting) (transmission) of data can also be performed via a wired or wireless transmission medium, and can also be performed via a mixed wired and wireless transmission medium. Advantageous Effects [0024]
According to the first to third aspects of the present invention, an appropriate process can be performed on a receiving side of the broadcasting system. Brief Description of Drawings [0025]
[Fig. 1] Fig. 1 is a block diagram illustrating an example of a configuration of an embodiment of a broadcasting system to which the present invention is applied.
[Fig. 2] Fig. 2 is a block diagram illustrating an example of a configuration of a sending apparatus 11.
[Fig. 3] Fig. 3 includes diagrams for explaining pieces of material data and effect information.
[Fig. 4] Fig. 4 is a diagram for explaining switching information.
[Fig. 5] Fig. 5 is a diagram for explaining display angle-of-view information.
[Fig. 6] Fig. 6 is a diagram for explaining data used for combining.
[Fig. 7] Fig. 7 is a diagram for explaining real-world information.
[Fig. 8] Fig. 8 is a flowchart for explaining a sending process.
[Fig. 9] Fig. 9 is a block diagram illustrating an example of a configuration of a receiving apparatus 12.
[Fig. 10] Fig. 10 is a flowchart for explaining a receiving process.
[Fig. 11] Fig. 11 is a diagram for explaining data that is supplied to a generating section 54 in a reality-mode process.
[Fig. 12] Fig. 12 is a block diagram illustrating an example of a configuration of the generating section 54 that performs the reality-mode process.
[Fig. 13] Fig. 13 includes diagrams schematically illustrating a state in which an image of an image-capture object is being captured by a camera and a display state on an output section 55 that is a display device.
[Fig. 14] Fig. 14 includes diagrams schematically illustrating a state in which an image of the image-capture object is being captured by the camera using optical zoom and a display state on the output section 55 that is a display device.
[Fig. 15] Fig. 15 includes diagrams schematically illustrating a state in which an image of the image-capture object is being captured by the camera using digital zoom and a display state on the output section 55 that is a display device.
[Fig. 16] Fig. 16 includes diagrams illustrating images displayed on the output section 55 that is a display device.
[Fig. 17] Fig. 17 is a flowchart for explaining the reality-mode process.
[Fig. 18] Fig. 18 is a diagram for explaining data that is supplied to the generating section 54 in an entertainment-mode process.
[Fig. 19] Fig. 19 is a block diagram illustrating an example of a configuration of the generating section 54 that performs the entertainment-mode process.
[Fig. 20] Fig. 20 is a block diagram illustrating an example of a configuration of an image processing unit 84.
[Fig. 21] Fig. 21 is a block diagram illustrating an example of another configuration of the receiving apparatus 12.
[Fig. 22] Fig. 22 is a block diagram illustrating an example of a configuration of an embodiment of a computer system to which the present invention is applied. Explanation of Reference Numerals [0026]
11 sending apparatus, 12 receiving apparatus, 13 transmission medium, 31 sensor section, 32 editing device, 33 encoding section, 34 sending section, 51 receiving section, 52 decoding section, 53 separation section, 54 generating section, 55 output section, 56 input section, 71 BF, 72 additional-information analyzing unit, 73 image processing unit, 81 image-data BF, 82 additional-information analyzing unit, 83 to-be-combined-data BF, 84 image processing unit, 91 image selecting part, 92 instruction selecting part, 93 display-region setting part, 94 gate part, 95 superimposing processing part, 101 generating section, 102 output section, 103 input section, 111 generating section, 112 output section, 113 input section, 201 bus, 202 CPU, 203 ROM, 204 RAM, 205 hard disk, 206 output unit, 207 input unit, 208 communication unit, 209 drive, 210 input/output interface, 211 removable recording medium Best Modes for Carrying Out the Invention [0027]
Fig. 1 illustrates an example of a configuration of an embodiment of a broadcasting system (the term system refers to a logical set of a plurality of apparatuses regardless of whether or not the apparatuses having individual configurations reside in the same housing) to which the present invention is applied. [0028]
In Fig. 1, the broadcasting system includes a sending apparatus 11 and a receiving apparatus 12. [0029]
The sending apparatus 11 is placed, for example, in a broadcast station, and sends various types of data as programs via a transmission medium 13 such as a ground wave, a satellite circuit, the Internet, or a CATV (Cable Television). [0030]
The receiving apparatus 12 is placed, for example, in home, and receives data that is sent from the sending apparatus 11 via the transmission medium 13. [0031]
Note that the receiving apparatus 12 is capable of functioning as, for example, a TV. [0032]
Furthermore, in the broadcasting system, the number of sending apparatuses 11 is not limited to one, and a plurality of sending apparatuses 11 can be provided. A similar configuration is also applied to the receiving apparatuses 12. [0033]
Fig. 2 illustrates an example of a configuration of the sending apparatus 11 illustrated in Fig. 1. [0034]
In Fig. 2, the sending apparatus 11 includes a sensor section 31, an editing device 32, an encoding section 33, and a sending section 34. [0035]
Note that the sensor section 11 and the editing device 32 can be provided as apparatuses different from the sending apparatus 11. [0036]
The sensor section 31 obtains material data that serves as a material in editing with the editing device 32, and supplies the material data to the editing device 32 and the encoding section 33. [0037]
Here, one type of sensor, one type of sensors, or a plurality of types of sensors can be employed as the sensor section 31. [0038]
Furthermore, examples of types of sensors include a camera that senses light and that outputs an image (that captures an image), a microphone that senses audio and that outputs a sound (that collects audio), and sensors that individually sense a temperature, a humidity, an air-flow direction, an air-flow speed, a vibration, a brightness, and so forth. [0039]
With the camera, an image is captured, and the image (data) is output as material data. Furthermore, with the microphone, audio is collected, and a sound (data) is output as material data. In addition, with each of the sensors, which senses a corresponding one of a temperature, a humidity, an air-flow direction, an air-flow speed, a vibration, and a brightness, data indicating a corresponding one of a temperature, a humidity, an air-flow direction, an air-flow speed, a vibration, and a brightness is output as material data. [0040]
The editing device 32 generates, on the basis of material data supplied from the sensor section 31, additional information that is to be used to process the material data with the receiving apparatus 12, and supplies the additional information to the encoding section 33. [0041]
Here, the additional information includes real-world information indicating a physical amount, in the real world, of a target that has been sensed by the sensor section 31, and effect information that is information which is to be used to process material data. [0042]
The effect information is generated, for example, in accordance with an operation performed by operating the
editing device 32 with a program producer who produces a
program in the broadcast station.
[0043]
The real-world information is generated regardless of the operation performed by the program producer. [0044]
Note that, in the editing device 32, the additional information is generated in for each type of material data. Accordingly, when a plurality of types of sensors are employed in the sensor section 31 and a plurality of types of material data are supplied from the sensor section 31 to the encoding section 33, additional information corresponding to each of the plurality of types of material data is supplied from the editing device 32 to the encoding section 33. [0045]
The encoding section 33 encodes the material data supplied from the sensor section 31 and the additional information supplied from the editing device 32. Note that the encoding section 33 performs multiplexing on the material data and encoded data on an as-needed basis. Then, the encoding section 33 supplies the encoded data that is obtained as a result of encoding to the sending section 34. [0046]
The sending section 34 sends the encoded data supplied
from the encoding section 33, i.e., the material data acquired by the sensor section 31 and the additional information generated by the editing device 32, via the transmission medium 13 (Fig. 1). [0047]
Next, the effect information included in the additional information that is generated by the editing device 32 will be described. [0048]
Fig. 3 illustrates pieces of material data and effect information corresponding to the pieces of the material data. [0049]
In other words, part A of Fig. 3 illustrates a plurality of streams of images (moving images) that are provided as pieces of material data. [0050]
For example, when an image for a certain program is configured by editing N, which is a plural number, streams of images S#l, S#2, ..., and S#N, the N streams of images S#l to S#N serve as pieces of material data for the image for the program. [0051]
Here, the N streams of images S#l to S#N may be images individually acquired by N cameras which are provided as N sensors, or may be images acquired by M (< N) cameras which
are provided as M sensors where M is smaller than N (and is
equal to and larger than one).
[0052]
In other words, for example, N streams of images that are obtained by capturing images of landscapes of different places with one camera at different times can be employed as the N streams of images S#l to S#N. [0053]
Furthermore, N streams of images that can be employed as the N streams of images S#l to S#N are as follows: N streams of images obtained by individually capturing images of an entire orchestra, a conductor, a player playing a specific music instrument, an auditorium, and so forth, for example, with N cameras; N streams of images obtained by individually capturing images of an entire stadium, a score board, a specific player, an auditorium, and so forth in sports coverage such as football coverage; and N streams of images obtained by individually capturing images of an entire studio, each cast, and so forth in a music program or the like. [0054]
Part B of Fig. 3 illustrates effect information corresponding to the N streams of images S#l to S#N that are pieces of material data illustrated in part A of Fig. 3. [0055]
The effect information includes, for example, switching information, display image-frame information, data used for combining, and so forth. [0056]
The switching information is, for example, information for switching among images displayed on a display device, which is described below, of the receiving apparatus 12 by selecting one steam of images from among the N streams of images S#l to S#N that are pieces of material data. [0057]
The display image-frame information indicates, for example, a region corresponding to a portion, which to be displayed on the display device of the receiving apparatus 12, of an image (in Fig. 3, one stream of images among the images S#l to S#N) that is material data. [0058]
The data used for combining is, for example, data that is to be used to be combined with an image (in Fig. 3, one stream of images among the images S#l to S#N) which is material data, and includes to-be-combined data, which is to be combined with the image. [0059]
Here, examples of the to-be-combined data include data (data for picture-in-picture) concerning an image to be displayed in a small screen using picture-in-picture
(hereinafter, referred to a PinP image), data concerning a
telop (data concerning a subtitle), and so forth.
[0060]
Next, the switching information included in the effect information illustrated in Fig. 3 will be described with reference to Fig. 4. [0061]
For example, as described above, when an image for the certain program A is configured by editing the N streams of images S#l to S#N, the switching information is information for switching among images displayed on the display device of the receiving apparatus 12 by selecting one steam of images from among the N streams of images S#l to S#N. [0062]
For example, in the editing device 32 illustrated in Fig. 2, cutting and editing are performed by sequentially connecting the images S#l, which is a piece of material data, in a range from a time code tsi to a time code tei, the images S#2, which is a piece of material data, in a range from a time code tS2 to a time code te2, the images S#3, which is a piece of material data, in a range from a time code tS3 to a time code te3, the images S#l, which is a piece of material data, in a range from a time code tS4 to a time code te4, ... with each other in accordance with an operation performed by the program producer as illustrated in Fig. 4,
thereby producing (an image for) the certain program A. In this case, for example, information with which pieces of material data that configure an image for the program A and time codes of the pieces of material data are specified at individual times in the program A serves as the switching information. [0063]
Note that, in the editing device 32, when a special effect, such as a wipe, is provided to a connection (a so-called edit point) between a certain stream of images S#i and another stream of images #i', the switching information includes information concerning the special effect. [0064]
Next, the display image-frame information included in the effect information illustrated in Fig. 3 will be described with reference to Fig. 5. [0065]
The display image-frame information indicates, for example, as described above, a region corresponding to a portion, which is to be displayed on the display device of the receiving apparatus 12, of an image that is material data. [0066]
In other words, for example, when material data is an image, in the editing device 32, editing can be performed,
in which, considering a rectangular region corresponding to a portion of the image, which is material data, as a region (hereinafter, referred to as a display region) corresponding to a portion, which is to be displayed on the display device of the receiving apparatus 12, of the image, the portion of the image is determined as an image for a program in accordance with an operation performed by the program producer as illustrated in Fig. 5. [0067]
In this case, in the editing device 32, the program producer can use, as the maximum region, the region of the image that is material data, and can produce a program as if the program producer had captured an image by operating a camera using a pan operation, a tilt operation, or a zoom operation in the region. [0068]
Here, the maximum display region is the region of the image that is material data. Accordingly, the angle of view of a camera that is provided as the sensor section 31 (Fig. 2) is set to be the maximum angle of view (to be on the widest angle side), and an image is captured, whereby the size of the display region can be maximized. [0069]
Note that any information with which the display region can be specified may be used as the display angle-of-view
information. [0070]
In other words, for example, a coordinate system is defined with respect to the region of the image that is material data, and in the coordinate system, coordinates between one vertex of the display region and a vertex that is diagonally opposite the vertex, coordinates of one vertex of the display region and horizontal and vertical lengths of the display region, or the like can be employed as the display angle-of-view information. [0071]
Furthermore, for example, the display region that is located at a predetermined position and that has a predetermined size can be determined as a default display region, and the history of the pan operation, the tilt operation, and the zoom operation that are performed for the default display region in editing by the program producer can be employed as the display angle-of-view information. [0072]
Next, the data used for combining included in the effect information illustrated in Fig. 3 will be described with reference to Fig. 6. [0073]
As described above, the data used for combining is data that is to be used to be combined with material data. The
data used for combining includes to-be-combined data, which is to be combined with material data, and timing information indicating a timing at which combining using the to-be-combined data is performed. [0074]
In other words, for example, material data is an image, and, in the editing device 32 illustrated in Fig. 2, editing in which a PinP image or a telop is superimposed on (combined with) the image, which is material data, in accordance with an operation performed by the program producer, thereby producing a program. In this case, the PinP image or the telop serves as the to-be-combined data. [0075]
Furthermore, information indicating a time over which the PinP image or the telop, which serves as the to-be-combined data, is superimposed (for example, a time at which the superimposing starts and a time at which the superimposing finishes), e.g., time codes for a program, serves as the timing information. [0076]
Note that the data used for combining includes, in addition, for example, information concerning a position, on the image, at which the PinP image or the telop that serves as the to-be-combined data is superimposed. [0077]
Moreover, for example, when material data is a sound, BGM (Background Music) or the like serves as the to-be-combined data. [0078]
When a certain type of material data is a sound and the sound is a sound accompanied with an image (a moving image) that is another type of material data, information indicating a time with respect to the sound can also be employed as timing information for BGM that is to be combined with the sound, and time codes of the image accompanied with the sound can also be employed. [0079]
Next, the real-world information included in the additional information that is generated by the editing device 32 (Fig. 2) will be described. [0080]
As described above, the real-world information is information indicating a physical amount, in the real world, of a target that has been sensed by the sensor section 31 (Fig. 2) . For example, when the sensor section 31 is a camera, image-capture-range information indicating an image-capture range at a position of an image-capture target (hereinafter, also referred to as an image-capture object) whose image is captured by the camera, and so forth is included in the real-world information.
[0081]
Here, the image-capture range of the camera will be described with reference to Fig. 7. [0082]
Fig. 7 is a top view schematically illustrating a state in which an image is being captured by the camera. [0083]
Now, for simplicity of description, it is supposed that an optical system (a group of lenses) of the camera has no distortion. It is supposed that an angle of view (hereinafter, also referred to as a "camera angle of view") of the camera in the horizontal direction is 9. In addition, it is supposed that a distance (hereinafter, also referred to as an "object distance") from the camera to the image-capture object is X. [0084]
Note that it is supposed that the camera angle of view 8 is known. [0085]
Furthermore, the object distance X can be measured utilizing, for example, a so-called automatic focus technique. [0086]
In other words, the object distance X can be measured using, for example, an infrared ray sensor or a sonar.
Furthermore, the object distance X can be measured utilizing a contrast detection scheme, in which the focus is adjusted so that the contrast of an image to be captured by the camera is maximized, a phase-difference detection scheme, in which the focus is adjusted so that differences among phases of images to be captured by the camera are removed, or the like. [0087]
Note that, in a case in which the object distance X is known in advance when capture of an image with the camera is performed, measurement of the object distance X is unnecessary. [0088]
Now, for simplicity of description, between horizontal and vertical lengths of the real world whose image is captured at the position of the image-capture object, for example, attention is focused on only the horizontal length. It is supposed that the horizontal length is an image-capture range D at the position of the image-capture object. [0089]
The image-capture range D can be determined in accordance with an equation D = 2 x X x tan(9/2) using the camera angle of view 9 and the object distance X. [0090]
Hence, in addition to the image that is material data,
the editing device 32 illustrated in Fig. 2 also acquires the camera angle of view 9 and the object distance X from the camera that is provided as the sensor section 31. The editing device 32 determines the image-capture range D from the camera angle of view 8 and the object distance X. Then, the editing device 32 causes the image-capture range D to be included as image-capture-range information in real-world information, and supplies the real-world information to the encoding section 33. [0091]
Next, a process (a sending process) that is performed by the sending apparatus 11 illustrated in Fig. 2 will be described with reference to Fig. 8. [0092]
In the sending apparatus 11, material data that is acquired by the sensor section 31 is supplied to the editing device 32 and the encoding section 33. [0093]
In the editing device 32, additional information that includes either or both of effect information and real-world information is generated on the basis of the material data supplied from the sensor section 31 and so forth, and the additional information is supplied to the encoding section 33. [0094]
When the material data is supplied from the sensor section 31 to the encoding section 33 and the additional information is supplied from the editing device 32 to the encoding section 33 as described above, in step Sll, the encoding section 33 encodes the material data supplied from the sensor section 31 and the additional information supplied from the editing device 32. The encoding section 33 supplies encoded data that is obtained as a result of encoding to the sending section 34, and the process proceeds to step S12. [0095]
In step S12, the sending section 34 sends the encoded data, which has been supplied from the encoding section 33, via the transmission medium 13 (Fig. 1). [0096]
Next, Fig. 9 illustrates an example of a configuration of the receiving apparatus 12 illustrated in Fig. 1. [0097]
In Fig. 9, the receiving apparatus 12 includes a receiving section 51, a decoding section 52, a separation section 53, a generating section 54, an output section 55, and an input section 56. [0098]
Note that the output section 55 and the input section 56 can be provided as apparatuses different from the
receiving apparatus 12. [0099]
The receiving section 51 receives encoded data that is sent from the sending apparatus 11 via the transmission medium 13, i.e., material data and additional information, and supplies the encoded data to the decoding section 52. [0100]
The decoding section 52 decodes the encoded data supplied from the receiving section 51 to obtain data, and supplies the data to the separation section 53. [0101]
The separation section 53 separates the data supplied from the decoding section 52 into the material data and the additional information, and supplies the material data and the additional information to the generating section 54. [0102]
The generating section 54 generates output data on the basis of the material data and the additional information supplied from the separation section 53, and supplies the output data to the output section 55. [0103]
The output section 55 produces predetermined output on the basis of the output data supplied from the generating section 54. [0104]
Here, a device that stimulates the five senses of a person can be employed as the output section 55. [0105]
In other words, a device that stimulates the sense of sight, such as a display device that displays an image or an illumination device that emits light, can be employed as the output section 55. Furthermore, a device that stimulates the sense of hearing, such as a speaker that emits a sound, can be employed as the output section 55. [0106]
Moreover, a device that stimulates the sense of touch, such as an air conditioner that can perform so-called air conditioning (air adjustment) (air balance) for temperature, humidity, and air flow or a shaking device that provides vibration, can be employed as the output section 55. [0107]
The input section 56 is operated by a user who uses the receiving apparatus 12. When the user operates the input section 56 in order to provide an instruction, the input section 56 accepts the instruction provided by the user, and supplies instruction information indicating the instruction to the generating section 54. [0108]
Here, when the instruction information is supplied from the input section 56 to the generating section 54, the
generating section 54 generates output data on the basis of
the instruction information.
[0109]
Next, a process (a receiving process) that is performed by the receiving apparatus 12 illustrated in Fig. 9 will be described with reference to Fig. 10. [0110]
The receiving section 51 waits for encoded data sent from the sending apparatus 11 via the transmission medium 13. In step S31, the receiving section 51 receives the encoded data, and supplies the encoded data to the decoding section
52. The process proceeds to step S32.
[0111]
In step S32, the decoding section 52 decodes the encoded data supplied from the receiving section 51 to obtain data, and supplies the data to the separation section
53. The process proceeds to step S33.
[0112]
In step S33, the separation section 53 separates the data (decoded data), which has been supplied from the decoding section 52, into material data and additional information, and supplies the material data and the additional information to the generating section 54. The process proceeds to step S34. [0113]
In step S34, the generating section 54 generates output data on the basis of the material data and the additional information supplied from the separation section 53, and supplies the output data to the output section 55. The process proceeds to step S35. [0114]
In step S35, the output section 55 produces predetermined output on the basis of the output data supplied from the generating section 54. [0115]
As described above, in the receiving apparatus 12 illustrated in Fig. 9, the process of generating output data is performed on the basis of the material data and the additional information by the generating section 54. The process includes processes, and the processes are classified into a reality-mode process and an entertainment-mode process. [0116]
In other words, real-world information and effect information are included in additional information. Among the processes of generating output data, a process performed using the real-world information is the reality-mode process, and a process performed using the effect information is the entertainment-mode process. [0117]
Hereinafter, the reality-mode process and the entertainment-mode process will be described. [0118]
Fig. 11 illustrates data (information) that is supplied to the generating section 54 when the receiving apparatus 12 performs the reality-mode process. [0119]
In the reality-mode process, material data and real-world information that is included in additional information are supplied from the separation section 53 to the generating section 54. [0120]
Furthermore, in the reality-mode process, when the user operates the input section 56 so as to provide a zoom instruction, instruction information corresponding to the operation is supplied from the input section 56 to the generating section 54. [0121]
Further, in the reality-mode process, characteristics information indicating the characteristics of the output section 55 is supplied to the generating section 54, for example, from the output section 55. [0122]
Then, in the reality-mode process, the generating section 54 generates output data on the basis of the
material data, the real-world information, and the characteristics information concerning the output section 55 so that a physical amount recognized from an output of the output section 55 will be identified as being a physical amount indicated by the real-world information. [0123]
Here, when the instruction information is supplied from the input section 56 to the generating section 54, the generating section 54 generates the output data also on the basis of the instruction information. [0124]
Note that, for the reality-mode process, the receiving apparatus 12 can be configured without provision of the input section 56. When the receiving apparatus 12 is configured without provision of the input section 56, no instruction information is supplied to the generating section 54. [0125]
Next, for example, supposing that the output section 55 is a display device and output data is an image to be displayed on the output section 55 which is a display device, the reality-mode process will be described. [0126]
Here, it is supposed that the image-capture-range information D, which is described with reference to Fig. 7,
is included in real-world information. Furthermore, it is supposed that size information F indicating a size of a display screen in which the output section 55 that is a display device displays an image is included in characteristics information. [0127]
Note that, here, for simplicity of description, it is supposed that the size information F indicates, between horizontal and vertical lengths of the display screen of the output section 55 that is a display device, for example, the horizontal length as in the case of the image-capture-range information D (the image-capture range D). [0128]
Fig. 12 illustrates an example of a configuration of the generating section 54 in a case in which the receiving apparatus 12 performs the reality-mode process. [0129]
In Fig. 12, the generating section 54 includes a BF (buffer) 71, an additional-information analyzing unit 72, and an image processing unit 73. [0130]
An image that is material data is supplied from the separation section 53 (Fig. 11) to the BF 71. The BF 71 stores the image that is material data supplied from the separation section 53.
[0131]
Real-world information included in additional information is supplied from the separation section 53 to the additional-information analyzing unit 72. The additional-information analyzing unit 72 analyzes the real-world information supplied from the separation section 53. The additional-information analyzing unit 72 extracts, for example, the image-capture-range information D (Fig. 7) included in the real-world information, and supplies the image-capture-range information D to the image processing unit 73. [0132]
As described above, the image-capture-range information D is supplied from the additional-information analyzing unit 72 to the image processing unit 73. Further, the image that is material data stored in the BF 71 is supplied to the image processing unit 73. In addition, the size information F that is characteristics information concerning the display device is supplied from the output section 55 (Fig. 11), which is a display device, to the image processing unit 73. [0133]
Furthermore, when the user operates the input section 56 (Fig. 11) so as to provide the zoom instruction, instruction information corresponding to the operation is supplied from the input section 56 to the image processing
unit 73. [0134]
The image processing unit 73 processes the image, which is material data, on the basis of the image that is material data supplied from the BF 71, the image-capture-range information D that has been supplied from the additional-information analyzing unit 72, and the size information F that is characteristics information supplied from the output section 55, thereby generating an image which is output data so that the size of an object (the image-capture object) recognized from the image displayed on the output section 55 which is a display device will be identified as being the size of the image-capture object in the real world. The image processing unit 73 supplies the output data to the output section 55 (Fig. 11) that is a display device. [0135]
Furthermore, the image processing unit 73 processes the image, which is material data, on the basis of the image that is material data supplied from the BF 71, the image-capture-range information D that has been supplied from the additional-information analyzing unit 72, and the size information F that is characteristics information supplied from the output section 55, thereby generating an image as output data so that the size of the image-capture object recognized from the image displayed on the output section 55
that is a display device will not exceed the size of the image-capture object in the real world. The image processing unit 73 supplies the output data to the output section 55 (Fig. 11) that is a display device. [0136]
In other words, when the instruction information is supplied from the input section 56 to the image processing unit 73, the image processing unit 73 performs a process of magnifying the image, which is material data supplied from the BF 71, only by a zoom magnification that is specified in the instruction information supplied from the input section 56, thereby generating output data. [0137]
However, in this case, the image processing unit 73 limits magnification of the image, which is material data, so that the size of the image-capture object recognized from the image displayed on the output section 55 which is a display device will not exceed the size of the image-capture object in the real world. [0138]
Note that, supposing that a magnification in a case of magnifying an image that is material data is a display zoom magnification z, magnification in a case in which the display zoom magnification z is smaller than one (and larger than zero) means reduction.
[0139]
Furthermore, in Fig. 12, it is supposed that the size information F is supplied from the output section 55, which is a display device, to the image processing unit 73 of the generating section 54. However, otherwise, the size information F can be stored in advance, for example, in the generating section 54. [0140]
Next, the processes performed by the image processing unit 73 will further be described. [0141]
Note that, here, for simplicity of description, it is supposed that the display region (Fig. 5) indicated by the display angle-of-view information is equal to the entire region of an image that is material data. [0142]
Fig. 13 schematically illustrates a state in which an image of the image-capture object is being captured by the camera and a state in which the image that is material data obtained by the image capture is displayed on the output section 55 that is a display device. [0143]
In other words, part A of Fig. 13 is a top view schematically illustrating a state in which an image is being captured by the camera, and is a drawing the same as
Fig. 7. [0144]
As described above, the image-capture-range information D can be determined in accordance with the equation D = 2 x X x tan0/2 using the camera angle of view 0 and the object distance X. [0145]
Part B of Fig. 13 illustrates a display state in which the image that is material data obtained by performing image capture with the camera as illustrated in part A of Fig. 13 is displayed on the output section 55, which is a display device, without being magnified. [0146]
On the output section 55 that is a display device, the real world whose size is indicated by the image-capture-range information D is displayed in the display screen whose size is indicated by the size information F. [0147]
Accordingly, the image-capture object is displayed in a size that is F/D times the size of the image-capture object in the real world. [0148]
Now, supposing that the image-capture-range information D is equal to or larger than the size information F (F < D), if the display zoom magnification z is equal to or lower
than D/F, the size of the image-capture object displayed on the output section 55 that is a display device is equal to or smaller than the size of the image-capture object in the real world. [0149]
Accordingly, in a case in which the image-capture-range information D is equal to or larger than the size information F, if the display zoom magnification z is equal to or smaller than D/F, loosing of the reality of the image-capture object displayed on the output section 55, which is a display device, because of a display of the image-capture object on the output section 55, which is a display device, in a size that is larger than the size of the image-capture object in the real world does not occur. [0150]
Hence, in the image processing unit 73 (Fig. 12), in a case in which the image-capture-range information D is equal to or larger than the size information F, the display zoom magnification z that is used when magnification of an image which is material data is performed is limited to being equal to or lower than D/F. [0151]
In contrast, in a case in which the image-capture-range information D is smaller than the size information F (F > D) , when the image that is material data is displayed on the
output section 55, which is a display device, without performing any process on the image, the size of the image-capture object displayed on the output section 55 exceeds the size of the image-capture object in the real world. [0152]
Accordingly, when the image-capture object is displayed on the output section 55, which is a display device, in a size that is larger than the size of the image-capture object in the real world, the reality of the image-capture object displayed on the output section 55 that is a display device is lost, so that the sense of realism is lost. [0153]
As described above, for example, a case in which the output section 55 is a display device having a large screen is considered as a case in which the image-capture-range information D is smaller than the size information F. [0154]
Furthermore, for example, also in a case in which an image that is material data is captured using optical zoom or digital zoom, the image-capture-range information D may be smaller than the size information F. [0155]
Fig. 14 schematically illustrates a state in which an image of the image-capture object is being captured by the camera using optical zoom and a state in which the image
that is material data obtained by the image capture is displayed on the output section 55 that is a display device. [0156]
In other words, part A of Fig. 14 is a top view schematically illustrating a state in which an image is being captured by the camera using optical zoom. [0157]
The camera angle of view 6 is reduced (made to be smaller) using optical zoom. As a result, the image-capture-range information D is made, for example, to be smaller than that in a case illustrated in part A of Fig. 13. [0158]
Part B of Fig. 14 illustrates a display state in which the image that is material data obtained by image capture as illustrated in part A of Fig. 14 is displayed on the output section 55 that is a display device. [0159]
On the output section 55 that is a display device, the real world whose size is indicated by the image-capture-range information D is displayed in the display screen whose size is indicated by the size information F. [0160]
In Fig. 14, the image-capture-range information D is smaller than the size information F. For this reason, the image-capture object is displayed on the output section 55,
which is a display device, in a size that is larger than (in a size that is F/D times) the size of the image-capture object in the real world. [0161]
Fig. 15 schematically illustrates a state in which an image of the image-capture object is being captured by the camera using digital zoom and a state in which the image that is material data obtained by the image capture is displayed on the output section 55 that is a display device. [0162]
In other words, part A of Fig. 15 is a top view schematically illustrating a state in which an image is being captured by the camera using digital zoom. [0163]
In a case of digital zoom, an image that is obtained by trimming, in which, for example, a central portion of the image captured by the camera is cut and magnified by signal processing, is output as an image that is material data. [0164]
Regarding the image that is material data obtained using digital zoom, the camera angle of view 6 is reduced as in the case of optical zoom (Fig. 14). As a result, the image-capture-range information D is made, for example, to be smaller than that in a case illustrated in part A of Fig. 13.
[0165]
Part B of Fig. 15 illustrates a display state in which the image that is material data obtained by image capture as illustrated in part A of Fig. 15 is displayed on the output section 55 that is a display device. [0166]
On the output section 55 that is a display device, the real world whose size is indicated by the image-capture-range information D is displayed in the display screen whose size is indicated by the size information F. [0167]
In Fig. 15, the image-capture-range information D is smaller than the size information F. For this reason, the image-capture object is displayed on the output section 55, which is a display device, in a size that is larger than (in a size that is F/D times) the size of the image-capture object in the real world. [0168]
As described above, when an image that is material data is captured using optical zoom or digital zoom, the image-capture-range information D may be smaller than the size information F. In such a case, the image-capture object is displayed in a size that is larger than the size of the image-capture object in the real world, so that the reality is lost.
[0169]
Hence, in the image processing unit 73 (Fig. 12), in a case in which the image-capture-range information D is smaller than the size information F, the display zoom magnification z is set to be D/F, and the image that is material data is magnified by D/F (here, the image is reduced because D/F is smaller than one).
[0170]
In this case, the image-capture object, which is displayed in a size that is F/D times the size of the image-capture object in the real world if magnification
(reduction) is not performed in the image processing unit 73, is displayed in a size that is equal to the size of the image-capture object in the real world.
[0171]
Fig. 16 illustrates images displayed on the output section 55, which is a display device, in a case in which the image-capture-range information D is smaller than the size information F.
[0172]
In other words, part A of Fig. 16 illustrates a display state in which an image that is material data captured using optical zoom is displayed on the output section 55 that is a display device. Part B of Fig. 16 illustrates a display state in which an image that is material data captured using
digital zoom is displayed on the output section 55 that is a
display device.
[0173]
In a case in which the image-capture-range information D is smaller than the size information F, the image processing unit 73 sets the display zoom magnification z to be D/F (< 1), magnifies the image that is material data by D/F, and supplies the magnified image as output data to the output section 55. [0174]
As a result, on the output section 55 that is a display device, the image-capture object is displayed in a size that is equal to the size of the image-capture object in the real world. [0175]
Here, in the image processing unit 7 3, the image that is material data is magnified (reduced) by D/F (< 1), and, as a result, an image (output data) that is obtained by the magnification is an image whose horizontal length is equal to the image-capture-range information D. [0176]
And, in this case, because the image-capture-range information D is smaller than the size information F, the horizontal length F'(= D) of the image that is output data is smaller than, the size information F, i.e., the
horizontal length of the display screen of the output
section 55 that is a display device.
[0177]
For this reason, on the output section 55 that is a display device, black (a so-called black frame) or the like is displayed in a portion of the display screen other than a portion, in which the image that is output data is displayed, of the display screen, for example, as illustrated in part A of Fig. 16. [0178]
Note that, when an image that is material data is captured using digital zoom, a portion (a portion shaded with diagonal lines in part A of Fig. 15) (hereinafter, also referred to as a "trimmed portion") of the image, which is captured by the camera, other than the central portion, which is magnified using digital zoom, of the image may remain. [0179]
In this case, on the output section 55 that is a display device, the trimmed portion is displayed in a portion of the display screen other than a portion, in which the image that is output data is displayed, of the display screen, for example, as illustrated in part B of Fig. 16. [0180]
Next, the reality-mode process that is performed in the
image processing unit 73 illustrated in Fig. 12 will be
described with reference to Fig. 17.
[0181]
In step S51, the image processing unit 73 determines a maximum zoom magnification of D/F, which is the maximum value of the display zoom magnification z with which an image that is material data is maximized, on the basis of the image-capture-range information D included in the real-world information that is supplied from the additional-information analyzing unit 72 and the size information F that is characteristics information concerning the output section 55 which is a display device. In addition, the image processing unit 73 sets a criterion zoom magnification Zi that is the display zoom magnification z in a case in which no instruction information is supplied from the input section 56 (Fig. 11). The process proceeds to step S52. [0182]
Here, in a case in which the image-capture-range information D is equal to or larger than the size information F, the criterion zoom magnification zi is set to be one. In case in which the image-capture-range information D is smaller than the size information F, the criterion zoom magnification Zi is set to be the maximum zoom magnification of D/F. [0183]
In step S52, the image processing unit 73 determines whether or not instruction information for the zoom instruction has been supplied from the input section 56. [0184]
In step S52, when it is determined that no instruction information has been supplied from the input section 56 to the image processing unit 73, the process proceeds to step S53. The image processing unit 73 sets the display zoom magnification z to be the criterion zoom magnification zi. The process proceeds to step S57. [0185]
Furthermore, in step S52, when it is determined that instruction information has been supplied from the input section 56 to the image processing unit 73, the process proceeds to step S54. The image processing unit 73 determines whether or not a zoom magnification (hereinafter, also referred to as an instruction zoom magnification A) that is specified in the instruction information supplied from the input section 56 is equal to or lower than the maximum zoom magnification of D/F. [0186]
In step S54, when it is determined that the instruction zoom magnification A is equal to or lower than the maximum zoom magnification of D/F, the process proceeds to step S55. The image processing unit 73 sets the display zoom
magnification z to be the instruction zoom magnification A.
The process proceeds to step S57.
[0187]
Furthermore, in step S54, when it is determined that the instruction zoom magnification A is not equal to or lower than the maximum zoom magnification of D/F, i.e., when a zoom magnification with which the image-capture object is displayed in a size larger than the size of the image-capture object in the real world and which exceeds the maximum zoom magnification of D/F is specified in the instruction, the process proceeds to step S56. The image processing unit 73 sets the display zoom magnification z to be the maximum zoom magnification of D/F. The process proceeds to step S57. [0188]
In step S57, the image processing unit 73 magnifies the image that is stored as material data in the BF 71 only by the display zoom magnification z, thereby generating a magnified image as output data. The image processing unit 73 supplies the magnified image to the output section 55 (Fig. 11) that is a display device. [0189]
Then, on the output section 55 that is a display device, the magnified image that is output data supplied from the image processing unit 73 is displayed. In this manner, as
described with reference to Fig. 13, the image-capture object is displayed in a size that is F/D times the size of the image-capture object in the real world. [0190]
Accordingly, when no instruction information has been supplied from the input section 56 to the image processing unit 73, i.e., when the user has not performed an operation of providing the zoom instruction, the image-capture object is displayed on the output section 55, which is a display device, in a size (z(=D/F) x F/D) that is equal to the size of the image-capture object in the real world or in a size (z(=l) x F/D) that is F/D times the size of the image-capture object in the real world. [0191]
In other words, in a case in which the image-capture-range information D is smaller than the size information F, as described above, the criterion zoom magnification zi, and therefore, the display zoom magnification z, is set to be the maximum zoom magnification of D/F. Thus, the image-capture object is displayed in a size that is D/F x F/D times the size of the image-capture object in the real world, i.e., in a size that is equal to the size of the image-capture object in the real world. [0192]
Furthermore, in a case in which the image-capture-range
information D is equal to or larger than the size information F, as described above, the criterion zoom magnification zi, and therefore, the display zoom magnification z, is set to be one. Thus, the image-capture object is displayed in a size that is F/D (< 1) times the size of the image-capture object in the real world. [0193]
In contrast, when instruction information has been supplied from the input section 56 to the image processing unit 73, i.e., when the user has performed an operation of providing the zoom instruction for zoom using the instruction zoom magnification A, in a case in which the instruction zoom magnification A is equal to or lower than the maximum zoom magnification of D/F, the image-capture object is displayed in a size (z(=A) x F/D) that is F/D x A times the size of the image-capture object in the real world. In a case in which the instruction zoom magnification A is not equal to or lower than the maximum zoom magnification of D/F, the image-capture object is displayed in a size (z(=D/F) x F/D) that is equal to the size of the image-capture object in the real world. [0194]
In other words, in a case in which the instruction zoom magnification A is equal to or lower than the maximum zoom magnification of D/F, as described above, the display zoom
magnification z is set to be the instruction zoom magnification A. Thus, the image-capture object is displayed in a size that is A(
Documents