IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND PROGRAM
TECHNICAL FIELD [0001]
The present invention relates to an image processing apparatus, an image processing method, and a program. BACKGROUND ART [0002]
In recent years, a technique related to distribution of digital video data (hereinafter referred to as video data) has been greatly advancing. Specifically, development in the technique related to distribution and recording of high image quality video data is outstanding. Among them, attention has been given to the technique related to coding and decoding of the video data. The video data having high spatial resolution and temporal resolution has a very large data size and thus such video data is desirably distributed or recorded by being efficiently compressed after coding. The coding technique enabling the high image quality video data to be compressed at higher compression rate, and the decoding technique enabling reproduction at higher spatial resolution have been desired.
[0003] For instance, Patent Documents 1 and 2 disclose a basic technique for generating the video data having high spatial resolution and temporal resolution by combining first video data (e.g., moving image data etc.) which has low spatial resolution but high temporal resolution and second video data (e.g., series of still image data etc.) which has high spatial resolution but low temporal resolution. Such technique predicts the motion vector between frames from the first video data, and compensates the high frequency component of the first video data using the motion vector and the second video data. Such technique generates a frame of an arbitrary time point not contained in the second video data using the motion vector detected from the first video data and the frame of the second video data at a time point close to such arbitrary time point. Japanese Patent Application Laid-Open No. 2004-312276 and No. 2004-312277 include a description related to an image data recording and reproducing device for generating video data having high spatial resolution and temporal resolution using the above-described technique. [0004]
[Patent Document 1] Japanese Patent Application Laid-Open No. 2004-312276
[Patent Document 2] Japanese Patent Application Laid-Open No. 2004-312277
DISCLOSURE OF THE INVENTION [0005]
However, if the time interval at which each of a plurality of still image data is recorded is large, the motion vector is difficult to accurately predict, and thus high image quality video data is difficult to generate. In other words, since the amount of change in the moving body in the image data becomes greater the more the recorded time of the video data to generate is distant from the recorded time of the recorded still image data, the motion compensation using the high frequency component contained in the recorded high resolution still image becomes difficult tcr perform. For instance, if the time difference h between the recorded time of the still image data and the recorded time of
the desired created picture becomes greater than 0.1 second, the compensation of the high frequency component by motion compensation becomes difficult to perform. If the number of recording of the high definition still image data increases, the amount of data to be saved obviously becomes enormous.
[0006]In light of the foregoing, it is desirable to provide a novel and improved image processing apparatus, an image processing method, and a program capable of realizing the motion compensation at high accuracy when executing the motion prediction using the low resolution image data series and executing the motion compensation using the high resolution image data to generate the high resolution image data even if the time of the image data to generate and the time of the high resolution image data are distant.
[0007]In order to solve the above issue, according to another embodiment of the present invention, there is provided an image processing apparatus including: an acquiring unit for acquiring moving image data containing a plurality of successive frames, and one or a plurality of image data corresponding to the frames and having a spatial resolution higher than the frames; a motion prediction unit for detecting a motion vector between the frames using the moving image data; and an image generation unit for generating motion compensated image data corresponding to a predetermined frame based on the image data and the motion vector. [0008]
The image generation unit generates the motion compensated image data being positioned between the predetermined frame and a frame corresponding to the image data and corresponding to the frame, and generates motion compensated image data corresponding to the predetermined frame based on the motion compensated image data and the motion vector. [0009]The image generation unit may execute motion compensation in order from a frame corresponding to a time close to the image data, and sequentially generate motion compensated image data corresponding to the predetermined frame based on the motion compensated image data and the motion vector.
[0010]The image processing apparatus may further include a storage unit for recording the moving image data and the one or the plurality of image data. In this case, the acquiring unit may be configured to acquire the image data and the one or the plurality of image data recorded in the storage unit.
[0011]A first image data series having low spatial resolution and high temporal resolution may be recorded in the storage unit as the moving image data, and a second image data series having higher spatial resolution and lower temporal resolution than the first image data series may be recorded in the storage unit so as to correspond to the frame contained in the first image data series as the one or the plurality of image data.
[0012]The moving image generation unit may be configured to generate, with a frame contained in the first image data series and not contained in the second image data series as the predetermined frame, the image data corresponding to the predetermined frame, and enhance the temporal resolution of the second image data series.
[0013]An image data series obtained by down sampling an imaged image signal may be recorded in the storage unit as the moving image data.
[0014]The image generation unit may be configured to change a ratio of combining the predetermined frame, which is a reference source of the motion vector, and the image data, which is a reference destination of the motion vector, according to a difference amount in the motion compensation if the difference amount is greater than a predetermined value. [0015]
If a difference amount between the frame corresponding to the image data and the predetermined frame is smaller than a predetermined value, the image generation unit may be configured to output the image data as image data corresponding to the predetermined frame without performing motion compensation.
[0016] In order to solve the above issue, according to another embodiment of the present invention, there is provided an image processing method, including the steps of: acquiring moving image data containing a plurality of successive frames, and one or a plurality of image data corresponding to the frames and having a spatial resolution higher than the frames; detecting a motion vector between the frames using the moving image data; and generating motion compensated image data corresponding to a predetermined frame based on the image data and the motion vector.
[0017] In the image generation step, the motion compensated image data being positioned between the predetermined frame and a frame corresponding to the image data and corresponding to the frame is generated, and motion compensated image data corresponding to the predetermined frame is generated based on the motion compensated image data and the motion vector.
[0018] In order to solve the above issue, according to another embodiment of the present invention, there is provided a program which causes a computer to realize the functions of the image processing apparatus. A recording medium recorded with such program can be also provided.
[0019] The image processing apparatus generates a high resolute image from a time close to the recorded time of the already-recorded still image data, saves the high resolution image generated in the past in the frame memory, and generates a new high resolution image with reference again to the saved image. Through such sequentially executed process, high definition still image can be generated even at a time distant from the recorded time.
[0020] According to the embodiments of the present invention described above, when executing the motion prediction using the image data series of low resolution and executing the motion compensation using the image data of high resolution to generate the image data of high resolution, the motion compensation can be performed at high accuracy even if the time of the image data to be generated and the time of the high resolution image data are distant.
BRIEF DESCRIPTION OF THE DRAWINGS
[0021]
FIG. 1 is an explanatory view describing a generation method of a high resolution created picture;
FIG. 2 is an explanatory view showing an apparatus configuration of an image processing apparatus according to one embodiment of the present embodiment;
FIG. 3 is an explanatory view describing an operation of an image generation circuit according to the present embodiment;
FIG. 4 is an explanatory view describing an operation of the image generation circuit according to the present embodiment;
FIG. 5 is an explanatory view describing an operation of the image generation circuit according to the present embodiment;
FIG. 6 is an explanatory view showing a generation method of a high resolution image according to the present embodiment; and
FIG. 7 is an explanatory view showing a hardware configuration example of an information processing device capable of realizing the functions of the image processing apparatus according to the present embodiment.
DESCRIPTION OF REFERENCE NUMERALS
[0022]
100 image processing apparatus
102 imaging lens
104 imaging element
106 image generation timing controller
108 display circuit
110 camera signal processing block
112 A/D converter
114 digital signal processing circuit
120 moving image recording and reproducing block
122^ 132 storage unit
124 moving image compression/decompression circuit
126 image density conversion circuit
130 still image recording and reproducing block
134 still image compression/decompression circuit
202 image generation circuit
204 frame memory
212 motion prediction unit
214 motion compensation unit
HRP high resolution picture
LRP low resolution picture
MP magnified picture
CP^P' created picture
BEST MODE FOR CARRYING OUT THE INVENTION [0023]
Hereinafter, preferred embodiments of the present invention wili be described in detail with reference to the appended drawings. Note that, in this specification and the
appended drawings, structural elements that have substantially the same function and
structure are denoted with the same reference numerals, and repeated explanation of
these structural elements is omitted.
[0024]
(Generation method of high resolution image)
First, prior to describing an image processing apparatus and an image processing method according to the embodiment of the present invention, an idea of using image data series (hereinafter referred to as moving image data) in which an image of low spatial resolution is successively recorded and image data of high spatial resolution (hereinafter referred to as still image data or HRP (High Resolution Picture)) discretely recorded in correspondence to an appropriate frame (hereinafter referred to as LRP (Low Resolution Picture)) contained in the moving image data to generate image data of high spatial resolution (hereinafter referred to as CP (Created Picture) or high resolution created picture) corresponding to an arbitrary frame contained in the moving image data will be briefly described with reference to FIG. 1. FIG. 1 is an explanatory view showing one example of a generation method of the high resolution image. [0025]
In FIG. 1, a high resolution picture HRP(t), low resolution pictures LRP(t), LRP(t+h), magnified pictures MP(t), MP(t+h), and a created picture CP(t+h) are illustrated. The character in the parentheses represent the photographed time. For instance, the created picture CP(t+h) represents the created picture corresponding to the low resolution picture LRP(t+h) photographed at time t+h.
[0026]
The created picture CP is image data created by executing motion compensation using the motion vector detected using the low resolution picture LRP and the high resolution picture HRP. As shown in FIG. 1, the motion vector (MV) between time t and time t+h is detected using the magnified picture MP(t) created by magnifying the low resolution picture LRP(t) and the magnified picture MP(t+h) created by magnifying the low resolution picture LRP(t+h) (SI). Then, a reference block of the high resolution picture HRP(t) corresponding to a reference block (BLK) of the low resolution picture LRP(t) is extracted based on the motion vector MV. The pixels of the reference block and the pixels of the magnified picture MP(t+h) are then combined at a predetermined ratio to create the created picture (CP(t+h) (S2).
[0027]
In the above-described example, the motion prediction and the motion compensation are executed using substantially different images. The magnified picture MP created by magnifying the low resolution picture LRP magnifies the spatial resolution same as the high resolution picture HRP, but lacks in high frequency component compared to the high resolution picture HRP. Thus, a deviation may occur in the motion vector detected using the magnified picture MP and the motion vector the high resolution picture HRP is to essentially have. In particular, if the accuracy of the motion prediction is low, the position of the reference block in the high resolution picture HRP shifts, and thus a remarkable noise is added to the high frequency component of the created picture CP created through motion compensation.
[0028]
Thus, a devisal of suppressing the noise from being added due to motion compensation when executing motion compensation based on a plurality of image data having spatial resolutions different from each other is desired. The error of the motion
vector becomes large when the time t at which the high resolution picture HRP(t) is photographed and the time t+h at which the created picture CP(t+h) is to be created are distant. A devisal of obtaining a high definition created picture CP(t+h) when the time interval h is large is thus desired.
[0029]
As hereinafter described, the image processing apparatus according to one embodiment of the present invention first executes the motion prediction and the motion compensation on the low resolution picture LRP(t+h") photographed at a time close to the time t the high resolution picture HRP(t) is photographed to generate a tentative created picture CP'(t+h"), and further executes the motion prediction and the motion compensation based on such tentative created picture CP'(t+h). The created picture CP(t+h) at the desired time t+h can be generated at high accuracy by sequentially repeating such process. This is because the time interval between frames referenced when executing one motion prediction is short, and the accuracy of the motion prediction enhances. The specific function configuration of the image processing apparatus will be described below.
[0030]
An image processing apparatus 100 according to one embodiment of the present invention will be described in detail below. [0031] [Apparatus configuration of image processing apparatus 100]
First, the apparatus configuration of the image processing apparatus 100 according to the present embodiment will be described with reference to FIG. 2. FIG. 2 is an explanatory view showing one example of the apparatus configuration of the image processing apparatus 100 according to the present embodiment.
[0032]
As shown in FIG. 2, the image processing apparatus 100 mainly includes an imaging lens 102, an imaging element 104, a camera signal processing block 110, a moving image recording and reproducing block 120, a still image recording and reproducing block 130, an image generation circuit 202, a frame memory 204, an image generation timing controller 106, and a display circuit 108.
[0033]
The imaging lens 102 is an optical lens for forming an image (hereinafter referred to as collected image) of a subject by collecting light. The imaging element 104 is a photoelectric element for converting the light collected by the imaging lens 102 to an electrical signal. A semiconductor element such as CCD image sensor (Charge Coupled Device Image Sensor) and CMOS image sensor (Complementary Metal Oxide Semiconductor Image Sensor) is used for the imaging element 104.
[0034]
The imaging element 104 has greater number of pixels than the number of effective pixels of the spatial resolution of the moving image data. The display circuit 108 causes a display device (not shown) to display the image data when recording or reproducing the image data or when checking the image data. The image generation timing controller 106 is a device for the operator to make an instruction to acquire the still image data while reproducing the moving image data. [0035] (Camera signal processing block 110)
As shown in FIG. 2, the camera signal processing block 110 includes an A/D converter 112 and a digital signal processing circuit 114.
[0036]
The A/D converter 112 is an analog-to-digital converter for converting an input analog signal to a digital signal, and outputting the same. The A/D converter 112 converts the analog signal of the collected image input from the imaging element 104 to the digital signal, and outputs the same to the digital signal processing circuit 114. The digital signal processing circuit 114 performs processes such as white balance adjustment and gamma correction on the digital signal (hereinafter referred to as image data) of the collected image input from the A/D converter 112. [0037] (Moving image recording and reproducing block 120)
As shown in FIG. 2, the moving image recording and reproducing block 120 includes an image density conversion circuit 126, a moving image compression/decompression circuit 124, and a storage unit 122.
[0038]
The image density conversion circuit 126 performs a decimation process (e.g., inter-frame differential coding etc.) on the series of image data (moving image data) input from the digital signal processing circuit 114 when recording the moving image data. The image density conversion circuit 126 outputs the moving image data after the decimation process to the moving image compression/decompression circuit 124. The moving image compression/decompression circuit 124 performs compression process on the moving image data of after decimation input from the image density conversion circuit 126 and records the compressed moving image data in the storage unit 122.
[0039]
When reproducing the moving image data, the moving image compression/decompression circuit 124 reads out the moving image data recorded on the storage unit 122 and performs decompression process thereon. The moving image compression/decompression circuit 124 outputs the decompressed moving image data to the image density conversion circuit 126. The image density conversion circuit 126 performs an interpolation process (e.g., restoring differenced data) on the moving image data input from the moving image compression/decompression circuit 124. [0040] (Still image recording and reproducing block 130)
As shown in FIG. 2, the still image recording and reproducing block 130 includes a still image compression/decompression circuit 134 and a storage unit 132. The storage unit 132 may realize the function using the storage device common with the storage unit 122.
[0041]
When recording the still image data, the still image compression/decompression circuit 134 performs compression process on the image data input from the digital signal processing circuit 114 and records in the storage unit 132. When reproducing the still image data, the still image compression/decompression circuit 134 reads out the image data recorded in the storage unit 122 and performs decompression process thereon.
[0042] (Image generation circuit 202)
Although not clearly shown in the figure, the image generation circuit 202 includes a motion prediction unit 212 and a motion compensation unit 214. The motion
compensation unit 214 serves an image generation unit. [0043]
The motion prediction unit 212 detects the motion vector by executing the motion prediction between frames on the moving image data transmitted from the moving image recording and reproducing block 120. For instance, the motion prediction unit 212 selects the still image data recorded at the time closest to the frame recorded at a predetermined time. The motion prediction unit 212 detects the motion vector between frames by executing the motion prediction between the frame corresponding to the still image data and the frame recorded at a predetermined time. In this case, the motion prediction unit 212 magnifies each frame (LRP) to reference to convert to the magnified picture (MP), and detects the motion vector between the relevant magnified pictures.
[0044]
As another example, the motion prediction unit 212 can extract a plurality of still image data recorded at a time point close to the frame recorded at a predetermined time, and then select the frame corresponding to the still image data most approximate to the frame recorded at the predetermined time. The motion prediction unit 212 may detect the motion vector between frames by executing the motion prediction between the frame corresponding to the selected still image data and the frame recorded at the predetermined time. The motion prediction unit 212 may detect the motion vector between frames without magnifying each frame (LRP), and magnify the motion vector so as to adapt to the spatial resolution of the created picture (CP). [0045]
The motion compensation unit 214 executes motion compensation based on the information of the motion vector input from the motion prediction unit 212 and the still image data input from the still image recording and reproducing block 130. For instance, the motion compensation unit 214 specifies the reference block of the still image data corresponding to the reference block of the frame recorded at the predetermined time based on the motion vector. The motion compensation unit 214 generates the image data (created picture CP') by combining the specified reference block of the still image data and the reference block of the frame recorded at the predetermined time at a predetermined ratio.
[0046]
When generating the created picture CP(t+h), the image generation circuit 202 develops the created picture CP'(t+h")(t n, M > m, AT > t. Furthermore, the variable h is assumed to satisfy
the relationship 0 < h < AT.
[0059]
The method of generating the created picture CP(t+h) of time t+h will be
described below with reference to the example of FIG. 3. The processing method of the image processing apparatus 100 according to the present embodiment is not limited thereto. The created picture CP(t+h) can be generated with reference to also the high
resolution picture HRP(t+AT) of other time t+AT (not shown). In this case, the image
processing apparatus 100 can perform the process of selecting the high resolution picture HRP to be referenced according to the magnitude, speed and the like of the motion of the subject.
[0060]
(SI)
First, the image processing apparatus 100 executes the motion prediction using the low resolution pictures LRP(t) and LRP(t+h") by the motion prediction unit 212. In this case, the motion prediction unit 212 magnifies the low resolution pictures LRP(t) and LRP(t+h") to generate the magnified pictures MP(t) and MP(t+h"). The motion prediction unit 212 then compares the magnified pictures MP(t) and MP(t+h") to detect the motion vector MV between MP(t) and MP(t+h"). The motion prediction method includes block matching method, phase correlation method, optical flow method, or the like.
[0061] (S2)
The image processing apparatus 100 then executes the motion compensation using the motion vector MV and the high resolution picture HRP(t) by the motion compensation unit 214. The motion compensation unit 214 uses the motion vector MV to specify the reference block of the high resolution picture HRP(t) corresponding to the reference block of the magnified picture MP(t+h"). The motion compensation unit 214 combines such reference block and the reference block of the magnified picture MP(t+h") at a predetermined ratio, and generates an intermediate created picture CP' (t+h").
[0062] (S3)
The image processing apparatus 100 then executes the motion prediction using the low resolution pictures LRP(t+h) and LRP(t+h") by the motion prediction unit 212. In this case, the motion prediction unit 212 magnifies the low resolution pictures LRP(t+h) to generate the magnified pictures MP(t+h). The motion prediction unit 212 then compares the magnified pictures MP(t+h") and MP(t+h) to detect the motion vector MV between MP(t+h") and MP(t+h). [0063] (S4)
The image processing apparatus 100 then executes the motion compensation using the motion vector MV and the intermediate created picture CP' (t+h") by the motion compensation unit 214. The motion compensation unit 214 uses the motion vector MV to specify the reference block of the created picture CP' (t+h") corresponding to the reference block of the magnified picture MP(t+h). The motion compensation unit 214 combines such reference block and the reference block of the magnified picture MP(t+h) at a predetermined ratio, and generates an ultimate created picture CP (t+h).
[0064]
If the time interval h "between the created picture CP(t+h) and the high resolution picture HRP(t) is very short, noise barely generates in the created picture CP(t+h) due to
the motion prediction accuracy even if motion compensation is performed using such high resolution picture HRP(t). However, if the time interval h is long, the possibility the noise generates in the information of the high frequency component contained in the high resolution picture HRP is high. That is, there is a possibility the correlation between the reference block of the high resolution picture HRP used in the motion compensation and the reference block of the magnified picture MP corresponding thereto may lower depending on the accuracy of the motion prediction.
[0065]
Generally, the process of motion prediction is such that more accurate motion prediction can be performed the lesser the motion of the subject, that is, the smaller the temporal distance from the prediction source to the prediction destination. Therefore, the accuracy of the prediction is obviously more satisfactory in the motion prediction performed from the magnified picture MP(t+h") to the magnified picture MP(t) than in the motion prediction performed from the magnified picture MP(t+h) to the magnified picture MP(t). The quality of the created picture CP'(t+h") generated using the motion prediction result from the magnified picture MP(t+h") to the magnified picture MP(t) is thus higher than the created picture generated using the motion prediction result from the magnified picture MP(t+h) to the magnified picture MP(t).
[0066]
Furthermore, as shown in the example of FIG. 4, the image generation circuit 202 can also reference an image other than the image positioned on the temporally back side when generating the created picture CP' or CP. An example of performing the motion prediction and the motion compensation with reference to the image positioned on the temporally back side has been described, but the image positioned on the front side may be referenced. Both front and back images may be referenced. The image generation circuit 202 can also generate the created picture CP' or CP with reference to a plurality of images positioned on the temporally front side or back side. The image generation circuit 202 can also perform the process of sequentially generating an intermediate created picture CP' at a very fine time interval to reach the ultimate created picture CP.
[0067]
Various modifications can be made in the image generation circuit 202 according
to the present embodiment. For instance, if the high resolution picture HRP(t+AT)
exists at time t+AT(h bmax, the image generation circuit 202 terminates the image generation
process. If b < b_max, the motion prediction unit 212 executes motion prediction from a current frame (MP(t+h")) to a key frame (MP(t)) or from a current frame (MP(t+h)) to the frame from which high resolution image is already generated (MP(t+h")) (S208). The reference frame of the motion prediction depends on the position of the reference source frame of the sequentially executed motion prediction. The image generation circuit 202 then determines whether or not the execution of the motion compensation is possible based on the magnitude of a prediction residual signal (S210).
[0077]
If determined that the execution of the motion compensation is not possible in step S210, the motion compensation unit 214 outputs the magnified picture MP(t+h) of step S202 as the created picture CP(t+h) (S216). If determined that the execution of the motion compensation is possible in step S210, the motion compensation unit 214
executes a differential image determination (determination of "D>Th" or "D
Documents
|