Specification
[Name of Document] Specification [Title of the Invention]
Control Method, Control Apparatus and Control Program for Photographing Apparatus [Technical Field]
The present invention relates to a control method, a control apparatus and a control program, which are provided for a photographing apparatus. More particularly, the present invention relates to a control method, a control apparatus and a control program, which are each designed with a capability of determining a photographing mode for obtaining a sharper image, to serve respectively as a control method, control apparatus and control program provided for a photographing apparatus such as a camera. [Background Art]In recent years, digital cameras have been becoming a main stream of cameras. In a digital camera including an image-pickup device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Mental Oxide Semiconductor), an image (hereinafter, referred to as taken image) produced by the image pickup device is displayed on a monitor such as an LCD (Liquid Crystal Display) device employed in the digital camera so as toallow the user to confirm the image. Then, an image signal of the taken image is digitalized before being transferred to typically an image processing section employed in a personal computer (PC) through a recording medium such as a flash memory. As an alternative, the taken image is transmitted to a PC by a wired communication using a cable or a radio communication using an infrared ray or the like. In the personal computer serving as a destination of the transmission of the taken image, the image taken by the digital camera is displayed on a monitor such as a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display) display section of the personal computer so as to allow the user to verify or edit the taken image.If a digital camera is used to take a picture of a photographing object, which is not sufficiently clear, for example in a shadow area, at a place with no enough sunlight, in a slightly dark room or at a similar location, it is necessary to decrease the speed of a shutter employed in the digital camera or lengthen the exposure time of the photographing object in order to provide enough exposure of the photographing object to the digital camera.In an image-pickup operation with such a longexposure time, the digital camera is typically fixed on a tripod or the like so as to prevent the digital camera from shaking or trembling. Thus, it is possible to obtain a taken image free of blurring and with proper exposure according to the brightness of the photographing object. When the digital camera is used to take an image of a photographing object by for example holding the camera by a hand, however, the camera shakes due to trembling of the hand. Then, if the digital camera trembles while the shutter of the camera is in an opened state or while the photographing object is being exposed to the camera, the resulting taken image is inadvertently a blurring image caused by the trembling of the digital camera. Such a blurring image is referred to as a hand-shaken image or known as an image resulting from camera trembling.In addition to the technique to decrease the shutter speed or increase the exposure time, as a method to obtain sufficient exposure, there has been proposed a method to give exposure, which is equivalent to exposure provided by the technique to increase the exposure time, by merely summing up a plurality of taken images cumulatively. For more information on this proposed method, refer to documents such as Patent Document 1. With the method disclosed in Patent Document 1,
however, a plurality of taken images is merely summed up cumulatively. Thus, if a hand holding the digital camera trembles as described above, the camera will produce a blurring image as is the case with the technique to increase the exposure time.As a method for preventing a taken image from blurring or a hand-shaken image from being produced even if a hand holding the digital camera trembles, there has been introduced a method adopted by a digital camera made by for example Canon Corporation. This method is referred to as an IS (Image Stabilizer).
With the Image Stabilizer, a pre-sensor is provided in an optical-system lens as a sensor for detecting trembling, or vibration of the digital camera. Then, in accordance with a digital signal representing the detected trembling or vibration of the digital camera, a portion of a lens group serving as a correction optical system is moved in a direction perpendicular to the optical axis so as to refract the ray of light in a direction canceling the trembling of the taken image.In accordance with the Image Stabilizer, it is possible to suppress image trembling due to shaking of a hand holding the digital camera or due to infinitesimal vibration shaking caused by a blowing wind as shaking of
the base of the photographing apparatus. As a result, the user can be provided with a sharp taken image.With the Image Stabilizer, however, it is necessary to provide a dedicated sensor for detecting trembling and a mechanism for moving a portion of a lens group serving as a correction optical system at a high speed. Thus, the Image Stabilizer raises a problem of a complicated structure of the digital camera and a problem of a high cost to manufacture the camera.As another method for avoiding a hand-shaken image, there is known a method by which shift quantities of the second and all subsequent ones among a plurality of taken images are each detected as the quantity of a shift from the first taken image, the positions of the second and all subsequent taken images are then corrected by their respective quantities of the shifts and, finally, the shifted second and all subsequent taken images are sequentially added to the first taken image. For more information on this method, the reader is suggested to refer to documents such as Patent Documents 2, 3, 4, 5, 6, 7 and 8. In accordance with the method disclosed in Patent Documents 2, 3, 4, 5, 6, 7 and 8, an interpolated image having a data array identical with the first taken image is created by interpolation based on each of thesecond and all subsequent taken images completing a process to correct the positions of the images and the interpolated image is then simply added to the first taken image in pixel units.In accordance with the method disclosed in Patent Documents 2, 3, 4, 5, 6, 7 and 8, since images are taken consecutively at a high speed and, hence, at a short exposure time, the amount of blurring is small even though a dark picture is resulted in. In order to solve the problem of a dark picture, an image created by interpolation based on the second and all subsequent taken images is added to the first taken image to result in a finally obtained image having brightness similar to an image taken with a proper exposure time.
In the process to create an image created by interpolation based on the second and all subsequent taken images in accordance with the method described in Patent Documents 2 to 8, chrominance signals (or color data) are subjected to an interpolation process adopting an interpolation method using a variety of interpolation functions such as the linear interpolation and the Bi-Cubic interpolation. The chrominance signals include an R (Red) signal (representing red-color data), a G (Green) signal (representing green-color data) and a B (Blue) -
signal (representing blue-color data), which pertain to one pixel.[Patent Document 1]Japanese Patent Laid-open No. Hei 05-236442 [Patent Document 2]Japanese Patent Laid-open No. 2000-217032 [Patent Document 3]Japanese Patent Laid-open No. 2000-224460 [Patent Document 4]Japanese Patent Laid-open No. 2000-244803 [Patent Document 5]
Japanese Patent Laid-open No. 2000-244797 [Patent Document 6]Japanese Patent Laid-open No. 2000-069352 [Patent Document 7]
Japanese Patent Laid-open No. Hei 10-341367 [Patent Document 8]
Japanese Patent Laid-open No. Hei 09-261526 [Disclosure of the Invention] [Problems to be Solved by the Invention]By the way, photographing modes adopted in a digital camera as a mode for carrying out a photographing operation may include an ordinary photographing mode and a hand-trembling correction mode. The ordinaryphotographing mode is a mode for taking an ordinary image and outputting the taken image as it is as a result of the photographing operation. On the other hand, the hand-trembling correction mode is a mode, in accordance with which, a plurality of images is taken consecutively at a high speed by using a high-speed shutter and the taken images are then summed up to produce an image with a proper exposure or an exposure desired by the photographer and with hand trembling corrected. Nevertheless, a method to determine whether the ordinary photographing mode or the hand-trembling correction mode is to be adopted in the photographing operation is not disclosed.
Therefore, a decision to select either the ordinary photographing mode or the hand-trembling correction mode for an existing scene (or an existing condition) is left entirely to a judgment formed by the photographer. As a result, it is quite within the bounds of possibility that the photographer makes a wrong decision unless the photographer is a well experienced one. For this reason, the digital camera is required to include an embedded automatic processing mechanism for determining which photographing mode is to be selected as a mode more suitable for an existing scene (or an existing condition).
In order to solve the problems described above, the present invention provides a capability of automatically determining a photographing mode that is capable of producing a sharper image. [Means for Solving the Problems]
A control method provided by the present invention for a photographing apparatus is characterized in that, in accordance with the control method, at least one of a first exposure time, which is a computed exposure time of a photographing operation to take an input image on the assumption that the photographing operation is to be carried out in the first mode, and a second exposure time, which is a computed exposure time of a photographing operation to take a plurality of input images on the assumption that the photographing operation is to be carried out in the second mode, is compared with a predetermined threshold value in order to produce a result of determination as to whether to carry out a photographing operation in a first mode selected as a photographing mode or carry out a photographing operation in a second mode selected as the photographing mode.The control method may include: a firstdetermination step of producing a result of determination as to whether or not the first exposure time is equal to or shorter than a first threshold value; a first decision step of making a decision to take an image in the first mode if the determination result produced at the first determination step indicates that the first exposure time is equal to or shorter than the first threshold value; and a second decision step of making a decision to take an image in the second mode if the determination result produced at the first determination step indicates that the first exposure time is neither equal to nor.shorter than the first threshold value.The control method may: further include a second determination step of producing a result of determination as to whether or not the second exposure time is equal to or shorter than a second threshold value if the determination result produced at the first determination step indicates that the first exposure time is neither equal to nor shorter than the first threshold value; and at the second decision step, make a decision to take an image in the second mode if the determination result produced at the second determination step indicates that the second exposure time is equal to or shorter than .the second threshold value.The second determination step may further include a step of producing a result of determination as to whether
or not the second exposure time is equal to or longer than the larger one of a threshold value based on a proper exposure time found from the brightness of a photographing object and a threshold value based on an image-pickup interval of successive image-pickup operations carried out by the photographing apparatus at a highest speed, and the second decision step may be a step of making a decision to take an image in the second mode if the determination result produced at the second determination step indicates that the second exposure time is equal to or shorter than the second threshold value and that the second exposure time is equal to or longer than the larger one of the threshold value based on a proper exposure time found from the brightness of a photographing object and the threshold value based on an image-pickup interval of successive image-pickup operations carried out by the photographing apparatus at a highest speed.The control method may include: a determination step of producing a result of determination as to whether or not the second exposure time is equal to or longer than a threshold value based on a proper exposure time found from the brightness of a photographing object; a first decision step of making a decision to take an imagein the second mode if the determination result produced at the determination step indicates that the second exposure time is equal to or longer than the threshold value; and a second decision step of making a decision to take an image in the first mode if the determination result produced at the determination step indicates that the second exposure time is neither equal to nor longer than the threshold value.The control method may include: a determination step of producing a result of determination as to whether or not the second exposure time is equal to or longer than a threshold value based on an image-pickup interval of successive image-pickup operations carried out by the photographing apparatus at a highest speed; a first decision step of making a decision to take an image in the second mode if the determination result produced at the determination step indicates that the second exposure time is equal to or longer than the threshold value; and a second decision step of making a decision to take an image in the first mode if the determination result produced at the determination step indicates that the second exposure time is neither equal to nor longer than the threshold value.A control apparatus provided by the presentinvention is characterized in that, in the control apparatus, at least one of a first exposure time, which is a computed exposure time of a photographing operation to take an input image on the assumption that the photographing operation is to be carried out in the first mode, and a second exposure time, which is a computed exposure time of a photographing operation to take a plurality of input images on the assumption that the photographing operation is to be carried out in the second mode, is compared with a predetermined threshold value in order to produce a result of determination as to whether to carry out a photographing operation in a first mode selected as a photographing mode or carry out a photographing operation in a second mode selected as the photographing mode.A control program provided by the present invention as a program to be executed by a computer is characterized in that, in accordance with the control program, at least one of a first exposure time, which is a computed exposure time of a photographing operation to take an input image on the assumption that the photographing operation is to be carried out in the first mode, and a second exposure time, which is a computed exposure time of a photographing operation to take aplurality of input images on the assumption that the photographing operation, is to be carried out in the second mode is compared with a predetermined threshold value in order to produce a result of determination as to whether to carry out a photographing operation in a first mode selected as a photographing mode or carry out a photographing operation in a second mode selected as the photographing mode.In accordance with the control method, the control apparatus and the control program, which are provided by the present invention, at least one of a first exposure time and a second exposure time is compared with a predetermined threshold value in order to produce a result of determination as to whether to carry out a photographing operation in a first mode selected as a photographing mode or carry out a photographing operation in a second mode selected as the photographing mode, wherein: the first exposure time is a computed exposure time of a photographing operation to take an input image on the assumption that the photographing operation is to be carried out in the first mode; and the second exposure time is a computed exposure time of a photographing operation to take a plurality of input images on the assumption that the photographing operation is to becarried out in the second mode. [Effects of the Invention]In accordance with the present invention, it is possible to dynamically determine a photographing mode capable of obtaining a sharper image. [Brief Description of the Drawings] [FIG. 1]FIG. 1 is a block diagram showing a typical configuration of an embodiment implementing a digital camera 1 to which the present invention is applied. [FIG. 2]FIG. 2 shows a flowchart referred to in explanation of photographing processing carried out by the digital camera 1 shown in FIG. 1. [FIG. 3]FIG. 3 is a diagram showing a two-dimensional layout of pixels of an image-pickup device 4 employed in the digital camera 1 shown in FIG. 1. [FIG. 4]FIG. 4 is a block diagram showing a detailed typical configuration of a signal processing circuit 7 employed in the digital camera 1 shown in FIG. 1. [FIG. 5]FIG. 5 is a diagram showing a first taken image.
[FIG. 6]FIG. 6 is a diagram showing a second taken image. [FIG. 7]FIG. 7 is a diagram showing a third taken image. [FIG. 8]FIG. 8 is an explanatory diagram showing pixel values related to the first taken image in a coordinate system. [FIG. 9]FIG. 9 is a diagram showing an output image. [FIG. 10]FIG. 10 is a diagram showing a reference coordinate system including plotted positions of pixels. [FIG. 11]FIG. 11 is an explanatory diagram referred to in explanation of a process to infer the light quantity Lg (I', J') of the green color at a position (I', J'). [FIG. 12]FIG. 12 shows a flowchart referred to in explanation of processing to generate an image. [FIG. 13]FIG. 13 is a diagram showing the waveform of a Cubic function. [FIG. 14]FIG. 14 is an explanatory diagram showing pixel values at positions in a reference coordinate system. [FIG. 15]
FIG. 15 is an explanatory diagram showing an exception state. [FIG. 16]FIG. 16 is a diagram referred to in explanation of exception processing of a G signal. [FIG. 17]FIG. 17 is a diagram referred to in explanation of exception processing of a G signal. [FIG. 18]FIG. 18 is a diagram referred to in explanation of exception processing of an R signal. [FIG. 19]FIG. 19 is a diagram referred to in explanation of exception processing of an R signal. [FIG. 20]FIG. 20 shows a flowchart referred to in explanation of processing to generate an image. [FIG. 21]FIG. 21 shows a flowchart referred to in explanation of processing to generate an image. [FIG. 22]FIG. 22 shows a flowchart referred to inexplanation of processing to find a light quantity Lg (I', J') of the green color. [FIG. 23]FIG. 23 shows a flowchart referred to inexplanation of processing to find a light quantity Lr (I', J') of the red color. [FIG. 24]FIG. 24 shows a flowchart referred to inexplanation of processing to find a light quantity Lb (I', J') of the blue color. [FIG. 25]FIG. 25 is a diagram showing taken images 4011 to 4018. [FIG. 26]FIG. 26 is an explanatory diagram showing an output image for a case in which the first taken image is used as a reference image. [FIG. 27]
FIG. 27 is an explanatory diagram showing an output image for a case in which a middle taken image is used as a reference image. [FIG. 28]FIG. 28 shows a flowchart referred to inexplanation of processing carried out by the digital camera 1 shown in FIG. 1. [FIG. 29]FIG. 29 is a diagram showing the plan view of a reference picture. [FIG. 30]FIG. 30 shows a flowchart referred to in explanation of processing to generate an image. [FIG. 31]FIG. 31 shows a flowchart referred to in explanation of processing to generate an image. [FIG. 32]FIG. 32 shows a flowchart referred to in explanation of processing to generate an image. [FIG. 33]FIG. 33 shows a flowchart referred to in explanation of processing to generate an image. [FIG. 34]FIG. 34 shows a flowchart referred to in explanation of processing to generate an image. [FIG. 35]FIG. 35 shows a flowchart referred to in explanation of processing to generate an image. [FIG. 36]
FIG. 36 shows a flowchart referred to in explanation of processing to generate an image. [FIG. 37]FIG. 37 shows a flowchart referred to in explanation of processing to generate an image. [FIG. 38]FIG. 38 shows a flowchart referred to in explanation of processing to generate an image. [FIG. 39]FIG. 39 shows a flowchart referred to in explanation of processing to generate an image. [FIG. 40]
FIG. 40 shows a flowchart referred to in explanation of processing to generate an image. [FIG. 41]FIG. 41 shows a flowchart referred to in explanation of processing to infer a pixel value of a G signal (or a light quantity of the green color). [FIG. 42]FIG. 42 shows a flowchart referred to inexplanation of processing to infer a pixel value of an R signal (or a light quantity of the red color). [FIG. 43]FIG. 43 shows a flowchart referred to in
explanation of processing to infer a pixel value of a B signal (or a light quantity of the blue color). [FIG. 44]
FIG. 44 shows a flowchart referred to in explanation of photographing processing carried out by the digital camera 1 shown in FIG. 1. [FIG. 45]
FIG. 45 shows a flowchart referred to in explanation of processing to determine a photographing mode. [FIG. 46]FIG. 46 shows a flowchart referred to in explanation of first determination processing. [FIG. 47]
FIG. 47 shows a flowchart referred to in explanation of second determination processing. [FIG. 48]FIG. 48 shows a flowchart referred to in explanation of third determination processing. [FIG. 49]FIG. 49 shows a flowchart referred to in explanation of fourth determination processing. [FIG. 50]FIG. 50 is a diagram referred to in explanation of
effects of a blanking period on an output image. [Description of Reference Characters]1: digital camera, 2: lens, 3: diaphragm, 4: image-pickup device, 5: correlation double sampling circuit, 6: A/D converter, 7: signal processing circuit, 8: timing generator, 9: D/A converter, 10: video encoder, 11: monitor, 12: codec, 13: memory, 14: bus, 15: CPU, 16: input device, 17: image processing circuit, 18: control circuit, 19: exposure meter, 21: shift circuit, 221 to 22N: frame memory, 231 to 23N_1: motion-vector detection circuit, 24: processing circuit, 25: controller. [Best Mode for Carrying out the Invention]Embodiments of the present invention are explained by referring to diagrams as follows.FIG. 1 is a block diagram showing a typical configuration of an embodiment implementing a digital (still) camera 1 to which the present invention is applied.The digital camera 1 shown in FIG. 1 includes a lens 2, a diaphragm 3, an image-pickup device 4, a correlation double sampling circuit 5, an A/D (Analog/Digital) converter 6, a signal processing circuit 7, a timing generator 8, a D/A (Digital/Analog) converter 9, a video encoder 10, a monitor 11, a codec 12, a memory
13, a bus 14, a CPU (Central Processing Unit) 15, a storage section 15A, an interface (I/F) 15B, an input device 16 and an exposure meter 19.The correlation double sampling circuit 5, the A/D converter 6, the signal processing circuit 7, the D/A converter 9, the video encoder 10 and the codec 12 form the image processing circuit 17 whereas the timing generator 8, the CPU 15, the storage section 15A and the interface 15B form the control circuit 18. The A/D converter 6 includes a shift circuit 21 while the signal processing circuit 7 includes a frame memory 22.
A light beam coming from a photographing object not shown in the figure hits the image-pickup device 4 after passing through an optical system including the lens 2 and the diaphragm 3. The lens 2 is a near-focus lens having a fixed focal distance or a zoom lens having a variable focal distance. If a zoom lens is used as the lens 2, the lens 2 is normally designed as a lens group including a plurality of lenses. By varying positional relations oriented in the direction of an optical axis as relations among the lenses composing the lens group, the focal distance of the lens 2 can be changed. Control to change the focal distance in this way is executed by a control signal output by the control circuit 18 to a
driving circuit for driving the lens 2. It is to be noted that the driving circuit itself is not shown in the figure. It is also worth noting that the lens 2 is shown in FIG. 1 as a single lens in order to simplify the drawing.The diaphragm 3 is a component for adjusting the quantity of light hitting the image-pickup device 4 by shielding some of the light passing through the lens 2. The control of the diaphragm 3 to adjust the quantity of light is executed by a control signal output by the control circuit 18.The image-pickup device 4 is typically a single-board sensor made of a material such as a CCD or a CMOS. The image-pickup device 4 has a predetermined number of pixels each serving as a light receiving element. In accordance with an exposure timing signal output by the timing generator 8, the image-pickup device 4 receives incoming light from the photographing object only during a predetermined exposure time and at predetermined intervals. Then, the image-pickup device 4 carries out an opto-electrical conversion process to convert the quantity of light arriving at the light receiving elements provided on an image-pickup face of the image-pickup device 4 into an electrical signal. Subsequently,the image-pickup device 4 supplies the electrical signal obtained as a result of the opto-electrical conversion process to the correlation double sampling circuit 5. Since the image-pickup device 4 is a single-plate sensor, the electrical signal supplied to the correlation double sampling circuit 5 is a chrominance signal (or color data) selected among R, G and B signals for every pixel.As an alternative, the image-pickup device 4 can also be an image-pickup device referred to as a binning image-pickup device having a function known as a binning function. The binning function is a function to sum up pixel values of a plurality pixels adjacent to each other inside the image-pickup device 4, which is a sensor for receiving light, and use the sum obtained as a result of the summing-up operation as the pixel value of one pixel. When the binning function is executed, the image-pickup device 4 sums up pixel values of a plurality of pixels adjacent to each other and outputs the sum obtained as a result of the summing-up operation as the pixel value of one pixel. Thus, the number of pixels each represented by such an output pixel value is smaller than the number of pixels actually composing the image-pickup device 4. To be more specific, the number of pixels each represented by an such output pixel value is a fraction of the numberof pixels actually composing the image-pickup device 4.To put it concretely, let us assume for example that the image-pickup device 4 sums up pixel values of a plurality of (or 22) pixels adjacent to each other (that is, two pixels arranged in the vertical direction and two pixels arranged in the horizontal direction) and outputs the sum by execution of a function referred to hereafter as the 2x2 binning function. In this case, the number of pixels represented each by such.a sum is one-fourth of the number of pixels actually composing the image-pickup device 4. This is because the number of pixels arranged in the vertical direction is reduced to a half and the number of pixels arranged in the horizontal direction is also reduced to a half. As another example, the image-pickup device 4 sums up pixel values of a plurality of (or 3x3) pixels adjacent to each other (that is, three pixels arranged in the vertical direction and three pixels arranged in the horizontal direction) and outputs the sum by execution of a function referred to hereafter as the 3x3 binning function. In this case, the number of pixels each represented by such a sum is one-ninth of the number of pixels actually composing the image-pickup device 4. This is because the number of pixels arranged in the vertical direction is reduced to one-third and the
number of pixels arranged in the horizontal direction is also reduced to one-third.When the binning function is executed as described above, the number of pixels each represented by a sum output by the image-pickup device 4 decreases, reducing the amount of data to be processed by the digital camera 1. In comparison with a case in which the binning function is not executed, that is, a case in which the number of pixels processed by the image-pickup device 4 is equal to the number of pixels actually composing the image-pickup device 4, the continuous image-pickup speed can be increased. In comparison with the continuous image-pickup speed of a case in which the binning function is not executed, for example, the continuous image-pickup speed of the 2x2 binning function is four times faster. This is because the number of pixel values output by the image-pickup device 4 is reduced to one-fourth. By the same token, the continuous image-pickup speed of the 33 binning function is nine times faster because the number of pixel values output by the image-pickup device 4 is reduced to one-ninth.That is to say, let us assume that the lower limit of the continuous image-pickup interval of the image-pickup device 4 carrying out a high-speed image-pickupprocess is to or the continuous image-pickup speed is I/ to. In this case, the image-pickup device 4 is capable of carrying out an image-pickup process at intervals of t0 seconds or intervals longer than t0 seconds without execution of the binning function. By execution of the 2×2 binning function, however, the image-pickup device 4 is capable of carrying out an image-pickup process at intervals of to/4 seconds or intervals longer than to/4 seconds. By execution of the 3x3 binning function, however, the image-pickup device 4 is capable of carrying out an image-pickup process at intervals of t0/9 seconds or intervals longer than to/9 seconds.It is to be noted that a binning control signal output by the control circuit 18 to the image-pickup device 4 controls a process to determine whether or not the binning function is to be executed and, if the binning function is to be executed, the control signal indicates whether to execute the 2x2 binning function or the 3x3 binning function.
In one photographing operation or in one operation carried out on a release button, the image-pickup device 4 produces one taken image at a shutter speed for a proper exposure or for an exposure desired by the photographer, that is, during an exposure time, or the
image-pickup device 4 produces a plurality of taken images at a speed higher than the shutter speed for a proper exposure or for an exposure desired by the photographer, that is, during a shorter exposure time. In the following description, the number of taken images produced at the higher speed is N.In the following description, the photographing mode of a photographing operation carried out to generate an output image from a taken image (or an input image) produced by the image-pickup device 4 or the photographing mode of a photographing operation carried out to output a taken image (or an input image) produced by the image-pickup device 4 as it is referred to as an ordinary photographing mode or a first photographing mode. On the other hand, the photographing mode of a photographing operation carried out to generate an output image from N taken images (or N input images) produced by the image-pickup device 4 at the higher speed cited.above is referred to as a hand-trembling correction photographing mode or a second photographing mode.
In the ordinary photographing mode, a taken image produced by the image-pickup device 4 is taken at a proper exposure or an exposure desired by the photographer. In the hand-trembling correction mode, onthe other hand, each of the N taken images is taken in an exposure time shorter than the exposure time for the proper exposure or the exposure desired by the photographer. Thus, each of the N taken images produced by the image-pickup device 4 is darker than an image taken at the proper exposure or has a brightness equal to the one-Mkth of the brightness of the image taken at the proper exposure, that is, has a brightness equal to the 1/Mk of the brightness of the image taken at the proper exposure where k is an integer in the range 1 to N.For too large values of Mk, the taken image is extremely dark so that each of the N taken images is inevitably buried among noises. In addition, an output image generated from N taken images buried among noises as such is also unavoidably an image containing a large number of noises. In order to solve this problem, an upper limit Mmax is determined as the maximum of the values of Mk. That is to say, Mmax is an Mk maximum value that still results in brightness not causing an image to be buried among noises. It is thus possible to generate a proper output image with few noises from N taken images each having a brightness at least equal to the l/Mmax of the brightness of the image taken at the proper exposure. The brightness at least equal to the l/Mmax of the
brightness of the image taken at the proper exposure is thus an upper limit of the darkness of the N taken images
Let us assume for example that Mmax is 8 . In this case, it is not possible to generate a proper output image from N consecutively taken images each having a darkness greater than 1/8 of the darkness of the image taken at the proper exposure, that is, it is not possible to generate a proper output image from N consecutively taken images each taken in an exposure time shorter than the exposure time of the proper exposure. This is because the N taken images are each extremely dark. Conversely, it is possible to generate a sharp output image from N consecutively taken images each having a brightness greater than 1/8 of the brightness of the image taken at the proper exposure, that is, it is possible to generate a proper output image from N consecutively taken images each taken in an exposure time longer than the exposure time of the proper exposure. It is to be noted that the values of Mk and Mmax are each a value determined by the performance of the digital camera 1.In the image processing circuit 17, the correlation double sampling circuit 5 is a circuit for eliminating noise components from an image signal, which is generated by the image-pickup device 4 as an electrical signal, by
adoption of a correlation double sampling technique. The correlation double sampling circuit 5 then supplies the image signal with noises eliminated from it to the A/D converter 6. The A/D converter 6 is a component for carrying out an A/D conversion process to convert the image signal received from the correlation double sampling circuit 5 as an image signal with noises eliminated from it into a digital signal. That is to say, the A/D converter 6 is a component for quantizing the image signal by adoption of a sampling technique.
In the ordinary photographing mode, the A/D converter 6 supplies a taken image represented by the digital signal obtained as a result of the A/D conversion process typically as it is to the D/A converter 9 or the codec 12 by way of the signal processing circuit 7. In the hand-trembling correction photographing method, on the other hand, the shift circuit 21 embedded in the A/D converter 6 shifts a taken image, which is represented by the digital signal obtained as a result of the A/D conversion process as a dark image, by typically n' bits to multiply the image by Mk in order to convert the image into a taken image represented by an image signal as an image having a brightness (or a value) similar to a taken image of the normal exposure. The process to convert the
taken image represented by a digital signal into an image having a brightness (or a value) similar to a taken image of the normal exposure is referred to as a gain-up process. The A/D converter 6 then supplies the image obtained as a result of the gain-up process to the signal processing circuit 7.In the correlation double sampling circuit 5, noise components of the image signal are eliminated but not all the noise components are eliminated completely. Thus, the image signal still includes noise components that cannot be removed by the correlation double sampling circuit 5. If the image signal still includes noise components that cannot be removed by the correlation double sampling circuit 5, the noise components become an error of the signal. In this case, in the shift circuit 21, the error is multiplied by Mk as the image signal is. Thus, the error included in the image signal supplied to the signal processing circuit 7 depends on the gain-up quantity used in the shift circuit 21. Let us assume that the quantity of noise components not eliminated by the correlation double sampling circuit 5 is E. In this case, the image signal supplied from the shift circuit 21 employed in the A/D converter 6 to the signal processing circuit 7 includes noise components having a quantity of about ExMk.It is possible to set an assumed maximum value dependent on the characteristics of the image-pickup device 4 as the maximum value of the noise quantity E. For Mk = 8, for example, the bit count n' by which the image signal is shifted by the shift circuit 21 is 3. By shifting the image signal by n' bits, it is possible to generate an output image having a brightness equal to the brightness for the proper exposure.By multiplying the image signal by Mk in the shift circuit 21 employed in the A/D converter 6, the image signal of N taken images each completing the gain-up process to give the same brightness as the proper exposure is temporarily stored (or recorded) in the frame memory 22 employed in the signal processing circuit 7.
In the ordinary photographing mode, the signal processing circuit 7 carries out the same image processing as an ordinary digital camera on a taken picture received from the A/D converter 6 and supplies the taken picture completing the image processing to the D/A converter 9 or the codec 12 as an output image.In the hand-trembling correction photographing mode, on the other hand, the signal processing circuit 7 stores N taken images received from the A/D converter 6 in the frame memory 22 and carries out predetermined imageprocessing on image signals representing the N taken images.To put it in detail, the signal processing circuit 7 takes for example the first one of the N taken images as a reference image and each of the second to Nth ones as target images. Then, the signal processing circuit 7 determines how much each of the target images has been shifted from the reference image. That is to say, the signal processing circuit 7 detects the quantity of a shift between the reference image and each of the target images in order to determine a positional relation between the reference image and each of the target images Subsequently, on the basis of the shift quantities, the signal processing circuit 7 finds an output image having all G, R and B signals for every pixel as a sharp output image completing a process to correct hand trembling. Finally, the signal processing circuit 7 supplies an image signal representing the sharp output image to the D/A converter 9 or the codec 12. The signal processing circuit 7 can be implemented typically by a DSP (Digital Signal Processor).When the ordinary photographing mode is adopted as the photographing mode, the timing generator 8 supplies an exposure timing signal to the image-pickup device 4,the correlation double sampling circuit 5, the A/D converter 6 and the signal processing circuit 7 in order to drive the image-pickup device 4, the correlation double sampling circuit 5, the A/D converter 6 and the signal processing circuit 7 to produce a taken image in one photographing operation. When the hand-trembling photographing mode is adopted as the photographing mode, on the other hand, the timing generator 8 supplies an exposure timing signal to the image-pickup device 4, the correlation double sampling circuit 5, the A/D converter 6 and the signal processing circuit 7 in order to drive the image-pickup device 4, the correlation double sampling circuit 5, the A/D converter 6 and the signal processing circuit 7 to produce N taken images at a high speed at predetermined intervals in one photographing operation.
The D/A converter 9 is a component for carrying out a D/A conversion process to convert an image signal received from the signal processing circuit 7 as an image signal representing an output picture into an analog signal and supplying the analog signal to the video encoder 10. The video encoder 10 is a component for converting the analog signal received from the D/A converter 9 as an analog signal, which has resulted from
the D/A conversion process to convert an image signal, into a video signal that can be displayed on the monitor 11 and supplying the video signal to the monitor 11. The monitor 11 is a component for typically playing the role of the finder of the digital camera 1. The monitor 11 is typically an LCD device for displaying a video signal received from the video encoder 10. Thus, the monitor 11 displays the output image.The codec 12 is a component for coding an image signal from the signal processing circuit 7 as an image signal of an output picture in accordance with a predetermined coding method such as a JPEG (Joint Photographic Experts Group) method, an MPEG (Moving Picture Experts Group) method or a DV (Digital Video) method, and supplying a signal obtained as a result of the coding process to the memory 13.The memory 13 is a storage device used for storing (or recording) an image signal, which is received from the codec 12 as a result of the coding process, temporarily or permanently. It is to be noted that, as a substitute for the memory 13, a recording medium such as a magnetic disk or an optical disk can be used. The memory 13 and the substitute for the memory 13 can be mounted onto and removed from the digital camera 1. It isalso worth noting that the digital camera 1 may be provided with both a recording medium embedded in the digital camera 1 and a recording medium that can be mounted onto and removed from the digital camera 1.In accordance with typically a control signal supplied by the input device 16 by way of the bus 14, for example, the CPU 15 employed in the control circuit 18 outputs signals to a variety of components of the digital camera 1 by way of the bus 14 in order to request the components to carry out various kinds of processing. Examples of the control signal supplied to the CPU 15 include a photographing-mode control signal, a binning control signal and an exposure timing signal. The photographing-mode control signal is a signal indicating whether the ordinary photographing mode or the hand-trembling correction photographing mode is to be adopted as the photographing mode in which a photographing operation is to be carried out. The binning control signal is a signal for controlling the binning function, that is, a signal for specifying that an image-pickup process is to be carried out without execution of the binning function or an image-pickup process is to be carried out by execution of the 2×2 binning function or the 3×3 binning function for example. The exposure timingsignal is a signal for controlling the timing of receiving lights by the image-pickup device 4 at a photographing, that is, the timing of a light receiving start time and end time, further in the hand-trembling correction mode, the timing of a light receiving start time and end time for each Nth taken images.The storage section 15A is typically a non-volatile memory such as a flash memory or an EEPROM. As an alternative, the storage section 15A is a combination of a non-volatile memory and a volatile memory. The storage section 15A is used for storing a program to be executed by the CPU 15 and temporarily storing data required by the CPU 15 in carrying out processing. The interface 15B is typically a USB (Universal Serial Bus) or an IEEE 1394 interface. By connecting the interface 15B to an external computer, the digital camera 1 is capable of updating a program stored in the storage section 15A with a program received from the external computer and exchanging various kinds of data including output pixels with the external computer.The input device 16 has operation buttons such as the release button cited earlier, an exposure correction dial and a zoom button (or a zoom lever). The release button is a button for providing a photographing trigger.The exposure correction dial is a dial for correcting the exposure of a photographing operation. The zoom button is a button to be operated to set a focal distance of the zoom lens in order to adjust the zoom. A variety of signals, which are generated when the user operates the operation buttons, is supplied by the input device 16 to the CPU 15 by way of the bus 14. The CPU 15 controls the other components to carry out processing according to the signals supplied by the input device 16 to the CPU 15 by way of the bus 14. It is to be noted that one or more operation buttons employed in the input device 16 can be displayed on the monitor 11. An operation button is displayed on themonitor 11 typically as a transparent tablet and can be operated by detecting the tablet.The exposure meter 19 is a component for measuring the brightness of the photographing object (or the image-pickup object) and supplying a value obtained as a result of the measurement to the control circuit 18. On the basis of the measured value of the brightness, the CPU 15 employed in the control circuit 18 determines the value of the diaphragm (or the value of the iris) and the exposure time, which are used in automatic exposure control and other applications.By the way, when the photographer desires the so-called under or over photographing operation to be carried out by using the digital camera 1, the photographer operates the exposure correction dial of the input device 16 in order to set an exposure correction value. In this way, the exposure for an actual photographing operation can be corrected to the proper exposure. That is to say, by operating the exposure correction dial, the photographer is capable of setting an exposure for a photographing operation of the proper exposure, deliberately setting the exposure at a value on the under side so as to result in an intentionally darkened output image or deliberately setting the exposure at a value on the over side so as to result in an intentionally brightened output image. Information generated in the setting operation as an exposure correction value is supplied by the input device 16 to the CPU 15 employed in the control circuit 18 to be used as the basis of processing carried out by the CPU 15.With the lens 2 used as a zoom lens, the photographer is capable of setting a focal distance by operating the zoom button employed in the input device 16 of the digital camera 1. That is to say, information generated as a result of an operation carried out on the zoom button is supplied by the input device 16 to the
control circuit 18. In accordance with the information supplied by the input device 16, the control circuit 18 then controls lenses composing the lens 2 to set the focal distance set by the photographer. In addition, the control circuit 18 also uses the information supplied by the input device 16, that is, the information on focal distance at photographing, in processing to determine a photographing mode of a photographing operation to be described later.The control circuit 18 is a circuit for determining a proper diaphragm value F and a proper exposure time Tp from a value measured by the exposure meter 19 as the value of the brightness of a photographing object in the same way as an ordinary digital camera. The control circuit 18 also determines an actual exposure time Ta also referred to as a shutter speed from a corrected exposure value output by the exposure correction dial, which is operated to set the corrected exposure value.That is to say, with the exposure correction dial set at 0, the control circuit 18 takes the proper exposure time Tp as it is as the exposure time Ta. In this case, when the image-pickup device 4 produces a taken image at a diaphragm value F and the exposure time Ta, the taken image produced by the image-pickup process
is an image having a proper brightness.With the exposure correction dial set at a value representing a 1-stage under brightness, the control circuit 18 sets the exposure time Ta at Tp/2. In this case, when the image-pickup device 4 produces a taken image at a diaphragm value F and the exposure time Ta, the taken image produced by the image-pickup process is an image with the 1-stage under brightness. It is to be noted that, if the actual exposure time in this case is set at 2> 1, the process to change the size of the photographing-object portion projected at the position (X2, Y2) by a magnification L2 is referred to as an enlargement process. For L2 < 1, on the other hand, the process to change the size of the photographing-object portion projected at the position (X2, Y2) by a magnification L2 is referred to as a shrinking process.It is to be noted that, in most cases of hand trembling, the hand trembles (or is shaken) in a direction parallel to the light reception face of the image-pickup device 4 and, only in few cases of hand trembling, does the hand tremble (or is the hand shaken)in a direction perpendicular to the light reception face of the image-pickup device 4. Thus, it is possible to assume that there is no hand trembling in a direction perpendicular to the light reception face of the image-pickup device 4. In this case, L2 = 1.By the same token, the motion-vector detection circuit 232 eceives the first taken image used as the reference image from the frame memory 221 and the third taken image serving as a target image from the frame memory 223.Then, for every selected pixel of the third taken image, the motion-vector detection circuit 232 detects which position in the first taken image corresponds to a selected pixel of the third taken image. Then, on the basis of the detection result, the motion-vector detection circuit 232 finds transformation parameters (a3, b3, c3, d3, s3 and t3) defining affine transformation of Eg. (2) expressing a positional relation between the first taken image and the third taken image and supplies the parameters to the processing circuit 24(Figure Removed)position included in a third coordinate system on the third taken image as the position of a specific pixel of the third taken image. On the other hand, coordinates (X1(3), Y1(3)) of a position included in the first coordinate system on the first taken image as the position of the same portion of the photographing object as the portion corresponding to the specific pixel are coordinates, which are obtained as a result of a process to transform the coordinates (X3, Y3) of the position of the specific pixel on the third taken image when the position of the specific pixel is mapped onto the position in the first coordinate system on the first taken image. The inferior figures (3) attached to the coordinates (X1(3), Y1(3)) of the position (X1(3), Y1(3)) in the first coordinate system on the first taken image indicate that the coordinates (X1(3), Y1(3)) are coordinates obtained as a result of transforming the coordinates (X3, Y3) of the position (X3, Y3) in the third coordinate system on the third taken image in a process of mapping the position (X3, Y3) onto the position (X1(3), Y1(3)). Define the parameters a3, b3, c3 and d3 of the transformation parameters (a3, b3, c3, d3, s3 and t3) as follows:a3 = d3 = L3cos3, and -b3 = c3 = L3sin93.In this case, the affine transformation process expressed by Eq. (2) can be said to be a process defined by a rotation angle 63, a scale L3 and a parallel shift quantity (s3, t3) in the same way as Eq. (1) is defined as described above.Thereafter, by the same token, the motion-vector detection circuit 23k-1 receives the first taken image used as the reference image from the frame memory 221and the kth taken image serving as a target image from the frame memory 22k.For every selected pixel of the kth taken image, the motion-vector detection circuit 23k_1 detects which position on the first taken image corresponds to the pixel and, on the basis of the result of the detection, finds transformation parameters (ak, bk, ck, dk, sk and tk) defining affine transformation of Eq. (3) expressing a positional relation between the first taken image and the kth taken image and supplies the parameters to the processing circuit 24.
(Figure Removed)
In Eq. (3), coordinates (Xk, Yk) represent aposition included in a kth coordinate system on the kthtaken image as the position of a specific pixel of thekth taken image. On the other hand, coordinates (X1(k), Y1(k) of a position included in the first coordinate system on the first taken image as the position of the same portion of the photographing object as the portion corresponding to the specific pixel are coordinates, which are obtained as a result of a process to transform the coordinates (Xk, Yk) of the position of the specific pixel on the kth taken image when the position of the specific pixel is mapped onto the position in the first coordinate system on the first taken image. The inferior figures (k) attached to the coordinates (X1(k), Y1(k)) of the position (X1(k), y1(k))in the first coordinate system on the first taken image indicate that the coordinates (X1(k), Y1(k)) are coordinates obtained as a result of transforming the coordinates (X3, Y3) of the position (Xk, Yk) in the kth coordinate system on the kth taken image in a process of mapping the position (Xk, Yk) onto the position (X1(k), Y1(k)). Let us define the parameters ak, bk, ck and dk of the transformation parameters (ak, bk, ck/ dk, sk and tk) as follows:ak = dk = Lkcos9k, and -bk = ck = Lksin6k.In this case, the affine transformation process expressed by Eq. (3) can be said to be a process defined by a rotation angle 9k, a scale Lk and a parallel shiftquantity (Sk, tk) in the same way as Eq. (1) is defined as described above.As described above, the transformation parameters (ak, bk, Ck, dk, sk and tk) are found from a result of detecting a position included in the first coordinate system on the first taken image as the position of the same photographing-object portion as the photographing-object portion projected at a pixel position in the kth coordinate system on the kth taken image for every position on the kth taken image. As an alternative, the transformation parameters (ak, bk, ck, dk, sk and tk) can also be found in the so-called mechanical way from signals output by sensors provided on the digital camera 1. Examples of the sensors are an acceleration sensor and an angular-speed sensor.The processing circuit 24 is a circuit for receiving N taken images from the frame memories 221 to 22N. The processing circuit 24 also receives the transformation parameters (ak, bk, ck, dk, sk and tk) representing a relation between the position of the first taken image and the position of the kth taken image from the motion-vector detection circuits 231 to 23N-1.On the basis of the transformation parameters (ak, bk, ck, dk, sk and tk) received from the motion-vectordetection circuits 231 to 23N-1 as parameters representing a relation between the position of the first taken image and the position of each of the second to Nth taken images, the processing circuit 24 identifies pixels of the first to Nth taken images as pixels to be used in inference of pixel values of pixels on an output image in a process to generate the output image as will ,be described later. Then, on the basis of the pixel values of the identified pixels, the processing circuit 24 infers pixel values (that is, the R, G and B signals) of a sharp output image with its hand trembling corrected in the process to generate the output image. Subsequently, the processing circuit 24 supplies the output image obtained as a result of the image generation process to the D/A converter 9 or the codec 12.Each of the N taken images supplied by the A/D converter 6 to the signal processing circuit 7 is an image, every pixel of which has a pixel value, which is either one of the R, G and B signals. On the other hand, the output image generated by the processing circuit 24 is an image with every pixel thereof having three pixel values, i.e., the R, G and B signals, which are each a chrominance signal.In accordance with control executed by the CPU 15,the controller 25 controls components such as the frame memories 221 to 22N, the motion-vector detection circuits 231 to 23N_1and the processing circuit 24, which are employed in the signal processing circuit 7 . It is to be noted that, as a substitute for the controller 25, the CPU 15 employed in the digital camera 1 shown in FIG. 1 may also control the components such as the frame memories 221 to 22N, the motion-vector detection circuits 231 to 23N_1 and the processing circuit 24, which are employed in the signal processing circuit 7. In this case, the controller 25 can be eliminated.It is to be noted that, in the case of a single-plate sensor adopting the Bayer two-dimensional array, the number of pixels each generating an R signal and the number of pixels each generating a B signal are small in comparison with the number of pixels each generating a G signal. Thus, the number of errors (or noises) for the R signal in the output image generated by the signal processing circuit 7 and the number of errors (or noises) for the B signal in the same output image are greater than the number errors (or noises) for the G signal in some cases. In such a case, the noises can be eliminated or the number of noises can be reduced by providing a low-pass filter at a stage following the processingcircuit 24 as a filter for limiting the bandwidth of high-frequency components of only the chrominance signal with the luminance signal passed as it is.The following description explains the processing circuit 24 employed in the signal processing circuit 7, which is shown in FIG. 4 as a component of the digital camera 1 shown in FIG. 1.It is to be noted that the pixel value of every pixel on the image-pickup device 4 (that is, the pixel value of a taken image) is a signal representing the light quantity of a light beam radiated to a point on the pixel from the object of photographing. An example of the point on the pixel is the gravitational center (or the geometrical center) of the pixel. That is to say, the pixel value of every pixel is used as data obtained as a result of a point sampling process carried out at the gravitational-center position of the pixel.In the following description, a kth taken image is also referred to simply as a kth image. In addition, an XY coordinate system taking the kth image as a reference is referred to as the coordinate system of the kth image. Strictly speaking, an XY coordinate system taking the kth image as a reference is an XY coordinate system taking the center of the pixel at the left upper corner of the
kth image as an origin, the horizontal (or right) direction as the X direction and the vertical (or downward) direction as the Y direction.When the pixel value obtained at every pixel of the image-pickup device 4 is used as data obtained as a result of a point sampling process carried out at the gravitational-center position of the pixel as described above, the pixel value obtained from a pixel located at the intersection of the ith column from the left end and the jth row from the top as one of pixels of the two-dimensional layout shown in FIG. 3 as the layout of the image-pickup device 4 corresponds to the light quantity of a light beam radiated by a photographing-object portion projected at a position expressed by coordinates (i-1, j-1) as typically the position of the gravitational center of the pixel located at the intersection of the ith column and the jth row.Let us assume for example that the pixel value obtained from a pixel located at the intersection of the ith column from the left end and the jth row from the top as one of pixels of the two-dimensional layout shown in FIG. 3 as the layout of the image-pickup device 4 is equal to the light quantity of a light beam radiated by a photographing-object portion projected at a positionexpressed by coordinates (i-1, j-1) as the position of the gravitational center of the pixel located at the intersection of the ith column and the jth row. In this case, for example, the pixel value Gobs (1, ig, jg) of a pixel located at the intersection of the ith column from the left end and the jth row from the top of the first taken image is the light quantity Lg (ig-1, jg-1) of the green color at the position (ig-1, jg-1) in the coordinate system of the first taken image, that is, the coordinate system taking the first taken image as the reference. By the same token, the pixel value Robs (1, ir, jr) of a pixel located at the intersection of the ith column from the left end and the jth row from the top of the first taken image is the light quantity Lr (ir-1, jr-1) of the red color at the position (ir-1, jr-1) in the coordinate system of the first taken image. In the same way, the pixel value Bobs (1, ib, jb) of a pixel located at the intersection of the ith column from the left end and the jth row from the top of the first taken image is the light quantity Lb (ib-1, jb-1) of the blue color at the position (ib-1, jb-1) in the coordinate system of the first taken image.FIG. 5 is a diagram showing the first taken image.Pixels shown in FIG. 5 as pixelsofatakenimageare denoted by using the same notations as those used in FIG. 3 showing the two-dimensional array of pixels on the image-pickup device 4. For a pixel G(jg-l) (ig-1) on the first taken image, a pixel value Gobs (1, ig, jg) of the G signal is observed at the gravitational center of the pixel G(jg-l)(ig-1) as shown in the figure as a black circle. By the same token, for a pixel R(jr-l)(ir-1) on the first taken image, a pixel value Robs (1, ir, jr) of the R signal is observed at the gravitational center of the pixel R(jr-l) (ir-1) as shown in the figure as a black square. In the same way, for a pixel B(jb-l)(ib-1) on the first taken image, a pixel value Bobs (1, ib, jb) of the B signal is observed at the gravitational center of the pixel B(jb-l) (ib-1) as shown in the figure as a black triangle.As described above, the pixel value of every pixel on the first taken image is observed at the gravitational position of the pixel in the coordinate system of the first taken image. To be more specific, the pixel value of a pixel located at the intersection of the ith column and the jth row is observed at the position (i-1, j-1).FIG. 6 is a diagram showing the second taken image.For a pixel G(jg-l)(ig-1) on the second taken image shown in FIG. 6, a pixel value Gobs(2, ig, jg) of the Gsignal is observed at the gravitational center of the pixel G(jg-l) (ig-1) as shown in the figure as a black circle. By the same token, for a pixel R(jr-l) (ir-1) on the second taken image, a pixel value Robs (2, ir, jr) of the R signal is observed at the gravitational center of the pixel R(jr-l) (ir-1) as shown in the figure as a black square. In the same way, for a pixel B(jb-l)(ib-1) on the second taken image, a pixel value Bobs (2, ib, jb) of the B signal is observed at the gravitational center of the pixel B(jb-l)(ib-1) as shown in the figure as a black triangle.As described above, the pixel value of every pixel on the second taken image is observed at the gravitational position of the pixel in the coordinate system of the second taken image as is the case with the first taken image. To be more specific, the pixel value of a pixel located at the intersection of the ith column and the jth row is observed at the position (i-1, j-1).
As an output image, the processing circuit 24 employed in the signal processing circuit 7 shown in FIG. 4 generates an output image from the N taken images in an image generation process. To be more specific, the processing circuit 24 generates an output image in a range projected on a reference image as a range of the
photographing object. The reference image is a reference image used in a process to detect relations between the positions of the N taken images. In this embodiment, the first taken image is taken as the reference image. That is to say, in order to generate the output image, for the position of each pixel on the output image, the processing circuit 24 identifies pixels of the N taken images to be used in inference of the pixel value at the position of the pixel on the output image on the basis of the relations between the positions of the N taken images Then, the image-pickup device 4 finds the pixel values of the identified pixels on the N taken images. Finally, on the basis of the pixel values of the identified pixels on the N taken images, the image-pickup device 4 infers the pixel value at the position of each pixel on the output image.In order to identify pixels of the N taken images to be used in inference of the pixel value at the position of a pixel on the output image for every pixel (or the position of every pixel) on the output image as described above, the processing circuit 24 maps the source position of each pixel on each of the N taken images onto the destination position on the first taken image serving as the reference image, transforming thecoordinates of the source position into the coordinates of the destination position so as to make photographing-object portions projected on each of the N taken images coincide with their respective photographing-object portions projected on the reference image.The coordinates are transformed by carrying out an affine transformation process based on an affine transformation equation using transformation parameters (a, b, c, d, s and t) found by the motion-vector detection circuit 23k-1 employed in the signal processing circuit 7 shown in FIG. 4. Examples of the affine transformation equation are Eqs. (1) to (3) each representing a relation between two of N taken images. In the following description, each of the transformation parameters may be properly expressed as a variable without a suffix appended thereto.For example, the coordinates of a pixel (or a point) on the second taken image can be transformed into the coordinates of a pixel (or a point) on the first taken image in an affine transformation process expressed by Eq. (1) using the transformation parameters (a2, b2, c2, d2, s2 and t2) found by the motion-vector detection circuit 231.FIG. 6 also shows the gravitational positions onthe first taken image as a result of the process to transform the coordinates of the gravitational positions included in the second taken image as described above as the gravitational positions of pixels having pixel values Gobs (2, ig, jg), Robs (2, ir, jr) and Bobs (2, ib, jb). To put it in detail, the pixel value Gobs (2, ig, jg) of the G signal for the pixel G(jg-l)(ig-1) in the coordinate system of the second taken image is also the light quantity Lg (x, y) (that should be) observed at the position (jg-1)(ig-1) transformed into the position ((ig-1)(2>/ (jg-l)(2)) in the coordinate system of the first taken image as the light quantity of the green color. That is to say, the coordinates ((ig-l)(2), (jg-l)(2>) are coordinates obtained as a result of an affine transformation process to transform the coordinates of a position (ig-1, jg-1) in the coordinate system of the second taken image by using the transformation parameters (a2, b2, C2, d2, s2 and t2) found by the motion-vector detection circuit 231. In the two-dimensional layout shown in FIG. 6, the position ((ig-l)(2), (jg-l){2)) included in the coordinate system of the first taken image as a position with its coordinates resulting from the affine transformation process to transform the coordinates of the position producing the pixel valueGobs (2, ig, jg) of the G signal is shown as a white circle.By the same token, the pixel value Robs (2, ir, jr) of the R signal for the pixel R(jr-l)(ir-1) in the' coordinate system of the second taken image is also the light quantity Lr (x, y) (that should be) observed at the position (jr-1) (ir-1) transformed into the position ((ir-1)(2), (jr-1) (2)) in the coordinate system of the first taken image as the light quantity of the red color. That is to say, the coordinates ((ir-l)(2), (jr-l)(2)) are coordinates obtained as a result of an affine transformation process to transform the coordinates of a position (ir-1, jr-1) in the coordinate system of the second taken image by using the transformation parameters (a2, b2, c2, d2, s2 and t2) found by the motion-vector detection circuit 231 . In the two-dimensional layout shown in FIG. 6, the position ((ir-l)(2), (jr-l)(2)) included in the coordinate system of the first taken image as a position with its coordinates resulting from the affine transformation process to transform the coordinates of the position producing the pixel value Bobs (2, ir, jr) of the R signal is shown as a white square.In the same way, the pixel value Bobs (2, ib, jb)of the B signal for the pixel B(jb-l)(ib-1) in the coordinate system of the second taken image is also the light quantity Lb (x, y) (that should be) observed at the position (jb-1) (ib-1) transformed into the position ( (ib-l)(2i, (jb-1) (2)) in the coordinate system of the first taken image as the light quantity of the blue color. That is to say, the coordinates ((ib-l)(2), (jb-l)(2)) are coordinates obtained as a result of an affine transformation process to transform the coordinates of a position (ib-1, jb-1) in the coordinate system of the second taken image by using the transformation parameters (a2, b2, c2, d2, s2 and t2) found by the motion-vector detection circuit 23i. In the two-dimensional layout shown in FIG. 6, the position ((ib-l)(2>, (jb-l)(2)) included in the coordinate system of the first taken image as a position with its coordinates resulting from the affine transformation process to transform the coordinates of the position producing the pixel value Bobs (2, ib, jb) of the B signal is shown as a white triangle.
FIG. 7 is a diagram showing the third taken image. For a pixel G(jg-l)(ig-1) on the third taken image shown in FIG. 7, a pixel value Gobs (3, ig, jg) of the G signal is observed at the gravitational center of the
pixel G(jg-l)(ig-1) as shown in the figure as a black circle. By the same token, for a pixel R(jr-l) (ir-1) on the third taken image, a pixel value Robs (3, ir, jr) of the R signal is observed at the gravitational center of the pixel R(jr-l)(ir-1) as shown in the figure as a black square. In the same way, for a pixel B(jb-l)(ib-1) on the third taken image, a pixel value Bobs (3, ib, jb) of the B signal is observed at the gravitational center of the pixel B(jb-l)(ib-1) as shown in the figure as a black triangle.As described above, the pixel value of every pixel on the third taken image is observed at the gravitational position of the pixel in the coordinate system of the third taken image as is the case with the first taken image. To be more specific, the pixel value of a pixel located at the intersection of the ith column and the jth row is observed at the position (i-1, j-1) .The coordinates of a pixel (or a point) on the third taken image can also be transformed into the coordinates of a pixel (or a point) on the first taken image in an affine transformation process expressed by Eq. (2) using the transformation parameters (a3, b3, c3, d3, s3 and t3) found by the motion-vector detection circuit 232 in the same way as the second taken image.FIG. 7 also shows the gravitational positions on the first taken image as a result of the process to transform the coordinates of the gravitational positions included in the third taken image as described above as the gravitational positions of pixels having pixel values Gobs (3, ig, jg), Robs (3, ir, jr) and Bobs (3, ib, jb). To put it in detail, the pixel value Gobs (3, ig, jg) of the G signal for the pixel G(jg-l)(ig-1) in the coordinate system of the third taken image is also the light quantity Lg (x, y) (that should be) observed at the position (ig-1, jg-1) transformed into the position ((ig-1)<3)A (jg-1) o)) in the coordinate system of the first taken image as the light quantity of the green color. That is to say, the coordinates ((ig-l)(3>, (jg-l)<3)) are coordinates obtained as a result of an affine transformation process to transform the coordinates of a position (ig-1, jg-1) in the coordinate system of the third taken image by using the transformation parameters (a3, b3, c3 d3, s3 and t3) found by the motion-vector detection circuit 232. In the two-dimensional layout shown in FIG. 7, the position ((ig-l)(3), (jg-l)(3>) included in the coordinate system of the first taken image as a position with its coordinates resulting from the affine transformation process to transform the
coordinates of the position producing the pixel value Gobs (3, ig, jg) of the G signal is shown as a white circle.By the same token, the pixel value Robs (3, ir, jr) of the R signal for the pixel R(jr-l)(ir-1) in the coordinate system of the third taken image is also the light quantity Lr (x, y) (that should be) observed at the position (ir-1, jr-1) transformed into the position ((ir-1)(3), (jr-1) (3)) in the coordinate system of the first taken image as the light quantity of the red color. That is to say, the coordinates ((ir-l)(3), (jr-l)(3)) are coordinates obtained as a result of an affine transformation process to transform the coordinates of a position (ir-1, jr-1) in the coordinate system of the third taken image by using the transformation parameters (a3, b3, c3, d3, s3 and t3) found by the motion-vector detection circuit 232. In the two-dimensional layout shown in FIG. 7, the position ((ir-l)(3), (jr-l)(3)) included in the coordinate system of the first taken image as a position with its coordinates resulting from the affine transformation process to transform the coordinates of the position producing the pixel value Robs (3, ir, jr) of the R signal is shown as a white square.
In the same way, the pixel value Bobs (3, ib, jb) of the B signal for the pixel B(jb-l)(ib-1) in the coordinate system of the third taken image is also the light quantity Lb (x, y) (that should be) observed at the position (ib-1, jb-1) transformed into the position ((ib-l)(3), (jb-1) (3)) in the coordinate system of the first taken image as the light quantity of the blue color. That is to say, the coordinates ((ib-l)(3), (jb-ljpj) are coordinates obtained as a result of an affine transformation process to transform the coordinates of a position (ib-1, jb-1) in the coordinate system of the third taken image by using the transformation parameters (a3, b3, c3, d3, s3 and ta) found by the motion-vector detection circuit 232. In the two-dimensional layout shown in FIG. 7, the position ((ib-l)(3), (jb-l)(3)) included in the coordinate system of the first taken image as a position with its coordinates resulting from the affine transformation process to transform the coordinates of the position producing the pixel value Bobs (3, ib, jb) of the B signal is shown as a white triangle.FIG. 8 is an explanatory diagram showing positions included in the coordinate system of the first taken image as destination positions with coordinates resultingfrom an affine transformation process to transform the coordinates of source positions, at which pixel values Gobs (1, ig, jg) to Gobs (N, ig, jg) of pixels G(jg-1)(ig-1) located on the first to Nth taken images respectively as pixels of the green color (or pixels each receiving a light beam of the green color) are observed, in a process to map the source position of each pixel on each of the N taken images onto the destination position on the first taken image serving as the reference image by carrying out the affine transformation process to transform the coordinates of the source position into the coordinates of the destination position so as to make photographing-object portions projected on each of the N taken images coincide with the photographing-object portion projected on the reference image.Let us pay attention to coordinates (I', J') in the coordinate system of the first taken image shown in FIG. 8. Typical pixels on the first to Nth taken images are located at positions with their coordinates transformed into the coordinates of the position represented by the coordinates (I', J') and the coordinates of positions around the position represented by the coordinates (I', J' ) as pixels of the green color (or pixels each receiving a light beam of the green color). The pixelvalue Gobs (1, ig, jg) of the first taken image is a pixel value observed at the gravitational position (or the center) of a typical pixel G(jg-l)(ig-1) located at the intersection of the ith column and the jth row on the first taken image. In this case, the coordinates (I', J') satisfy the following equations: I' = ig - 1, J' = jg - 1 At a position on the upper/left side of the center of the pixel G(jg-l) (ig-1), a pixel value Gobs (2, ig, jg) of a pixel on the second taken image is observed as a pixel value of a typical pixel, the coordinates of the position of which are transformed in an affine transformation process into the coordinates of a position close to the position represented by the coordinates (I', J') in the coordinate system of the first taken image. In addition, at a position on the lower/left side of the center of the pixel G(jg-l)(ig-1), a pixel value Gobs (3, ig, jg) of a pixel on the third taken image is observed as a pixel value of a typical pixel, the coordinates of the position of which are transformed in an affine transformation process into the coordinates of a position close to the position represented by the coordinates (I', J') in the coordinate system of the first taken image. Furthermore, at a position on the upper/right side of the center of the pixel G(jg-l)(ig-1), a pixel value Gobs (4, ig, jg)
of a pixel on the fourth taken image is observed as a pixel value of a typical pixel, the coordinates of the position of which are transformed in an affine transformation process into the coordinates of a position close to the position represented by the coordinates (I', J') in the coordinate system of the first taken image. It is to be noted that a pixel value Gobs (k, ig, jg) of a pixel on the kth taken image where k = 5 to N is not shown in the figure.The processing circuit 24 carries out an affine transformation process to transform the coordinates of the positions of pixels on the first to Nth taken images to the coordinates of the positions of pixels on the first taken image. Then, on the basis of pixel values Gobs (k, ig, jg) of the pixels, the coordinates of the positions of which have each been subjected to the affine transformation process, the processing circuit 24 infers the green-color light quantity Lg (i-1, j-1) of a pixel at coordinates in the coordinate system of the first taken image as the pixel value of the G signal at the position (i-1, j-1) on the output image where k = 1 to N.In this case, let notation (I', J') in the coordinate system of the first taken image serving as the reference image denote the coordinates of the centerposition (i-1, j-1) of a pixel located at the intersection of the ith column and the jth row. That is to say, the coordinates (I', J') satisfy the following equations: I' = i - 1, J' = j - 1. It is to be noted that notations I' and J' are each an integer at least equal to 0.FIG. 9 is a diagram showing positions in the coordinate system of the first taken image as positions at which the processing circuit 24 should infer the true green-color light quantity Lg (I', J' ) , the true red-color light quantity Lr (I', J') and the true blue-color light quantity Lb (I', J') as pixel values of pixels on the output image.In the two-dimensional array shown in FIG. 9, the center position (I', J') of each pixel on the output image (that is, the center position (I', J') included in the coordinate system of the first taken image as the center position (I', J') of each pixel on the image-pickup device 4), is shown as a black circle representing a position at which the light quantity Lg (I', J') of the green color, the light quantity Lr (I', J') of the red color or the light quantity Lb (I', J') of the blue color should be inferred. That is to say, at. the center positions (I', J') each shown by a black circle as thecenter position (I', J') of each pixel, the light quantity Lg (I', J') of the green color, the light quantity Lr (I', J') of the red color and the light quantity Lb (I', J') of the blue color can be inferred.
In the following description, the coordinate system of the reference image is referred to as a reference coordinate system. Since the first taken image is adopted as the reference image, the coordinate system of the first taken image is referred to as the reference coordinate system.As described above, the processing circuit 24 carries out an affine transformation process to transform the coordinates of the positions of pixels on the first to Nth taken images to the coordinates of the positions of pixels on the first taken image. Then, on the basis of pixel values Gobs (k, ig, jg) of the pixels, the coordinates of the positions of which have each been subjected to the affine transformation process, the processing circuit 24 infers the green-color light quantity Lg (I', J') of a pixel at the position (I', J' ) in the reference coordinate system as the pixel value of the G signal at the position (I', J') on the output image where k = 1 to N.If the processing circuit 24 infers the green-colorlight quantity Lg (I', J') of a pixel at the position (I', J' ) in the reference coordinate system as the pixel value of the G signal at the position (I', J') on the output image on the basis of all pixel values Gobs (k, ig, jg) of the pixels located on the first to Nth taken images as pixels, the coordinates of the positions of which have each been subjected to the affine transformation process, however, the precision of the inference will deteriorate.In order to solve this problem, the processing circuit 24 identifies only pixels located on the first to Nth taken images as pixels, the coordinates of which are transformed into coordinates of pixels in close proximity to the position (I', J') . The processing circuit 24 then takes the identified pixels as pixels to be used for inference of a pixel value of the output image, that is, for inference of the light quantity Lg (I', J') of the green color. The identified pixels located on the first to Nth taken images are pixels, the positions of which correspond to locations in close proximity to the position (I', J') as indicated by a result of the affine transformation process to transform coordinates in the coordinate systems of the first to Nth taken images into coordinates in the reference coordinate system. The position (I', J') is a position at which an attempt ismade to infer the light quantity Lg (I', J') of the green color by using the pixel values Gobs (k, ig, jg) of the G signals of the identified pixels located on the first to Nth taken images.FIG. 10 is a diagram showing the reference coordinate system including plotted positions of pixels located on the first to Nth taken images as pixels to be used by the processing circuit 24 to infer the light quantity Lg (I', J') of the green color.The processing circuit 24 sets a range surrounding a position (I', J') in the reference coordinate system as a range of values of x and y, which satisfy the relationsI'-l ≤ x < I'+l and J'-l ≤ y < J'+l. This range is referred to as a range close to the position (I', J') or a vicinity range of the position (I', J' ) . Then, the processing circuit 24 identifies pixels located on the first to Nth taken images as pixels having their positions mapped onto positions in the vicinity range of the position (I', J'). The processing circuit 24 takes the identified pixels as pixels to be used for inference of the light quantity Lg (I', J') of the green color. The pixels located on the first to Nth taken images as pixels having their positions mapped onto positions in the vicinity range of the position (I', J') are pixels, thepositions of which correspond to locations in the vicinity range of the position (I', J') as indicated by a result of the affine transformation process to transform coordinates in the coordinate systems of the first to Nth taken images into coordinates in the reference coordinate system. That is to say, for the position (I', J'), the processing circuit 24 finds all sets of integers (k, ig and jg) in which the integer k is a sequence number assigned to a taken image associated with transformation parameters (ak, bk, ck, dk, sk and tk) used in an affine transformation process to transform a position (ig-1, jg-1) in the coordinate system of the kth taken image into the position (x, y) located in the reference coordinate system as a post-transformation position with the coordinates (x, y) thereof satisfying the relations I'-l≤ x < I'+l and J'-1 ≤ y < J'+1 expressing relations with the coordinates (I', J') .In the coordinate system shown in FIG. 10, as positions with their coordinates subjected to an affine transformation process to transform the coordinates into the aforementioned coordinates (x, y) in the reference coordinate system, there are five points A, B, C, D and E on the first to Nth taken images. Thus, the processing circuit 24 identifies pixels located at the points A, B,C, D and E on the first to Nth taken images as pixels with their positional coordinates subjected to an affine transformation process, being transformed into the coordinates (x, y) satisfying the relations described above. The processing circuit 24 then uses the pixels in a process to infer the light quantity Lg (I', J') of the green color.To be more specific, the processing circuit 24 uses pixel values Gobs (k, ig, jg) observed at the points A, B, C, D and E on the first to Nth taken images to infer the light quantity Lg (I', J') of the green color at the point (I', J' ) . To put it in detail, the processing circuit 24 uses G-signal pixel values Gobs (k, ig, jg) of pixels located at the points A, B, C, D and E on the first to Nth taken images as pixels with their positional coordinates subjected to an affine transformation process, being transformed into the coordinates (x, y), in order to infer the light quantity Lg (I', J' ) of the green color at the point (I', J' ) .FIG. 11 is an explanatory diagram showing a model of a process to infer the light quantity Lg (I', J') of the green color at a position (I', J' ) by using the pixel values Gobs (k, ig, jg) observed at the points A to E. Typically, the processing circuit 24 infers thelight quantity Lg (I', J') by using the following equation: (Figure Removed)
Notation in Eq. (4) for the green color denotes a sum computed for a position (I', J') with respect to all sets of integers (k, ig and jg) in which the integer k is a sequence number assigned to a taken image associated with transformation parameters (ak, bk, ck, dk, sk and tk) used in an affine transformation process to transform a position (ig-1, jg-1) in the coordinate system of the kth taken image into the position (x, y) located in the reference coordinate system as a post-transformation position with the coordinates (x, y) thereof satisfying the relations I'-l ^ x <•!'+! and J'-l ^ y < J'+1 expressing relations with the coordinates (I', J'). In the typical cases shown in FIGS. 10 and 11, for example, the sum is computed with respect to five sets of integers (k, ig and jg) corresponding to the five points A to E.In addition, the term w ((x, y) , (I', J') ) in Eq. (4) is a weight having the position (x, y) and the pixel position (I', J') as arguments. The position (x, y) is a position included in the reference coordinate system as aposition with its coordinates resulting from an affine transformation process to transform the coordinates of a position (ig-1, jg-1) by using the transformation parameters (ak, bk, ck, dk, sk and tk) . On the other hand, the pixel position (I', J') is a position, the quantity light Lg (I', J') of the green color at which is to be inferred.Thus, in accordance with Eq. (4), the light quantity Lg (I', Y') of the green color is inferred as a quotient obtained by dividing the sum of weighted pixel values by the sum of weights, where each of the weighted pixel values is a product obtained by multiplying an actually observed pixel value Gobs (k, jg, ig) by one of the weights. In the following description, Eq. (4) is referred to as a weighed addition equation for the light quantity of the green color.The weight w ((x, y), (I', J')) can be typically a value monotonously decreasing in proportion to the distance between the position (x, y) and the position (I', J' ) . The distance between the position (x, y) and the position (I', J') can be expressed as a function of coordinates (x, y) and (I', J') as follows:F ((x, y), (I', J') ) = √ { (x - I')2 + (y - J')2} In this case, as a value monotonously decreasing inproportion to the distance between the position (x, y) and the position (I', J') , for example, the value of the following expression can be used:V2 - F ( (x, y) , (I' , J' ) )The term V2 of the expression V2 - F ((x, y), (I', J')) is a maximum distance between the position (x, y) and the position (I', J'), which is the center of a range defined by boundaries (I'll) and (J'±l) as a range including positions (x, y), the coordinates of which are obtained as a result of transforming coordinates (ig-1, jg-1) of a pixel with its pixel values Gobs (k, ig, jg) used for inferring the green-color light quantity Lg (I', J') at the center (I', J') as shown in FIGS. 10 and 11. Thus, the expression √2 - F ( (x, y), (I', J')) is a non-negative function monotonously decreasing in proportion to the distance between the position (x, y) and the position (I', J')•If the expression V2 - F ((x, y) , (I', J')) is used as the weight w ((x, y), (I', J')), the pixel value Gobs (k, ig, jg) of a pixel in close proximity to the position (I', J') has a great effect on the inferred green-color light quantity Lg (I', J') at the position (I', J') in such a way that, the closer the pixel to the position (I', J'), the greater the effect of the pixel value Gobs (k,ig, jg) of the pixel on the inferred green-color light quantity Lg (I', J') at the position (I', J').It is to be noted that, as described earlier by referring to FIG. 1, the shift circuit 21 carries out a gain-up process with a magnification of Hk on the image signal output by the correlation double sampling circuit 5 by shifting the image signal by n' bits. Thus, noise components included in the image signal are also multiplied by Mk. As a result, the resulting pixel value Gobs (k, ig, jg) includes noise components with a magnitude of ExMk where notation E denotes the magnitude of the noise components included in the image signal output by the correlation double sampling circuit 5.In the process to infer the light quantity Lg (I', J' ) of the green color, the effect of the noise components included in the pixel value Gobs (k, ig, jg) should be eliminated. From this noise-elimination point of view, it is thus desirable to use a weight w ((x, y), (I', J')) that reduces magnitude E×Mk of the noise components included in the pixel value Gobs (k, ig, jg). Accordingly, it is desirable to use a weight w ((x, y) , (I', J' )) that decreases in proportion to the distance between the position (x, y) and the position (I', J') and decreases in proportion to the magnitude E×Mk of the
noise components included in the pixel value Gobs (k, ig, jg) . An example of such a weight w ((x, y) , (I', J') ) is a weight represented by the expression {√2 - F ((x, y), (I' , J' ) ) } / (Ex× Mk) .
In addition, as another example of the weight w ((x, y), (I', J')), it is possible to use a function having the characteristics of a low-pass filter for the distance between the position (x, y) and the position (I', J') as will be described later.
The processing circuit 24 infers a light quantity Lr (I', J') of the red color at the point (I', J') and a light quantity Lb (I', J') of the blue color at the point (I', J') in the same way as the process to infer a light quantity Lg (I', J') of the green color at the point (I', J') . That is to say, the processing circuit 24 infers a light quantity Lr (I', J') of the red color at the point (I', J') and a light quantity Lb (I', J') of the blue color at the point (I', J') in accordance with Eqs. (5) and (6) given below in the same way as with Eq. (4). [Eq. (5)]
Notation £ in Eq. (5) denotes a sum computed for a position (I', J' ) with respect to all sets of integers (k, ir and jr) in which the integer k is a sequence number assigned to a taken image associated with transformation parameters (ak, bk, ck, dk, sk and tk) used in a transformation process to transform a position (ir-1, jr-1) in the coordinate system of the kth taken image into the position (x, y) located in the reference coordinate system as a post-transformation position with the coordinates (x, y) thereof satisfying the relations I'-l ^ x < I'+l and J'-1≤ y < J'+1 expressing relations with the coordinates (I', J' ) . By the same token, notation £ in Eq. (6) denotes a sum computed for a position (I', J') with respect to all sets of integers (k, ib and jb) in which the integer k is a sequence number assigned to a taken image associated with transformation parameters (ak, bk, gi<, dk, sk and tk) used in a transformation process to transform a position (ib-1, jb-1) in the coordinate system of the kth taken image into the position (x, y) located in the reference coordinate system as a post-transformation position with the coordinates (x, y) thereof satisfying the relations I'-l ≤ x < I'+l and J'-1
≤ y < J'+1 expressing relations with the coordinates (I', J' ) -
In the following description, Eq. (5) is properly referred to as the weighted addition equation of the light quantity of the red color whereas Eq. (6) is properly referred to as the weighted addition equation of the light quantity of the blue color.
As described above, for the position (I', J'), the processing circuit 24 identifies pixels each expressed by a set of integers (k, i and j) in which the integer k is a sequence number assigned to a taken image associated with transformation parameters (ak, bk, ck, dk, sk and tk) used in an affine transformation process to transform a position (i-1, j-1) in the coordinate system of the kth taken image into the position (x, y) located in the reference coordinate system as a post-transformation position with the coordinates (x, y) thereof satisfying the relations I'-l ≤ x < I'4-1 and J'-1 ≤ y < J'+1 expressing relations with the coordinates (I', J'). In the following description, the identified pixels are properly referred to as identified pixels. Then, the processing circuit 24 finds (or infers) the light quantity Lg (I', J') of the green color, the light quantity Lr (I', J') of the red color and the light
quantity Lb (I', J') of the blue color on the basis of pixel values of the identified pixels.
By referring to a flowchart shown in FIG. 12, the following description explains the image generation processing carried out at the step S4 of the flowchart of FIG. 2 to generate an output image by inferring pixel values (that is, the light quantity of the green color, the red color and the blue color) as described above.
The flowchart begins with a step S71 at which the processing circuit 24 pays attention to a position (I', J') in the reference coordinate system. In the following description, the position (I', J') to which attention is paid is referred to an observed position (I', J') - The observed position (I', J') represents the center of a pixel (i-1, j-1) at the intersection of the ith column and the jth row of the first taken image, which is the reference image.Then, the flow of the image generation processing goes on from the step S71 to a step S72. At this step, for the position (I', J'), the processing circuit 24 finds a set of integers (k, ig and jg) for each of the first to Nth taken images as a set of integers (k, ig and jg) in which the integer k is a sequence number assigned to a taken image associated with transformationparameters (ak, bk, ck, dk, sk and tk) used in an affine transformation process to transform a position (ig-1, jg-1) in the coordinate system of the kth taken image into the position (x, y) located in the reference coordinate system as a post-transformation position with the coordinates (x, y) thereof satisfying the relations I'-l ≤ x < I'+l and J' -1 ≤= y < J'+1 expressing relations with the coordinates (I', J' ) . Then, the processing circuit 24 identifies a pixel located at the position (ig-1, jg-1) on the kth taken image as a pixel associated with the set of integers (k, ig and jg). Subsequently, the flow of the image generation processing goes on to the next step S73. It is to be noted that the motion-vector detection circuit 23k-i provides the processing circuit 24 with the transformation parameters (ak, bk, ck, dk, sk and tk) used in an affine transformation process to transform the position (ig-1, jg-1) of the identified pixel in the coordinate system of the kth taken image into the position (x, y) in the reference coordinate system. For k = 1, the set of integers (k, ig and jg) is associated with the first taken image, which is associated with transformation parameters (a1, b1, g1, d1, S1 and ti) having values of (1, 0, 0, 1, 0, 0). Thus, the position in the reference coordinate system is virtually notsubjected to an affine transformation process.
In the following description, the position (x, y) obtained as a result of an affine transformation process to transform the position of a pixel on a kth taken image is properly referred to as a post-transformation position (x, y) .At the step S73, the processing circuit 24 creates a weighted addition equation expressed by Eq. (4) as a weighted addition equation for finding the light quantity of the green color by using all integer sets (k, ig and jg) found at the step S72. Then, the flow of the image generation processing goes on to the next step S74. To put in detail, by using the pixel values Gobs (k, ig, jg) of the identified pixels represented by all the integer sets (k, ig and jg) found at the step S72, the processing circuit 24 finds the value of the denominator £w((x, y), (I', J' )) of the expression on the right side of Eq. (4) representing a weighted addition equation for finding the light quantity of the green color and the value of the numerator ∑{w((x, y) , (I', J' ) ) ×Gobs (k, ig, jg)} of the expression.At the step S74, for the position (I', J'), the processing circuit 24 finds a set of integers (k, ir and jr) for each of the first to Nth taken images as a set ofintegers (k, ir and jr) in which the integer k is a sequence number assigned to a taken image associated with transformation parameters (ak, bk, ck, dk, sk and tk) used in an affine transformation process to transform a position (ir-1, jr-1) in the coordinate system of the kth taken image into the position (x, y) located in the reference coordinate system as a post-transformation position with the coordinates (x, y) thereof satisfying the relations I'-l ≤ x < I'-f-l and J'-l ≤ y < J'+1 expressing relations with the coordinates (I', J' ) . Then, the processing circuit 24 identifies a pixel located at the position (ir-1, jr-1) on the kth taken image as a pixel associated with the set of integers (k, ir and jr) . Subsequently, the flow of the image generation processing goes on to the next step S75.At the step S75, the processing circuit 24 creates a weighted addition equation expressed by Eq. (5) as a weighted addition equation for finding the light quantity of the red color by using all integer sets (k, ir and jr) found at the step S74. Then, the flow of the image generation processing goes on to the next step S76. To put in detail, by using the pixel values Robs (k, ir, jr) of the identified pixels represented by all the integer sets (k, ir and jr) found at the step S74 and the post-107transformation position (x, y), the processing circuit 24 finds the value of the denominator ∑w((x, y), (I', J')) of the expression on the right side of Eq. (5) representing a weighted addition equation for finding the light quantity of the red color and the value of the numerator ∑{w((x, y), (I', J'}} × Robs (k, ir, jr)} of the expression.At the step S76, for the position (I', J') , the processing circuit 24 finds a set of integers (k, ib and jb) for each of the first to Nth taken images as a set of integers (k, ib and jb) in which the integer k is a sequence number assigned to a taken image associated with transformation parameters (ak, bk, ck, dk, sk and tk) used in an affine transformation process to transform a position (ib-1, jb-1) in the coordinate system of the kth taken image into the position (x, y) located in the reference coordinate system as a post-transformation position with the coordinates (x, y) thereof satisfying the relations I'-l ≤ x < I'+l and J'-1 ≤ y < J'+l expressing relations with the coordinates (I', J' ) . Then, the processing circuit 24 identifies a pixel located at the position (ib-1, jb-1) on the kth taken image as a pixel associated with the set of integers (k, ib and jb). Subsequently, the flow of the image generation processing
goes on to the next step S77.At the step S77, the processing circuit 24 creates a weighted addition equation expressed by Eq. (6) as a weighted addition equation for finding the light quantity of the blue color by using all integer sets (k, ib and jb) found at the step S76. Then, the flow of the image generation processing goes on to the next step S78. To put in detail, by using the pixel values Bobs (k, ib, jb) of the identified pixels represented by all the integer sets (k, ib and jb) found at the step S76 and the post-transformation position (x, y), the processing circuit 24 finds the value of the denominator ∑w((x ,y), (I', J')) of the expression on the right side of Eq. (6) representing a weighted addition equation for finding the light quantity of the blue color and the value of the numerator ∑{w((x, y), (I', J' ) ) x Bobs (k, ib, jb) } of the expression.At the step S78, the processing circuit 24 finds (or infers) the light quantity Lg (I', J') of the green color at the observed position (I', J') by dividing the value of the numerator ∑ (w((x, y), (I', J')) * Gobs (k, ig, jg)} of the expression on the right side of Eq. (4) by the value of the denominator ∑w((x, y), (I', J')) of the expression. The values of the numerator and thedenominator have been found at the step S73. In addition, the processing circuit 24 finds (or infers) the light quantity Lr (I', J') of the red color at the observed position (I', J') by dividing the value of the numerator ∑ {w((x, y), (I', J')) X Robs (k, ir, jr)} of the expression on the right side of Eq. (5) by the value of the denominator ∑w((x, y), (I', J' ) ) of the expression. The values of the numerator and the denominator have been ' found at the step S75. On top of that, the processing circuit 24 finds (or infers) the light quantity Lb (I', J') of the blue color at the observed-position (I', J') by dividing the value of the numerator ∑ {w((x, y), (I', J')) x Bobs (k, ib, jb)} of the expression on the right side of Eq. (6) by the value of the denominator ∑w((x, y), (I', J')) of the expression. The values of the numerator and the denominator have been found at the step S77. Subsequently, the flow of the image generation processing goes on to the next step S79.Thus, at the step S78, three pixel values of the G, R and B signals respectively at the position (I', J') on the output image are found.At the step S79, the processing circuit 24 produces a result of determination as to whether or not all positions (I', J') have been observed, that is, whetheror not the light quantity Lg (I', J') of the green color, the light quantity Lr (I', J') of the red color and the light quantity Lb (I', J') of the blue color have been found for the center points of all pixels on the output image or all pixels of the first taken image used as the reference image.If the determination result produced at the step S79 indicates that not all positions (I', J') have been taken as an observed position, the flow of the image generation processing goes back to the step S71 to repeat the execution of the processes of the steps S71 to S79. That is to say, the processing circuit 24 takes a position (I', J'), which has not been observed yet, as a new observed position (I', J') and finds the light quantity Lg (I', J') of the green color, the light quantity Lr (I', J') of the red color and the light quantity Lb (I', J') of the blue color at the new observed position (I', J').If the determination result produced at the step S79 indicates that all positions (I', J') have been taken as an observed position, on the other hand, the flow of the image generation processing goes on to the step S80. At the step S80, the processing circuit 24 generates (or obtains) an output image that has the light quantity Lg(I', J'), the light quantity Lr (I', J') and the light quantity Lb (I', J'), which have been found at the step S78 for the green color, the red color and the blue color respectively at every observed position (I', J'), as pixel values of the G, R and B signals respectively. The processing circuit 24 then supplies the output image to the D/A converter 9 or the codec 12 before returning control of the execution.As described above, relations between the positions of a plurality of taken images obtained as a result of a high-speed image-pickup process are detected and, on the basis of the positional relations, a pixel on each of a plurality of taken images is identified for the position of each pixel on the output image as an identified pixel to be used for inference of the pixel value of the pixel on the output image. Then, on the basis of the pixel value of every identified pixel, the pixel value of the pixel on the output image is inferred. The processes to identify pixels and infer a pixel value of a pixel on the output image on the basis of the pixel values of the identified pixels are carried out for every pixel on the output image to generate the output image. Thus, it is possible to obtain a sharp output image with no effects of hand trembling.In the above description, a weight w ((x, y), (I', J')) that decreases in proportion to the distance between the position (x, y) and the position (I', J') is taken as a weight in Eqs. (4) to (6) used for inferring the light quantity Lg (I', J') of the green color, the light quantity Lr (I', J') of the red color and the light quantity Lb (I', J' ) of the blue color respectively. An example of such a weight w ((x, y) , (I', J')) is a weight represented by the expression {√2 - F ( (x, y) , (I', J' ) ) } As the weight w ((x, y) , (I', J') ), however, it is also possible to use a function provided with the characteristic of a low-pass filter as a characteristic with respect to the distance between the position (x, y) and the position (I', J') .An example of the function provided with the characteristic of a low-pass filter as a characteristic with respect to the distance between the position (x, y) and the position (I', J') is the expression Cubic (I' -x) x Cubic (J' - y) expressed in terms of the Cubic function.Cubic (z) expressed by Eq. (7) below is a Cubic function.[Eq. (7)];a+2)z|3-(a+3)|z|2+l (|z| < 1)
Cubic (z) = J a|z|3 - 5a|z|2 + 8a|z| - 4a (1 < | z < 2)
0 (2 < |z|)
(7)
It is to be noted that notation 'a' used in Eq. (7) is a constant determined in advance. An example of such a constant is -1.
FIG. 13 is a diagram showing the waveform of the Cubic function referred to as Cubic (z).
The value of the Cubic function referred to as Cubic (z) is 0 for the variable z satisfying the equations 2 ≤|z| and z| =1. The value of the Cubic function referred to as Cubic (z) is negative for 1 < |z| < 2 but the value of the Cubic function referred to as Cubic (z) is positive for z| < 1. In addition, as the value of |z| increases, the value of the Cubic function referred to as Cubic (z) decreases. In other words, the Cubic function referred to as Cubic (z) is a function having the characteristic of a low-pass filter if the axis representing the variable z serves as the frequency axis of a Fourier transformation process.
If the value of the Cubic function referred to as Cubic (z) is used as the weight w ((x, y), (I', J'}}, the processing circuit 24 employed in the signal processing
circuit 7 shown in FIG. 4 infers the light quantity Lg (I', J') of the green color, the light quantity Lr (I', J') of the red color and the light quantity Lb (I', J') of the blue color as the pixel values of a pixel on the output image as follows.
FIG. 14 is a diagram showing points Gil to G19 located on the reference coordinate system (or the coordinate system of the first taken image) as post-transformation positions (x, y) with the coordinates (x, y) obtained as a result of an affine transformation process to transform coordinates (ig-1, jg-1) of positions on the N taken images with respect to a position (I' , J' ) .
The points Gil to G19 coincide with the post-transformation positions (x, y) shown in FIG. 14 as positions with the coordinates (x, y) thereof satisfying conditions of I'-2 ≤ x < I'+2 and J'-2 ≤ y < J'4-2. The processing circuit 24 identifies pixels on the first to Nth taken images as identified pixels located at positions transformed into the post-transformation positions (x, y) in the range I'-2 ≤ x < I'+2 and J'-2 ≤ y < J'+2, which is the vicinity range of the position I', J' .
That is to say, if the expression {√/2 - F ( (x, y) ,
(I', J' )) } decreasing in proportion to the distance between the position (x, y) and the position (I', J') is taken as the weight w ((x, y), (I', J')), the range I'-l ≤s x < I'+l and J'-l ≤= y < J'+1 is used as the vicinity range of the position (I', J') as shown in FIGS. 10 and 11. If the expression Cubic (I' - x) x Cubic (J' - y) expressed in terms of the Cubic function is taken as the weight w ((x, y), (I', J'}}, on the other hand, the range I'-2 ≤ x < I'+2 and J'-2 ≤ y < J'+2 is used as the vicinity range of the position (I', J' ) as shown in FIG. 14.
As described earlier, Cubic (z) shown in FIG. 7 as a Cubic function has a value according to the argument z for -2 ≤ z ≤ +2 (even though Cubic (z) shown in FIG. 7 as a Cubic function has a value of 0 independently of the argument z for |z| > 2). Thus, pixel values of pixels located at positions transformed into post-transformation positions (x, y) in the range I'-2 ≤ x < I'+2 and J'-2 ≤ y < J'+2 corresponding to the range -2 ≤ z ≤ +2 are used for inferring the pixel value at the position (I', J') on the output image.
On top of that, as the weight w ((x, y) , (I', J' )), it is also possible to adopt a function that has the characteristic of a low-pass filter with respect to the
distance between the position (x, y) and the position (I', J' ) and decreases in proportion to the magnitude E×Mk of the noise components as explained earlier in the description of Eq. (4). In this case, the weight w ((x, y), (I', J')) is represented by the expression Cubic (I' - x) x Cubic (J' - y) / (E×Mk .It is to be noted that, in the shift circuit 21 employed in the digital camera 1 shown in FIG. 1, if all the N taken images are subjected to a uniform gain-up process with a magnification M, the weight w ( (x, y) , (I', J') ) is represented by the expression Cubic (I' - x) x Cubic (J' - y) / (E × m). In this case, the term l/(E×M) of the weight w ((x, y), (I', J')) in the numerator and denominator of each of Eqs. (4) to (6) cancel each other. Thus, the use of the expression Cubic (I' - x) x Cubic (J' - y) / (E × M) as the weight w ((x, y), (I', J')) in computation of the expression on the right side of each of Eqs. 4 to 6 is equivalent to the use of the expression Cubic (I' - x) x Cubic (J' - y) as the weight w ((x, y),I', J') ) .Substituting the term Cubic (I'-x) × Cubic (J'-y) for the weight w ((x, y), (I', J')) in Eq. (4) results in Eq. (8) below:[Eq. (8)]∑ Cubic (I'-x) × Cubic (J'-y) × Gobs (k, ig, jgLg(I' , J'2, Cubic (I'-x) × Cubic (J'-y)... (8)Notation ∑in Eq. (8) denotes a sum computed for a position (I', J') with respect to all sets of integers (k, ig and jg) in which the integer k is a sequence number assigned to a taken image associated with transformation parameters (ak, bk, ck, dk, sk and tk) used in an affine transformation process to transform a position (ig-1, jg-1) in the coordinate system of the kth taken image into the position (x, y) located in the reference coordinate system as a post-transformation position with the coordinates (x, y) thereof satisfying the relations I'-2 ≤ x < I'+2 and J'-2 ≤ y < J'+2 expressing relations with the coordinates (I', J') . In the typical case shown in FIG. 14, for example, the sum is computed with respect toK
sets of integers (k, ig and jg) corresponding to the 9 points Gil to G19.It is to be noted that, much like Eq. (4), Eq. (8) is also referred to as a weighed addition equation for the light quantity of the green color. The numerator and denominator of Eq. 8 expressing the light quantity of the green color are rewritten as expressions (9) and (10)respectively as follows.
[Eq. (9)]Z Cubic (I'-x) x Cubic(J'-y) x Gobs(k, ig, jg) (9)
[Eq. (10)]2 Cubic (I'-x) x cubic (J'-y) (10)By the same token, substituting the term Cubic (I' x) x Cubic (J'-y) for the weight w ((x, y), (I', J')) in Eqs. (5) and (6) results in respectively Eq. (11) and (12) below. [Eq. (11)]2 Cubic (I'-x) x Cubic (J'-y) x Robs (k, ir, j r )Lr(I' , J ,2 Cubic (I'-x) × Cubic (J'-y)... (11) [Eq. (12)}∑ Cubic (I'-x) × Cubic (J'-y) x Bobs (k, ib, jb)
Lb(I' , J' ) =
∑ Cubic (I'-x) × Cubic (J'-y)
... (12)
It is to be noted that, much like notation ∑ in Eq. (5), notation ∑ in Eq. (11) denotes a sum computed for a position (I', J') with respect to all sets of integers (k, ir and jr) in which the integer k is a sequence number assigned to a taken image associated with transformation parameters (ak, bk, ck, dk, sk and tk) used in an affine transformation process to transform a position (ir-1, jr-1) in the coordinate system of the kth taken image into the position (x, y) located in the reference coordinate system as a post-transformation position with the coordinates (x, y) thereof satisfying the relations I'-2 ≤= x < I'+2 and J'-2 ≤ y < J' +2 expressing relations with the coordinates (I' , J' ) • By the same token, much like notation £ in Eq. (6), notation ∑ in Eq. (12) denotes a sum computed for a position (I', J') with respect to all sets of integers (k, ib and jb) in which the integer k is a sequence number assigned to a taken image associated with transformation parameters (ak, bk, ck, dk, sk and tk) used in an affine transformation process to transform a position (ib-1, jb-1) in the coordinate system of the kth taken image into the position (x, y) located in the reference coordinate system as a post-transformation position with the coordinates (x, y) thereof satisfying the relations I'-2 ≤ x < I'+2 and J'-2 ≤ y < J'+2 expressing relations with the coordinates (I', J') .
It is to be noted that, much like Eq. (5), Eq. (11) is also referred to as a weighed addition equation for the light quantity of the red color. By the same token, much like Eq. (6), Eq. (12) is also referred to as a weighed addition equation for the light quantity of the blue color.The numerator and denominator of Eq. 11 expressing
the light quantity of the red color are rewritten as
expressions (13) and (14) respectively.
[Expression (13)]I Cubic (I'-x) × Cubic (J'-y) ×Robs(k, ir, jr) (13)[Expression (14)] Cubic (I'-x) x Cubic (J'-y) (14)
By the same token, the numerator and denominator of
Eq. 12 expressing the light quantity of the blue color
are rewritten as expressions (15) and (16) respectively.
[Expression (15)]ZCubic(I'-x) x Cubic(J'-y) x Bobs (k, ib, jb) (15)[Expression (16)]ZCubic(I'-x) x Cubic(J'-y) (16)
By using Eq. (8) described above as the weighed addition equation for the light quantity of the green color, Eq. (11) described above as the weighed addition equation for the light quantity of the red color and Eq.
(12) described above as the weighed addition equation for the light quantity of the blue color, the processing circuit 24 is capable of finding respectively the light quantity Lg (I', J') of the green color, the light quantity Lr (I', J') of the red color and the light quantity Lb (I', J') of the blue color as picture values
at a position (I', J') on the output image.
By the way, when the processing circuit 24 finds the light quantity Lg (I', J') of the .green color, the light quantity Lr (I', J') of the red color and the light quantity Lb (I', J') of the blue color as picture values at a position (I', J') on the output image by using Eqs. (8), (11) and (12) respectively, the reliability of the pixel values may be low in some cases.
That is to say, in accordance with Eq. (8) described above as the weighed addition equation for the light quantity of the green color, the light quantity Lg (I', J') of the green color at a position (I', J') on the output image is found by dividing the numerator expressed by expression (.9) by a denominator expressed by expression (10). As is obvious from expression (9), the numerator is a sum of products each obtained by multiplying a pixel value Gobs (k, ig, jg) at a position (ig-1, jg-1) transformed into a post-transformation position (x, y) by a weight Cubic (I'-x) * Cubic (J'-y). As is obvious from expression (10), on the other hand, the denominator is a sum of weights Cubic (I'-x) * Cubic
(J'-y) .Thus, if the value of expression (10) expressing the denominator of Eq. (8) is 0 (or close to 0), the
light quantity Lg (I', J') found by using Eq. (8) as a light quantity of the green color at a position (I', J') has a value exhibiting instable (or indeterminate) and unreliable characteristics. In other words, at a position (I', J') for which the value of expression (10) expressing the denominator of Eq. (8) is 0, a small noise (or error) contained in the pixel value Gobs (k, ig, jg) included in the numerator of Eq. (8) is divided by the denominator of 0 and amplified to a large value. As a result, the light quantity Lg (I', J') found by using Eq. (8) as a light quantity of the green color at a position (I', J') has an unreliable value including a large noise.The value of expression (10) expressing the denominator of Eq. (8) is 0, for example, because at least either of the Cubic function referred to as Cubic (I'-x) or Cubic (J'-y) is 0 throughout all the summation range of Eq. (10) . As is obvious from the characteristics shown in FIG. 13 as the characteristics of the Cubic function, the Cubic function referred to as Cubic (I'-x) or Cubic (J'-y) is 0 in the range I'-x = ±1 or J'-y = ±1, that is, in the range x = I'±l or y =.J'±1.Thus, because the denominator of Eq. (8) is 0 (or almost 0) due to the fact that the coordinates (I', J') of the observed position in the reference coordinate
system and the coordinates (x, y) of a post-transformation position also in the reference coordinate system satisfy the relations x = I'±l or y = J'±l as described above, the value of the light quantity Lg (I', J') of the green color is found to be a value exhibiting instable (or indeterminate) and unreliable characteristics in a process to compute the light quantity Lg (I', J') at the position (I', J') in accordance with Eq. (8) by summing up pixel values Gobs (k, ig, jg) for all sets of integers (k, ig and jg) in which the integer k is a sequence number assigned to a taken image associated with transformation parameters (ak, bk, gx, die, Sk and tk) used in an affine transformation process to transform a position (ig-1, jg-1) in the coordinate system of the kth taken image into the position (x, y) with the coordinates (x, y) thereof satisfying the relations I'-2 ≤ x < I'+2 and J'-2 ≤ y < J'+2 expressing relations with the coordinates (I', J')• However, only in a very special or exceptional case do the coordinates (x, y) of all the positions (x, y)
located in the range I'-2 ≤ x < I'+2 and J'-2 ≤ y < J'+2 as the positions of the pixel values Gobs (k, ig, jg) satisfy the relations x = I'±l or y = J'±l. In the following description, the very special or exceptional
case is referred to simply as an exception state.
FIG. 15 is an explanatory diagram showing a position (I', J') in an exception state.
In the exception state shown in FIG. 15, the coordinates x of post-transformation positions Gil and G15 each obtained as a result of an affine transformation process to transform a position (ig-1, jg-1) by using the transformation parameters (ak, bk, ck, dk, sk and tk) satisfy the relation x = I'-l. On the other hand, the coordinates x of post-transformation positions G12 and G16 each obtained as a result of an affine transformation process to transform a position (ig-1, jg-1) by using the transformation parameters ( (ak, bk, ck, dk, sk and tk) satisfy the relation x = I'+l.
By the same token, the coordinates y of post-transformation positions G13 and G14 each obtained as a result of an affine transformation process to transform a position (ig-1, jg-1) by using the transformation parameters (ak, bk, ck, dk, sk and tk) satisfy the relation y = J'-l. On the other hand, the coordinates y of post-transformation positions G17 , G18 and G19 each obtained as a result of an affine transformation process to transform a position (ig-1, jg-1) by using the transformation parameters (ak, bk, ck, dk, sk and tk)
satisfy the relation y = J'+l.
As described above, in the exception state shown in FIG. 15, in the processing to infer the light quantity Lg (I', J') for the observed position (I', J') of the observed pixel by using pixel values Gobs (k, ig, jg) for all sets of integers (k, ig and jg) in which the integer k is a sequence number assigned to a taken image associated with transformation parameters (ak, bk, ck, dk, sk and tk) used in an affine transformation process to transform a position (ig-1, jg-1) in the coordinate system of the kth taken image into the position (x, y) with the coordinates (x, y) thereof satisfying the relations I'-2 ≤ x < I'+2 and J'-2 ≤ y < J'+2 expressing relations with the coordinates (I', J'), the relation x = I'±l or y = J'±l between the coordinates of the observed position (I', J') and the coordinates of the post-transformation position (x, y) may hold true in some cases. In this exception state, for the G signal, there is not even one piece of G data (that is, not even one pixel value Gobs (k, ig, jg)) associated with a non-zero weight w ((x, y), (I', J')) in the range centered at the observed position (I', J') as the range of coordinates (x, y) approximately within the boundaries I'±l and J'±1. The range centered at the observed position (I', J') as therange of coordinates (x, y) approximately within the boundaries I'±l and J'+l is a square area having dimensions of 2x2.A process to find a light quantity Lg (I', J') of the green color at the position (I', J') in such an exception state in accordance with Eq. (8) will result in a light-quantity value exhibiting instable (or indeterminate) and unreliable characteristics as described above.If the pixel located at the position (I', J') is in such an exception state, the processing circuit 24 does not carry out a process to find a light quantity Lg (I', J') of the green color at the position (I', J') in accordance with Eq. (8). Instead, the processing circuit 24 carries out an exception process described below as a process to find a light quantity Lg (I', J') of the green color at the position (I', J'). In contrast with the exception process, the processes to find a light quantity Lg (I', J') of the green color, a light quantity Lr (I', J') of the red color and a light quantity Lb (I', J') of the blue color at the position (I', J' ) in accordance with Eqs. (8), (11) and (12) respectively are each referred to as a normal process.That is to say, let us assume for example that theposition (I', J') is an observed position and the light quantity Lg (I', J') of the green color of the pixel at the observed position (I', J') on the output image is inferred. In the following description, the pixel at the observed position (I', J') on the output image is referred to as an observed pixel. If the observed pixel at the observed position (I', J') is in an exception state, the processing circuit 24 infers the light quantity Lg (I', J') of the green color of the observed pixel at the observed position (I', J') in the exception state by using not only the pixel value Gobs (k, ig, jg) of a pixel located on each kth taken image as a pixel at a position with its coordinates (ik-1, jk-1) transformed into the coordinates (x, y) of a post-transformation position (x, y) in close proximity to the observed position (I', J') but also the pixel value Gobs (k, ig, jg) of a pixel located on each kth taken image as a pixel at a position with its coordinates (ik-1, jk-1) transformed into the coordinates (x, y) of a post-transformation position (x, y) in close proximity to the position of a pixel located on the output image as a pixel in the vicinity of the observed position (I', J') .
A pixel in the vicinity of the observed position (I', J') in the exception process to infer the light
quantity Lg (I', J') of the green color of the pixel at the observed position (I', J') on the output image is referred properly to as a vicinity pixel of the observed pixel. In an example shown in FIG. 16, pixels at positions (I'-l, J'), (I'+l, J'), (I', J'-l) and (I', J'+l) can each be taken as a vicinity pixel.
That is to say, the image-pickup device 4 employed in the digital camera 1 has the form of the Bayer two-dimensional array as explained earlier by referring to FIG. 3. In the Bayer two-dimensional array, pixels receiving green-color components of light are arranged every other pixel in the X and Y directions.
If the coordinates of none of pixels each having a pixel value Gobs (k, ig, jg) of the G signal on the N taken images obtained by the image-pickup device 4 having the Bayer two-dimensional array are transformed into the coordinates of a position in close proximity to the observed position (I', J') in an affine transformation process to transform the coordinates of a position on any of the N taken images into the coordinates of a position in the reference coordinate system, the pixel located at the observed position (I', J') on the reference image can be regarded as a pixel of a color other than the green color.FIG. 17 is a diagram showing a reference image obtained by the image-pickup device 4 having the Bayer two-dimensional array. The Bayer two-dimensional array shown in FIG. 17 is an array of W columns arranged in the horizontal direction (or the X direction) and H rows arranged in the vertical direction (or the Y direction). That is to say, the reference image consists of W><2 and a vertical dimension shorter than the vertical dimension H of the reference image by a length equivalent to 0.08xf-[x2. That is to say, the high-picture-
quality square area 422 is a rectangle (or an oblong) having its left upper vertex located at a point (0.08>-2 area (tliat is, an area having vertical and horizontal dimensions of 2*2.) centered at a position having coordinates (a* (i-1.)-i-(3, ax(j"l) i-y) on H'1-output image. The pixel at the center of the area is a pixel, the pixel value of which is to he inferred. Then,the nex t step S317.Let notation N<] denote the number of aforementioned specific pixels identj.fi.ed at the step S316 from the fourth, taken image 401-,, notation G^ (p) denote the G signal (or the pixel value) of a sped Eic pixel tviving a pixel number p where p is an integer in the range 1, 2, , M4/ notation (X.i (p) , Y4 (p) ) denote coordinates included in the coordinate system of the fourth taken .image '1014 as the coordinates of the specific pixel and notation (X^(p), Y,!i(o)) denote pos t--t ransionna tion position coordinates obr.?-•.: nee as a resu.i.f ef the affine transformation process
specific pixel in accordance with Eq. (23). That is to say, the coordinates (X4 (p) , Y4(p)) and (X44 (p) , Y44 (p) ) correspond to respectively the coordinates (X4, Y4) and (X44, Y44) used in Eq. (23) .Thus, for any pixel number p, notation G4 (p) denotes the pixel value (or the G signal) of a specific pixel located at the coordinates (X4 (p) , Y4 (p) ) in the coordinate system of the fourth taken image 4014 as a pixel having the pixel number p. In addition, the coordinates (X44 (p) , Y44 (p) ) produced in the af fine transformation process as the coordinates of the post-transformation position in the reference coordinate system satisfy the relations a* (i-1)+|3-2 = X44(p) ^ ax(i-l)+p+2 and ax(j-l)+y-2 ^ Y44 (p) ^ ax(j-l)+y+2 expressing relations with the coordinates (ax(i-l)+(3, ax(j-l)+y).
At the step S317, the processing circuit 24 identifies all G-signal pixels (or pixels of the green color) on the fifth taken image 401s as specific pixels. The position of each of the selected G-signal pixels on the fifth taken image 4015 has coordinates (X5, Y5) transformable by an affine transformation process according to Eq. (24) into post-transformation coordinates (X45, Y45) included in the reference coordinate system as coordinates (X45, Y4s) that must satisfy the
relations a*(i-l)+p-2 ^ X45 ^ ax(i-l)+p+2 and ax(j-l)+Y-2 ^ Y45 = ax(j-l)+Y+2, where (ax(i-l)+p, ax(j-i)+Y) are coordinates of the position of a pixel on the output image. That is to say, the post-transformation position (X45, Y45) must be in a 2x2 area (that is, an area having vertical and horizontal dimensions of 2x2) centered at a position having coordinates (ax(i-l)+p, ax(j-l)+Y) on the output image. The pixel at the center of the area is a pixel, the pixel value of which is to be inferred. Then, the flow of the image generation processing goes on to the next step S318 shown in FIG. 32.Let notation ns denote the number of aforementioned specific pixels identified at the step S317 from the fifth taken image 401s, notation G5 (p) denote the G signal (or the pixel value) of a specific pixel having a pixel number p where p is an integer in the range 1, 2, , N5, notation (X5(p), Y5 (p) ) denote coordinates included in the coordinate system of the fifth taken image 4015 as the coordinates of the specific pixel and notation (X45(p), ¥45 (p)) denote post-transformation position coordinates obtained as a result of the affine transformation process carried out on the coordinates (X5 (p), Y5(p)) of the specific pixel in accordance with Eq. (24). That is to say, the coordinates (X5(p), Y5 (p) ) and (X45(p), Y45(p))
correspond to respectively the coordinates (X5, Y5) and (X45, Y45) used in Eq. (24).Thus, for any pixel number p, notation Gs(p) denotes the pixel value (or the G signal) of a specific pixel located at the coordinates (X5 (p) , Y5(p) ) in the coordinate system of the fifth taken image 4015 as a pixel having the pixel number p. In addition, the coordinates (X45(p), Y45 (p) ) produced in the affine transformation process as the coordinates of the post-transformation position in the reference coordinate system satisfy the relations a*(i-l)+p-2 ^ X45(p) ^ a*(i-l)+p+2 and a*(j-D+y-2 ^ Y45(P) ^ a* (j-D+Y+2.At the step S318, the processing circuit 24 identifies all G-signal pixels (or pixels of the green color) on the sixth taken image 4016 as specific pixels. The position of each of the selected G-signal pixels on the sixth taken image 401e has coordinates (X6, Y6) transformable by an affine transformation process according to Eq. (25) into post-transformation coordinates (X4e, Y4e) included in the reference coordinate system as coordinates (X46, Y46) that must satisfy the relations a*(i-l)+|3-2 ^ X46 ^ ax(i-l)+3 + 2 and a* ( j-1)-f-y-2 ^ Y46 = ax(j-l)+v+2, where (ax(i-l)+(3, ax(j-l)+Y) are coordinates of the position of a pixel on the output
image. That is to say, the post-transformation position (X46, Y46) must be in a 2><2 area (that is, an area having vertical and horizontal dimensions of 2^2) centered at a position having coordinates (ax(i-l)+|3, ax(j-l)+y) on the output image. The pixel at the center of the area is a pixel, the pixel value of which is to be inferred. Then, the flow of the image generation processing goes on to the next step S319.
Let notation Ne denote the number of aforementioned specific pixels identified at the step S318 from the sixth taken image 401e, notation G6(p) denote the G signal (or the pixel value) of a specific pixel having a pixel number p where p is an integer in the range 1, 2, , Ne, notation (X6(p), Ye(p)) denote coordinates included in the coordinate system of the sixth taken image 401e as the coordinates of the specific pixel and notation (X4e(p), Y4g(p)) denote post-transformation position coordinates obtained as a result of the affine transformation process carried out on the coordinates (Xg(p), Ye(p)) of the specific pixel in accordance with Eq. (25). That is to say, the coordinates (X6(p), Y6(p)) and (X46(p), Y46 (p) ) correspond to respectively the coordinates (Xe, Y6) and (X46, Y46) used in Eq. (25) .Thus, for any pixel number p, notation Ge(p)denotes the pixel value (or the G signal) of a specific pixel located at the coordinates (Xe(p), Yg(p)) in the coordinate system of the sixth taken image 401g as a pixel having the pixel number p. In addition, the coordinates (X46(p), Y46(p)) produced in the affine transformation process as the coordinates of the post-transformation position in the reference coordinate system satisfy the relations ax(i~l)+p-2 ^ X46(p) = ax(i-l)+p+2 and ax(j-l)+Y-2^ Y46(p) ^ ax(j-l)+Y+2.
At the step S319, the processing circuit 24 identifies all G-signal pixels (or pixels of the green color) on the seventh taken image 4017 as specific pixels. The position of each of the selected G-signal pixels on the seventh taken image 4017 has coordinates (X7, Y7) transformable by an affine transformation process according to Eq. (26) into post-transformation coordinates (X47, Y47) included in the reference coordinate system as coordinates (X47, Y47) that must satisfy the relations ax(i-l)+p-2 ^ X47 ^ c*x(i-i)+p + 2 and ax(j-1)+Y-2 ^ Y47 ^ ax(j-l)+y+2, where (ax(i-l)+p, ax(j-l)+y) are coordinates of the position of a pixel on the output image. That is to say, the post-transformation position (X47, Y47) must be in a 2x2 area (that is, an area having vertical and horizontal dimensions of 2x2) centered at aposition having coordinates (ax(i-l)+(3, ax(j-l)+Y) on the output image. The pixel at the center of the area is a pixel, the pixel value of which is to be inferred. Then, the flow of the image generation processing goes on to the next step S320.
Let notation N7 denote the number of aforementioned specific pixels identified at the step S319 from the seventh taken image 4017, notation 67 (p) denote the G signal (or the pixel value) of a specific pixel having a pixel number p where p is an integer in the range 1, 2, , N7, notation (X7(p), Y7(p)) denote coordinates included in the coordinate system of the seventh taken image 4017 as the coordinates of the specific pixel and notation (X47 (p), Y47 (p)) denote post-transformation position coordinates obtained as a result of the affine transformation process carried out on the coordinates (X7 (p), Y7(p)) of the specific pixel in accordance with Eq. (26). That is to say, the coordinates (X7 (p) , Y7(p)) and (X47(p), Y47 (p) ) correspond to respectively the coordinates (X7, Y7) and (X47, Y47) used in Eq. (26) .
Thus, for any pixel number p, notation G7(p) 'denotes the pixel value (or the G signal) of a specific pixel located at the coordinates (X7 (p) , Y7(p)) in the coordinate system of the seventh taken image 4017 as apixel having the pixel number p. In addition, the coordinates (X47(p), ¥47 (p) ) produced in the affine transformation process as the coordinates of the post-transformation position in the reference coordinate system satisfy the relations ax(i-l)+(3-2 ^ X47(p) 5= a*(i-D+P + 2 and ax(j-l)+Y-2 ^ Y47 (p) ^ ax(j-l)+Y+2.
At the step S320, the processing circuit 24 identifies all G-signal pixels (or pixels of the green color) on the eighth taken image 4018 as specific pixels. The position of each of the selected G-signal pixels on the eighth taken image 401s has coordinates (X8, Y8) transformable by an affine transformation process according to Eq. (27) into post-transformation coordinates (X48, Y48) included in the reference coordinate system as coordinates (X48, Y48) that must satisfy the relations a*(i-l)+(3-2 ^ X48 ^ ax(i-l)+p + 2 and a*(j-l)+Y-2 ^ Y48 ^ ax (j-i)+Y+2, where (ax(i-l)+(3, ax(j-l)+Y) are coordinates of the position of a pixel on the output image. That is to say, the post-transformation position (X48, Y4s) must be in a 2x2 area (that is, an area having vertical and horizontal dimensions of 2x2) centered at a position having coordinates (ax (i-1)+(3, a*(j-l)+v) on the output image. The pixel at the center of the area is a pixel, the pixel value of which is to be inferred. Then,the flow of the image generation processing goes on to the next step S321 shown in FIG. 33.
Let notation N8 denote the number of aforementioned specific pixels identified at the step S320 from the eighth taken image 401s, notation Gg(p) denote the G signal (or the pixel value) of a specific pixel having a pixel number p where p is an integer in the range 1, 2, , N8, notation (X8(p), Y8(p) ) denote coordinates included in the coordinate system of the eighth taken image 4018 as the coordinates of the specific pixel and notation (X48 (p) , Y48(p)) denote post-transformation position coordinates obtained as a result of the affine transformation process carried out on the coordinates (X8 (p) , Y8(p)) of the specific pixel in accordance with Eq. (27) . That is to say, the coordinates (X8 (p) , Y8(p)) and '(X48(p), Y48 (p) ) correspond to respectively the coordinates (X8, Y8) and (X48, Y48) used in Eq. (27) .Thus, for any pixel number p, notation G8 (p) denotes the pixel value (or the G signal) of a specific pixel located at the coordinates (X8 (p) , Y8(p) ) in the coordinate system of the eighth taken image 4018 as a pixel having the pixel number p. In addition, the coordinates (X48(p), Y48(p)) produced in the affine transformation process as the coordinates of the post-229
transformation position in the reference coordinate system satisfy the relations ax(i-l)+(3-2 ^ X48 (p) = a* (i-D+P+2 and ax(j-l)+v-2 ^ Y48 (p) ^ ax(j-l)+Y+2.
At the step S321, the processing circuit 24 finds the numerator of Eq. (8) representing the weighted addition equation used for finding the green-color light quantity of a pixel (i, j) located on the output image on the basis of all the specific pixels identified at the steps S313 to S320. At the same step, the processing circuit 24 also finds the denominator of Eq. (8). As described before, the numerator of Eq. (8) is expressed by expression (9) and the denominator of Eq. (8) is expressed by expression (10).
To put it concretely, the value of expression (10) expressing the denominator of Eq. (8) expressing a weighted addition equation used to find the light quantity of the green color is found by computing the value of expression (28) and the value of expression (9) expressing the numerator of Eq. (8) is found by computing the value of expression (29). Expressions (28) and (29) are given as follows., v Ns (Cubic (Xu - X,s(p)) x Cubic (Y0 - Y-r, (P)M Zj P==lN8 , ^ P = l
.f v i Cubic (Xa - X,,«(p)) x Cubic (Yn - Y,18(p))] (28)
[Expression (2.9} }
Nl {Cubic (X0 - X4i(p)) x Cubic (Y0 - YlU (p) ) x Gj. fp)}
(Cubic (X0 " Xll7 (p) ) x Cubic (Y0 - Y.r, (p) } x G7(p)}
+ y Ns (Cubic (X0 - X48(p)) x Cubic (Yo - Y48(p)) * G8(p)} L .P=l
(29)
It is to be noted that notation (Xo, yo) in expressions (28) and (29) denotes the coordinates of the aforementioned pixel (i, j), the pixel value of which is to be found. The coordinates (Xo, Y0) are coordinates of the pixel (i, j) on the output image where X0 = a*(i-l)+p and Y0 = oc<(j-l)+Y-
Expression (29) is a weighted addition expression used for finding the total of eight sums corresponding to k = 1 to 8 respectively. Notation k is an integer assigned to each of the eight taken images ranging from the first taken image 401i to the eighth taken image 401s, which are each subjected to a process to transform the coordinates of positions of specific pixels on the taken images into coordinates of positions on the reference image. Each of the eight sums is an expression used for finding the sum of weighted pixel values for p = 1 to Nk. Each of the weighted pixel values is a product obtained by multiplying the pixel value Gk(p) by a weight Cubic (X0-X4k(p) ) xCubic (Y0-Y4k(p)). The pixel value Gk(p) is a G signal observed for an aforementioned specific pixel at a position with its coordinates transformable into
coordinates (X4k(p), Y4k(p)) in close proximity to the
coordinates (ax(i-l)+p, ax(j-l)+y) of the position of a pixel (i, j), the pixel value of which is being inferred. As described above, the pixel (i, j) is a pixel on the output image. On the other hand, expression (28) is an expression used for finding the total of eight other sums corresponding to k = 1 to 8 respectively. In this case, each of the eight other sums is an expression used for finding the sum of weights each expressed by Cubic (X0-X4k(p)) x Cubic (Y0-Y4k(p)) for p = 1 to Nk. A process to divide the value of expression (29) by the value of expression (28) is equivalent to the process to compute the expression on the right side of Eq. (8). The result of the process is a weighted average value of all the pixel values Gk(p) each multiplied by a weight according to the distance between the position (Xo, yo) of the pixel (i, j), the pixel value of which is being found, and a position with its into coordinates (X4k(p), Y4k(p)) obtained as a result of an affine transformation process carried out on the coordinates of a specific pixel as described above. The pixel (i, j) is a pixel on the output image. Each of the pixel values Gk(p) is the pixel value of a specific pixel identified at any of the steps 5313 to S320 as a pixel on any of the taken images.
After the processing circuit 24 computes the values
of expressions (28) and (29) for the pixel (i, i) and stores the values obtained as a resu.'.t of the computation in a memory shown in none of the figures, the flow of the image generation processing goes on to the next step S322
Processes of the steps S322 to S330 are to the R signal what the processes of the steps S313 to S321 respectively are to the G signal. By the same token, processes of the steps S331 to S339 are to the P> signal what the processes of the steps S313 to S321 respectively are to the G signal.
To put it in detail, at the step S322, the processing circuit 24 identifies all R-signal pixels (or pixels of the red color) on the first taken image 4011 ^s specific pixels. The position of each of the selected R-signal pixels on the first taken image OOli has coordinates (Xi, Yi) transformable by an affine transformation process according to L?,g. (20) into post-transformation coordinates (X(p)} of the specific pixel in accordance with Eq. (21). That is to say, the coordinates (X?(p), Y2(p)) and (X,j?(p), Y.i/fp)} correspond to respective]1/ the coordinates (X;>, Y?) and (X4;:, Y,!2) used in Kq. (21) .
Thus, for any pixel number p, notation R2(p) denotes the pixel value (or the R signal) o£ a specific pixel ]. oca ted at the coordinates (X?(p), Y2(p)) in the coordinate system of the second taken image 401;> as a pixel ha vino the pixel number p. In addition, the coordinates •;X,'!,i(p), Y.-;:-(p)) produced i.n the affine
transformation process as the coordinates of the post-transformation position in the reference coordinate system satisfy the relations ax(i-l)+(3-2 ^ X42(p) = ax (i-
l)+p + 2 and ax(j-l)+Y-2 Y42 (p) ax(j-i)+Y+2.
At the step S324, the processing circuit 24
identifies all R-signal pixels (or pixels of the red color) on the third taken image 4013 as specific pixels. The position of each of the selected R-signal pixels on the third taken image 4013 has coordinates (X3, Y3) transformable by an affine transformation process according to Eq. (22) into post-transformation coordinates (X43, Y43) included in the reference coordinate system as coordinates (X43, Y43) that must satisfy the relations ax(i-l)+p-2 = X43 ^ a*(i-l)+p+2 and ax(j-l)+Y-2 ^ Y43 ^ ax(j-l)+Y+2, where (ax(i-l)+p, ax(j-l)+Y) are coordinates of the position of a pixel on the output image. That is to say, the post-transformation position (X43, Y43) must be in a 2x2 area (that is, an area having vertical and horizontal dimensions of 2x2) centered at a position having coordinates (a*(i-l)+|3, ax(j-l)-fv) on the output image. The pixel at the center of the area is a pixel, the pixel value of which is to be inferred. Then, the flow of the image generation processing goes on to the next step S325.
Let notation N3 denote the number of aforementioned specific pixels identified at the step S324 from the third taken image 4013, notation R3(p) denote the R signal (or the pixel value) of a specific pixel having a pixel number p where p is an integer in the range 1, 2, , N3, notation (X3(p), Y3(p)) denote coordinates included in the coordinate system of the third taken image 4013 as the coordinates of the specific pixel and notation (X43(p), Y43(p) ) denote post-transformation position coordinates obtained as a result of the affine transformation process carried out on the coordinates (X3(p) , Y3(p)) of the specific pixel in accordance with Eq. (22). That is to say, the coordinates (X3(p), Y3(p)) and (X43(p), Y43(p)) correspond to respectively the coordinates (X3, Y3) and (X43, Y43) used in Eq. (22) .
Thus, for any pixel number p, notation R3(p) denotes the pixel value (or the R signal) of a specific pixel located at the coordinates (X3(p), Y3(p)) in the coordinate system of the third taken image 4013 as a pixel having the pixel number p. In addition, the coordinates (X43(p), Y43(p)) produced in the affine transformation process as the coordinates of the post-transformation position in the reference coordinate system satisfy the relations a*(i-l)+p-2 ^ X43(p) ^ ax(i-l)+p+2 and ax(j-
l)+y-2 = Y43(p) ^ ax(j-l)+y+2 expressing relations with the coordinates (ax(i-l)+p, ax(j-l)+y).
At the step S325, the processing circuit 24 identifies all R-signal pixels (or pixels of the red color) on the fourth taken image 4014 as specific pixels. The position of each of the selected R-signal pixels on the fourth taken image 4014 has coordinates (X4, Y4) transformable by an affine transformation process according to Eq. (23) into post-transformation coordinates (X44, Y44) included in the reference coordinate system as coordinates (X44/ Y44) that must satisfy the relations ax(i-l)+p-2 ^ X44 ^ ax(i-l)+p + 2 and ax(j-l)+y-2 ^ Y44 ^ ax(j-l)+Y+2, where (ax(i-1)+p, ax(j-i)+y) are coordinates of the position of a pixel on the output image. That is to say, the post-transformation position (X44, Y44) must be in a 2x2 area (that is, an area having vertical and horizontal dimensions of 2x2) centered at a position having coordinates (ax(i-l)+(3, a*(j-l)+y) on the output image. The pixel at the center of the area is a pixel, the pixel value of which is to be inferred. Then, the flow of the image generation processing goes on to the next step S326.
Let notation N4 denote the number of aforementioned specific pixels identified at the step S325 from the
fourth taken image 4014, notation R4 (p) denote the R signal (or the pixel value) of a specific pixel having a pixel number p where p is an integer in the range 1, 2, , N4, notation (X4(p), Y4(p) ) denote coordinates included in the coordinate system of the fourth taken image 4014 as the coordinates of the specific pixel and notation (X44 (p) , Y44 (p)) denote post-transformation position coordinates obtained as a result of the affine transformation process carried out on the coordinates (X4 (p), Y4 (p) ) of the specific pixel in accordance with Eq. (23). That is to say, the coordinates (X4 (p) , Y4 (p) ) and (X44 (p) , Y44 (p) ) correspond to respectively the coordinates (X4, Y4) and (X44, Y44) used in Eq. (23) .
Thus, for any pixel number p, notation R4(p) denotes the pixel value (or the R signal) of a specific pixel located at the coordinates (X4 (p), Y4 (p)) in the coordinate system of the fourth taken image 4014 as a pixel having the pixel number p. In addition, the coordinates (X44(p), Y44(p)) produced in the affine transformation process as the coordinates of the post-transformation position in the reference coordinate system satisfy the relations a*(i-l)+p-2 X44 (p) ax (i-l)+(3 + 2 and ax(j-l)+y-2 Y44(p) ax(j-l)+Y+2 expressing relations with the coordinates (ax(i-l)+(3, ax(j-l)+y).
At the step S326, the processing circuit 24 identifies all R-signal pixels (or pixels of the red color) on the fifth taken image 4015 as specific pixels. The position of each of the selected R-signal pixels on the fifth taken image 4015 has coordinates (X5, Y5) transformable by an affine transformation process according to Eq. (24) into post-transformation coordinates (X45, Y45) included in the reference coordinate system as coordinates (X45, Y45) that must satisfy the relations α(i-l)+β-2 ≤ X45 ≤ a≤(i-l)+|3 + 2 and ax(j-l)+y-2 ^ Y45 ^ ax(j-l)+Y+2, where (a*(i-l)+|3, ax(j-l)+Y) are coordinates of the position of a pixel on the output image. That is to say, the post-transformation position (X45, Y45) must be in a 2x2 area (that is, an area having vertical and horizontal dimensions of 2x2) centered at a position having coordinates (ax (i-1)+(3, ax(j-l)+y) on the output image. The pixel at the center of the area is a pixel, the pixel value of which is to be inferred. Then, the flow of the image generation processing goes on to the next step S327 shown in FIG. 35.
Let notation N5 denote the number of aforementioned specific pixels identified at the step S326 from the fifth taken image 4015, notation Rs(p) denote the R signal (or the pixel value) of a specific pixel having a pixel
number p where p is an integer in the range 1, 2, , N5, notation (X5 (p), Y5 (p) ) denote coordinates included in the coordinate system of the fifth taken image 4015 as the coordinates of the specific pixel and notation (X45(p), ¥45(p)) denote post-transformation position coordinates obtained as a result of the affine transformation process carried out on the coordinates (X5(p), Y5 (p) ) of the specific pixel in accordance with Eq. (24). That is to say, the coordinates (X5 (p) , Y5(p)) and (X45(p), Y45 (p) ) correspond to respectively the coordinates (Xs, Y5) and (X45, Y45) used in Eq. (24) .
Thus, for any pixel number p, notation R5 (p) denotes the pixel value (or the R signal) of a specific pixel located at the coordinates (X5 (p), Y5 (p)) in the coordinate system of the fifth taken image 4015 as a pixel having the pixel number p. In addition, the coordinates (X45(p), Y45(p)) produced in the affine transformation process as the coordinates of the post-transformation position in the reference coordinate system satisfy the relations a(i-l)+p-2 ≤ X45 (p) ≤ ax(i-l)+p+2 and a(j-D+Y-2 ^ Y45(p) ^ ax (j-l)+Y+2.
At the step S327, the processing circuit 24 identifies all R-signal pixels (or pixels of the red color) on the sixth taken image 4016 as specific pixels.
The position of each of the selected R-signal pixels on the sixth taken image 4016 has coordinates (X6, Y6) transformable by an affine transformation process according to Eq. (25) into post-transformation coordinates (X46, Y46) included in the reference coordinate system as coordinates (X46, Y46) that must satisfy the relations ax(i-l)+p-2 ≤ X46 ≤ ax(i-l)+(3+2 and ax(j-i)+Y-2 ^ Y46 ^ ax(j-l)+Y+2, where (ax(i-l)+p, ax(j-i)+Y) are coordinates of the position of a pixel on the output image. That is to say, the post-transformation position (X46, Y46) must be in a 2x2 area (that is, an area having vertical and horizontal dimensions of 2x2) centered at a position having coordinates (a*(i-l)+p, ax(j-i)+Y) on the output image. The pixel at the center of the area is a pixel, the pixel value of which is to be inferred. Then, the flow of the image generation processing goes on to the next step S328.
Let notation N6 denote the number of aforementioned specific pixels identified at the step S327 from the sixth taken image 401e, notation Re(p) denote the R signal (or the pixel value) of a specific pixel having a pixel number p where p is an integer in the range 1, 2, , N6, notation (X6(p), Y6(p)) denote coordinates included in the coordinate system of the sixth taken image 401e as the
coordinates of the specific pixel and notation (X46(p), Y46(p)) denote post-transformation position coordinates obtained as a result of the affine transformation process carried out on the coordinates (X6(p), Y6(p)) of the specific pixel in accordance with Eq. (25). That is to say, the coordinates (X6(p), Y6(p)) and (X46(p), Y46(p)) correspond to respectively the coordinates (X6, Y6) and (X46, Y46) used in Eq. (25) .
Thus, for any pixel number p, notation R6(p) denotes the pixel value (or the R signal) of a specific pixel located at the coordinates (Xe(p), Y6(p)) in the coordinate system of the sixth taken image 401e as a pixel having the pixel number p. In addition, the coordinates (X4g(p), Y4e(p)) produced in the affine transformation process as the coordinates of the post-transformation position in the reference coordinate system satisfy the relations ax(i-l)+(5-2 ^ X46(p) ^ax(i-l)+p + 2 and ax(j-D+Y-2 ^ Y46(p) ^ a* (j-D+v+2.
At the step S328, the processing circuit 24 identifies all R-signal pixels (or pixels of the red color) on the seventh taken image 4017 as specific pixels. The position of each of the selected R-signal pixels on the seventh taken image 4017 has coordinates (X7, Y7) transformable by an affine transformation process
according to Eq. (26) into post-transformation coordinates (X47, Y47) included in the reference coordinate system as coordinates (X47, Y47) that must satisfy the relations a*(i-l)+|3-2 ^ X47 ^ ax(i-l)+p+2 and ax(j-l)+Y-2 ^ Y47 ^ ax (j-1)+Y+2, where (ax (i-1)+(3, ax(j-l)+Y) are coordinates of the position of a pixel on the output image. That is to say, the post-transformation position (X47, Y47) must be in a 2*2 area (that is, an area having vertical and horizontal dimensions of 2x2) centered at a position having coordinates («x (j_-i)+p, ax(j-l)+Y) on the output image. The pixel at the center of the area is a pixel, the pixel value of which is to be inferred. Then, the flow of the image generation processing goes on to the next step S329.
Let notation N7 denote the number of aforementioned specific pixels identified at the step S328 from the seventh taken image 4017, notation R7(p) denote the R signal (or the pixel value) of a specific pixel having a pixel number p where p is an integer in the range 1, 2, , N7, notation (X7 (p) , Y7(p)) denote coordinates included in the coordinate system of the seventh taken image 4017 as the coordinates of the specific pixel and notation (X47(p), Y47(p)) denote post-transformation position coordinates obtained as a result of the affine transformation process
carried out on the coordinates (X7(p), Y7 (p)) of the specific pixel in accordance with Eq. (26). That is to say, the coordinates (X7 (p) , Y7 (p) ) and (X47 (p) , Y47 (p) ) correspond to respectively the coordinates (X7, Y7) and (X47, Y47) used in Eq. (26) .
Thus, for any pixel number p, notation R7 (p) denotes the pixel value (or the R signal) of a specific pixel located at the coordinates (X7 (p) , Y7 (p) ) in the coordinate system of the seventh taken image 4017 as a pixel having the pixel number p. In addition, the coordinates (X47(p), Y47(p)) produced in the affine transformation process as the coordinates of the post-transformation position in the reference coordinate system satisfy the relations a*(i-l)+p-2 ^ X47(p) ^ a* (i-D+P+2 and ax(j-l)+y-2 ^ Y47(p) ^ a* (j-1)+Y+2.
At the step S329, the processing circuit 24 identifies all R-signal pixels (or pixels of the red color) on the eighth taken image 401s as specific pixels. The position of each of the selected R-signal pixels on the eighth taken image 4018 has coordinates (X8, Y8) transformable by an affine transformation process according to Eq. (27) into post-transformation coordinates (X4s, Y48) included in the reference coordinate system as coordinates (X4s, Y48) that must satisfy the
relations ax(i-l)+p-2 X48 ax(i-l)+p + 2 and ax(j-l)+Y-2 Y48 ax(j-l)+y+2, where (ax(i-l)+p, ax(j-i)+Y) are coordinates of the position of a pixel on the output image. That is to say, the post-transformation position (X48, Y48) must be in a 2x2 area (that is, an area having vertical and horizontal dimensions of 2x2) centered at a position having coordinates (a*(i-l)+fir ax(j-i)+Y) on the output image. The pixel at the center of the area is a pixel, the pixel value of which is to be inferred. Then, the flow of the image generation processing goes on to the next step S330 shown in FIG. 36.
Let notation N8 denote the number of aforementioned specific pixels identified at the step S329 from the eighth taken image 4018, notation R8(p) denote the R signal (or the pixel value) of a specific pixel having a pixel number p where p is an integer in the range 1, 2, , N8, notation (X8 (p), Y8 (p) ) denote coordinates included in the coordinate system of the eighth taken image 4018 as the coordinates of the specific pixel and notation (X4e(p), Y48(p)) denote post-transformation position coordinates obtained as a result of the affine transformation process carried out on the coordinates (X8(p), Y8(p)) of the specific pixel in accordance with Eg. (27). That is to say, the coordinates (X8(p), Y8 (p) ) and (X48(p), Y48 (p) )
correspond to respectively the coordinates (X8, Y8) and (X48, Y48) used in Eq. (27) .
Thus, for any pixel number p, notation R8(p) denotes the pixel value (or the R signal) of a specific pixel located at the coordinates (X8(p), Y8 (p)) in the coordinate system of the eighth taken image 4018 as a pixel having the pixel number p. In addition, the coordinates (X48(p), Y48(p)) produced in the affine transformation process as the coordinates of the post-transformation position in the reference coordinate system satisfy the relations ax(i-l)+(3-2 ^ X48(p) ^ a* (i-D+J3 + 2 and ax(j-l)+y-2 ^ Y48(p) ^ ax(j-l)+Y+2.
At the step S330, the processing circuit 24 finds the numerator of Eq. (11) representing the weighted addition equation used for finding the red-color light quantity of a pixel (i, j) located on the output image on the basis of all specific pixels identified at the steps S322 to S329. At the same step, the processing circuit 24 also finds the denominator of Eq. (11). As described before, the numerator of Eq. (11) is expressed by expression (13) and the denominator of Eq. (11) is expressed by expression (14).
To put it concretely, the value of expression (14) expressing the denominator of Eq. (11) expressing a
weighted addition equation used to find the light quantity of the red color is found by computing the value of expression (30) and the value of expression (13) expressing the numerator of Eq. (11) is found by computing the value of expression (31). Expressions (30) and (31) are given as follows.
[Expression (30)]
y Nl {Cubic (X0 - X4i(p)) x Cubic (Y0 - Y41 (p) ) } L P=l
Na {Cubic (X0 - X42(p)) x Cubic (Y0 - Y42 (p) ) } P=l
N3 {Cubic (X0 - X43(p)) x Cubic (Y0 - Y43 (p) ) }
e> -1
N4 P=l
N5 P=l N6 P=l
{Cubic (X0 - X44(p)) x Cubic (Y0 - Y44 (p) ) }
Ns {Cubic (X0 - X45(p)) x Cubic (Y0 - Y45(p))} P=l
Ne {Cubic (X0 - X46(p)) x Cubic (Y0 - Y46(p))} P=l
N7 {Cubic (X0 - X47(p)) x Cubic (Y0 - Y47 (p) ) } *
nb {Cubic (X0 - X48(p)) x Cubic (Yo - Y48(p))} (30) P=l
[Expression (31) ]
{Cubic (X0 - X4i(p)) x cubic (Y0 - Y41(p)) P=l
Nl P=
N2 {Cubic (X0 - X42(p)) >< Cubic (Y0 - Y42 (p) ) x R2(p)}
^ P=l
t
+ y (Cubic (X0 - X43 (p) ) x Cubic (Y0 - Y43 (p) ) * R3(p)} ^ P=l
+ v 4 {Cubic (X0 - X44(p)) x Cubic (Y0 - Y44 (p) ) x R4 (p) } L P=l
+ y Ns (Cubic (X0 - X45(p)) x cubic (Y0 - Y45(p)) x R5 (p) } A P=l
+ y 6 (Cubic (X0 - X46(p)) x Cubic (Y0 - Y46(p)) x R6(p)} L P=l
+ y ? (Cubic (X0 - X47 (p) ) x Cubic (Y0 - Y47(p)) x R7 (p) } ^ P=l
+ y N8 (Cubic (X0 - X48(p)) x Cubic (Y0 - Y48 (p) ) x R8(p)} ^ P=l
(31)
It is to be noted that notation (X0/ Y0) in expressions (30) and (31) denotes the coordinates of the aforementioned pixel (i, j), the pixel value of which is to be found. The coordinates (X0, Y0) are coordinates of the pixel (i, j) on the output image where Xo = a*(i-l)+p and Y0 = ax(j-l)+y.
Expression (31) is a weighted addition expression used for finding the total of eight sums corresponding to k = 1 to 8 respectively. Notation k is an integer assigned to each of the eight taken images ranging from the first taken image 401i to the eighth taken image 401s, which are each subjected to a process to transform the coordinates of positions of specific pixels on the taken images into coordinates of positions on the reference
image. Each of the eight sums is an expression used for
finding the sum of weighted pixel values for p = 1 to Nk. Each of the weighted pixel values is a product obtained by multiplying the pixel value Rk(p) by a weight Cubic (X0-X4k(p)) x Cubic (Y0-Y4k(p)). The pixel value Rk(p) is an R signal observed for an aforementioned specific pixel at a position with its coordinates transformable into coordinates (X4k(p), Y,jk(p)) in close proximity to the coordinates (a*(i-l)+p, a*(j-l)+Y) of the position of a pixel (i, j), the pixel value of which is being inferred. As described above, the pixel (i, j) is a pixel on the output image. On the other hand, expression (30) is an expression used for finding the total of eight other sums corresponding to k = 1 to 8 respectively. In this case, each of the eight other sums is an expression used for finding the sum of weights each expressed by Cubic (X0-X4k(p)) x Cubic (Y0-Y4k(p)) for p = 1 to Nk. A process to divide the value of expression (31) by the value of expression (30) is equivalent to the process to compute the expression on the right side of Eq. (11) . The result of the process is a weighted average value of all the pixel values Rk(p) each multiplied by a weight according to the distance between the position (X0, Y0) for the pixel (i, j), the pixel value of which is being found, and a position with its into coordinates (X4k(p), Y4k(p))
obtained as a result of an affine transformation process carried out on the coordinates of a specific pixel as described above. The pixel (i, j) is a pixel on the output image. Each of the pixel values Rj<(p) is the pixel value of a specific pixel identified at any of the steps S322 to S329 as a pixel on any of the taken images.
After the processing circuit 24 computes the values of expressions (30) and (31) for the pixel (i, j) and stores the values obtained as a result of the computation in the memory shown in none of the figures, the flow of the image generation processing goes on to the next step S331.
At the step S331, the processing circuit 24 identifies all B-signal pixels (or pixels of the blue color) on the first taken image 401i as specific pixels. The position of each of the selected B-signal pixels on the first taken image 401i has coordinates (Xi, yi) transformable by an affine transformation process according to Eq. (20) into post-transformation coordinates (X4i, Y4i) included in the reference coordinate system as coordinates (Xu, Y4i) that must satisfy the relations a*(i-l)+f3-2 ^ X4i ^ ax(i-l)+|3 + 2 and ax(j-l)+y-2 ^ Y41 ^ a* (j-1)+Y+2, where (ax(i-l)+p, ax(j-l)+y) are coordinates of the position of a pixel on the output
image. That is to say, the post-transformation position (X41, Y41) must be in a 2x2 area (that is, an area having vertical and horizontal dimensions of 2x2) centered at a position having coordinates (αx(i-l)+β, a(j-l)+y) on the output image. The pixel at the center of the area is a pixel, the pixel value of which is to be inferred. Then, the flow of the image generation processing goes on to the next step S332.
Let notation n1 denote the number of aforementioned specific pixels identified at the step S331 from the first taken image 401.1, notation b1 (p) denote the B signal (or the pixel value) of a specific pixel having a pixel number p where p is an integer in the range 1, 2, , ni, notation (X1(p), Y1(p)) denote coordinates included in the coordinate system of the first taken image 4011 as the coordinates of the specific pixel and notation (X41 (p) , Y4i (p)) denote post-transformation position coordinates obtained as a result of the affine transformation process carried out on the coordinates (X1(p), y1 (p)) of the specific pixel in accordance with Eq. (20) . That is to say, the coordinates (Xi(p), Yi(p)) and (X41 (p) , Y41 (p) ) correspond to respectively the coordinates (X1, y1) and (X41, Y41) used in Eq. (20) .
Thus, for any pixel number p, notation bi (p)
denotes the pixel value (or the B signal) of a specific pixel located at the coordinates (Xi (p) , Yi(p)) in the coordinate system of the first taken image 401i as a pixel having the pixel number p. In addition, the coordinates (X4i(p), Y41(p)) produced in the affine transformation' process as the coordinates of the post-transformation position in the reference coordinate system satisfy the relations α(i-l)+(3-2 X41 (p) ax(i-l)+(3 + 2 and a*(j-D+Y-2 ^ Y4i(p) ^ ax(j-l)+Y+2.
At the step S332, the processing circuit 24 identifies all B-signal pixels (or pixels of the blue color) on the second taken image 4012 as specific pixels. The position of each of the selected B-signal pixels on the second taken image 4012 has coordinates (X2, Y2) transformable by an affine transformation process according to Eq. (21) into post-transformation coordinates (X42, Y42) included in the reference coordinate system as coordinates (X42, Y42) that must satisfy the relations α(i-l)+β-2 ≤ X42 ≤ax(i-l)+p+2 and ax(j-l)+Y-2 ≤y42 ≤ α(j-l)+Y+2, where(αx(i-l)+p, ax(j-i)+y) are coordinates of the position of a pixel on the output image. That is to say, the post-transformation position (X42, Y42) must be in a 2x2 area (that is, an area having vertical and horizontal dimensions of 2x2) centered at a
position having coordinates (ax(i-l)+(3, ax(j-l)+y) on the output image. The pixel at the center of the area is a pixel, the pixel value of which is to be inferred. Then, the flow of the image generation processing goes on to the next step S333 shown in FIG. 37.
Let notation N2 denote the number of aforementioned specific pixels identified at the step S332 from the second taken image 4012, notation B2(p) denote the B signal (or the pixel value) of a specific pixel having a pixel number p where p is an integer in the range 1, 2, , N2, notation (X2 (p), Y2(p)) denote coordinates included in the coordinate system of the second taken image 4012 as the coordinates of the specific pixel and notation (X42(p), ¥42 (p)) denote post-transformation position coordinates obtained as a result of the affine transformation process carried out on the coordinates (X2(p), Y2(p)) of the specific pixel in accordance with Eq. (21). That is to say, the coordinates (X2(p), Y2(p)) and (X42 (p) , Y42(p)) correspond to respectively the coordinates (X2, Y2) and (X42, Y42) used in Eg. (21) .
Thus, for any pixel number p, notation B2 (p) denotes the pixel value (or the B signal) of a specific pixel located at the coordinates (X2(p), Y2(p)) in the coordinate system of the second taken image 4012 as a
pixel having the pixel number p. In addition, the coordinates (X42 (p) , Y42(p)) produced in the affine transformation process as the coordinates of the post-transformation position in the reference coordinate system satisfy the relations a (i-1)+[3-2 X42(p) = a (i-l)+(3 + 2 and a* (j-1)+Y-2 - Y42(p) = ax (j-1)+Y+2 .
At the step S333, the processing circuit 24 identifies all B-signal pixels (or pixels of the blue color) on the third taken image 401s as specific pixels. The position of each of the selected B-signal pixels on the third taken image 4013 has coordinates (X3, Y3) transformable by an affine transformation process according to Eq. (22) into post-transformation coordinates (X43, Y43) included in the reference coordinate system as coordinates (X43, Y43) that must satisfy the relations a(i-l)+p-2 X43 ax (i-1)+(3 + 2 and ax(j-l)+Y-2 Y43 x(i-l)+Y+2, where (ax(i-l)+β, αx(j-l)+Y) are coordinates of the position of a pixel on the output image. That is to say, the post-transformation position (X43, Y43) must be in a 2x2 area (that is, an area having vertical and horizontal dimensions of 2x2) centered at a position having coordinates (ax(i-l)+(3, a*(j-l)+Y) on the output image. The pixel at the center of the area is a pixel, the pixel value of which is to be inferred. Then,the flow of the image generation processing goes on to the next step S334.
Let notation N3 denote the number of aforementioned specific pixels identified at the step S333 from the third taken image 4013, notation B3(p) denote the B signal (or the pixel value) of a specific pixel having a pixel number p where p is an integer in the range 1, 2, , N3, notation (X3(p), Y3(p)) denote coordinates included in the coordinate system of the third taken image 4013 as the coordinates of the specific pixel and notation (X43(p), Y^3 (p) ) denote post-transformation position coordinates obtained as a result of the affine transformation process carried out on the coordinates (X3 (p), Y3 (p) ) of the specific pixel in accordance with Eq. (22). That is to say, the coordinates (X3(p), Y3 (p) ) and (X43(p), Y43(p)) correspond to respectively the coordinates (X3, Y3) and (X43, Y43) used in Eq. (22) .Thus, for any pixel number p, notation B3 (p) denotes the pixel value (or the B signal) of a specific pixel located at the coordinates (X3(p), Y3(p)) in the coordinate system of the third taken image 4013 as a pixel having the pixel number p. In addition, the coordinates (X43(p), Y43 (p) ) produced in the affine transformation process as the coordinates of the post-transformationposition in the reference coordinate system satisfy the relations ax(i-l)+p-2 X43(p) ax(i-i)+p+2 and a*(j-D+Y-2 Y43(p) ax(j-l)+Y+2.At the step S334, the processing circuit 24 identifies all B-signal pixels (or pixels of the blue color) on the fourth taken image 4014 as specific pixels. The position of each of the selected B-signal pixels on the fourth taken image 4014 has coordinates (X4, Y4) transformable by an affine transformation process according to Eq. (23) into post-transformation coordinates (X44, Y44) included in the reference coordinate system as coordinates (X44, Y44) that must satisfy the relations ax(i-l)+p-2 ^ X44 ^ ax (i-1) +(3+2 ' and ax(j-l)+y-2 ^ Y44 ^ a*(i-l)+Y+2, where (ax(i-l)+)3, ax(j-l)-fy) are coordinates of the position of a pixel on the output image. That is to say, the post-transformation position (X44, Y44) must be in a 2x2 area (that is, an area having vertical and horizontal dimensions of 2x2) centered at a position having coordinates (ax(i-l)+p, ax(j-l)-fy) on the output image. The pixel at the center of the area is a pixel, the pixel value of which is to be inferred. Then, the flow of the image generation processing goes on to the next step S335.
Let notation N4 denote the number of aforementioned
specific pixels identified at the step S334 from the fourth taken image 4014, notation B4 (p) denote the B signal (or the pixel value) of a specific pixel having a pixel number p where p is an integer in the range 1, 2, , N4, notation (X4(p), Y4 (p)) denote coordinates included in the coordinate system of the fourth taken image 4014 as the coordinates of the specific pixel and notation (X44(p), Y44 (p) ) denote post-transformation position coordinates obtained as a result of the affine transformation process carried out on the coordinates (X4(p) , Y4 (p)) of the specific pixel in accordance with Eq. (23). That is to say, the coordinates (X4 (p) , Y4 (p) ) and (X44 (p) , Y44 (p) ) correspond to respectively the coordinates (X4, Y4) and (X44, Y44) used in Eq. ' (23) .
Thus, for any pixel number p, notation B4(p) denotes the pixel value (or the B signal) of a specific pixel located at the coordinates (X4 (p), Y4 (p)) in the coordinate system of the fourth taken image 4014 as a pixel having the pixel number p. In addition, the coordinates (X44 (p) , Y44 (p) ) produced in the affine transformation process as the coordinates of the post-transformation position in the reference coordinate system satisfy the relations a*(i-l)+p-2 ^ X44 (p) ^ a*(i-l)+(3 + 2 and ax(j-l)+y-2 ^ Y44 (p) ^ ax (j-l)+Y+2 .
At the step S335, the processing circuit 24 identifies all B-signal pixels (or pixels of the blue color) on the fifth taken image 401s as specific pixels. The position of each of the selected B-signal pixels on the fifth taken image 4015 has coordinates (X5, Y5) transformable by .an affine transformation process according to Eq. (24) into post-transformation coordinates (X45, ¥45) included in the reference coordinate system as coordinates (X45, ¥45) that must satisfy the relations a*(i-l)+|3-2 ^ X45 ^ ax(i-l)+p+2 and ax(j-l)+y-2 ^ Y45 ^ ax(j-l)+y+2, where (ax(i-l)+(3, ax(j-i)+y) are coordinates of the position of a pixel on the output image. That is to say, the post-transformation position (X45, ¥45) must be in a 2x2 area (that is, an area having vertical and horizontal dimensions of 2x2) centered at a position having coordinates (a*(i-l)+|3, a* (j-i)+y) on the output image. The pixel at the center of the area is a pixel, the pixel value of which is to be inferred. Then, the flow of the image generation processing goes on to the next step S336 shown in FIG. 38.
Let notation N5 denote the number of aforementioned specific pixels identified at the step S335 from the fifth taken image 401s, notation B5 (p) denote the B signal (or the pixel value) of a specific pixel having a pixel
number p where p is an integer in the range 1, 2, , notation (X5 (p), Y5 (p) ) denote coordinates included in the coordinate system of the fifth taken image 4015 as the coordinates of the specific pixel and notation (X4s (p) , Y4s (p) ) denote post-transformation position coordinates obtained as a result of the affine transformation process carried out on the coordinates (X5(p) , Y5 (p) ) of the specific pixel in accordance with Eq. (24). That is to say, the coordinates (X5 (p) , Y5(p)) and (X45 (p) , Y45 (p) ) correspond to respectively the coordinates (X5, Y5) and (X45, Y45) used in Eq. (24) .
Thus, for any pixel number p, notation 65(p) denotes the pixel value (or the B signal) of a specific pixel located at the coordinates (Xs(p), Ys(p)) in the coordinate system of the fifth taken image 4015 as a pixel having the pixel number p. In addition, the coordinates (X4s (p) , Y45 (p) ) produced in the affine transformation process as the coordinates of the post-transformation position in the reference coordinate system satisfy the relations ax(i-l)+p-2 X45 (p) a*(i-l)+p+2 and a (j-D+Y-2 Y45(p) ax (j-l)+Y+2.At the step S336, the processing circuit 24 identifies all B-signal pixels (or pixels of the blue color) on the sixth taken image 401g as specific pixels.The position of each of the selected B-signal pixels on the sixth taken image 4016 has coordinates (Xe, Y6) transformable by an affine transformation process according to Eq. (25) into post-transformation coordinates (X4g, Y46) included in the reference coordinate system as coordinates (X46, Y4g) that must satisfy the relations a*(i-l)+p-2 ^ X46 ^ ax(i-l)+(3 + 2 and ax(j-l)+Y-2 ^ Y46 ^ ax(j-i)+y+2, where (ax(i-l)+(3, ax(j-l)+Y) are coordinates of the position of a pixel on the output image. That is to say, the post-transformation position (X4e, Y46) must be in a 2x2 area (that is, an area having vertical and horizontal dimensions of 2x2) centered at a position having coordinates (ax (i-1)+)3, a*(j-l)+Y) on the output image. The pixel at the center of the area is a pixel, the pixel value of which is to be inferred. Then, the flow of the image generation processing goes on to the next step S337.
Let notation N6 denote the number of aforementioned specific pixels identified at the step S336 from the sixth taken image 4016, notation B6(p) denote the B signal (or the pixel 'value) of a specific pixel having a pixel number p where p is an integer in the range 1, 2, , N6, notation (X6(p), Y6(p)) denote coordinates included in the coordinate system of the sixth taken image 401e as the
coordinates of the specific pixel and notation (X4e(p), Y4e(p)) denote post-transformation position coordinates obtained as a result of the affine transformation process carried out on the coordinates (X6(p), Y6(p)) of the specific pixel in accordance with Eq. (25). That is to say, the coordinates (X6(p), Y6(p)) and (X46(p), Y46(p)) correspond to respectively the coordinates (X6, Y6) and (X46, Y46) used in Eq. (25) .
Thus, for any pixel number p, notation Be(p) denotes the pixel value (or the B signal) of a specific pixel located at the coordinates (X6(p), Y6(p)) in the coordinate system of the sixth taken image 4016 as a pixel having the pixel number p. In addition, the coordinates (X4e (p) , Y46(p)) produced in the affine transformation process as the coordinates of the post-transformation position in the reference coordinate system satisfy the relations a* (i-1)+|3-2 ^ X46(p) ^ ax(i-l)+p+2 and a*(j-D+Y-2 ^ Y46(p) ^ ax(j-l)+Y+2.
At the step S337, the processing circuit 24 identifies all B-signal pixels (or pixels of the blue color) on the seventh taken image 401v as specific pixels. The position of each of the selected B-signal pixels on the seventh taken image 401? has coordinates (X7, Y7) transformable by an affine transformation process
according to Eq. (26) into post-transformation coordinates (X<2 and a vertical dimension shorter than the vertical dimension H/2 of the reference image by a length equivalent to 0.08><4) . The reciprocal l/Ta of the actual exposure time represents a shutter speed. It is to be noted that the method for determining the diaphragm value F, the proper exposure time Tp and the actual exposure time Ta as described above is the same technique as the method adopted by the ordinary digital camera as a method with the diaphragm value F taking precedence. Thus, the method for determining the diaphragm value F, the proper exposure time Tp and the actual exposure time Ta as described above is a method provided by a known, technology. For this reason, detailed description of the method is omitted. After the process of the step S502 is completed, the flow of the processing goes on to the next step S503.
At the step S503, the control circuit 18 carries
out a process to produce a result of determination as to whether or not the state of the release button employed in the input device 16 is still a half-pressed state as it is. If the determination result produced at the step S503 indicates that the state of the release button is still a half-pressed state as it is as indicated by the determination result produced at the step S501, the flow of the processing goes back to the step S503 in order to sustain the present state. The process of the step S503 is carried out repeatedly till the determination result produced at the step S503 indicates that the state of the release button is no longer a half-pressed state. As the determination result produced at the step S503 indicates that the state of the release button is not a half-pressed state anymore, the flow of the processing goes on to a step S504 at which the control circuit 18 carries out a process to produce a result of determination as to whether or not the state of the release button is a completely pressed state. If the determination result produced at the step S504 indicates that the state of the release button is not a completely pressed state, that is, if the determination result produced at the step S504 indicates that the state of the release button has been released, the flow of the processing goes back to the
step S501. That is to say, if the photographer releases its finger from the release button so that, as a result, the state of the release button is neither a half-pressed state nor a completely pressed state, the flow of the processing goes back to the step S501 to repeat the same processing.
If the determination result produced at the step S504 indicates that the state of the release button is a completely pressed state, on the other hand, the flow of the processing goes on to a step S505 at which the control circuit 18 carries out a process to determine a proper exposure time Tp once more. That is to say, the control circuit 18 again determines a proper exposure time Tp in the same way as the step S502 by using the measured value received from the exposure meter 19 as the measured value of the brightness of the photographing object and the diaphragm value F determined at the step S502. The control circuit 18 again determines a proper exposure time Tp because, by using a proper exposure time Tp obtained with a timing closer to a timing to actually carry out a photographing operation, a more proper image can be generated. It is to'be noted that the reciprocal 1/Tp is a shutter speed at the proper exposure.
After the process of the step S505 is completed,
the flow of the processing goes on to the next step S506 at which the control circuit 18 carries out the photographing-mode determination processing to make a decision of which of the first photographing processing, the second photographing processing, the third photographing processing and the fourth photographing processing is to be performed as photographing processing on the basis of the actual exposure time Ta determined at the step S502 and the proper exposure time Tp determined at the step S505. That is to say, the control circuit 18 selects either the first photographing processing, the second photographing processing, the third photographing processing or the fourth photographing processing as photographing processing. As will be described later in detail, the photographing-mode determination processing is carried out as processing to determine a photographing mode in which the photographing processing is performed to generate a sharp output image (almost) without effects of hand trembling.
In addition, if the control circuit 18 makes a decision to carry out the second photographing processing as photographing processing at the step S506, the control circuit 18 also determines the taken image count Na for the second photographing processing. By the same token,
if the control circuit 18 makes a decision to carry out the third photographing processing as photographing processing at the step S506, the control circuit 18 also determines the taken image count Nb for the third photographing processing. In the same way, if the control circuit 18 makes a decision to carry out the fourth photographing processing as photographing processing at the step S506, the control circuit 18 also determines the taken image count Nc for the third photographing processing. After the process of the step S506 is completed, the flow of the processing goes on to the next step S507.
At the step S507, the digital camera 1 carries out the photographing processing selected at the step S506.
That is to say, if the control circuit 18 makes a decision to carry out the first photographing processing as photographing processing at the step S506, the digital camera 1 carries out the first photographing processing to take an image at the diaphragm value F and the actual exposure time Ta, which have been determined at the step S502, and output the taken image as an output image.
If the control circuit 18 makes a decision to carry out the second photographing processing as photographing processing at the step S506, the digital camera 1 carries
out the second photographing processing to carry out an image-pickup process without executing the binning function of the image-pickup device 4 to produce Na taken images at a high speed corresponding to an exposure time of Ta/Na per taken image and at the diaphragm value F determined at the step S502 along with the actual exposure time Ta. Then, the digital camera 1 carries out the image generation processing by using the Na taken images to generate an output image.
If the control circuit 18 makes a decision to carry out the third photographing processing as photographing processing at the step S506, the digital camera 1 carries out the third photographing processing to carry out an image-pickup process by executing the 2^2 binning function of the image-pickup device 4 to produce Nb taken images at a high speed corresponding to an exposure time of Ta/Nb per taken image and at the diaphragm value F determined at the step S502 along with the actual exposure time Ta. Then, the digital camera 1 carries out the image generation processing by using the Nb taken images to generate an output image.
If the control circuit 18 makes a decision to carry out the fourth photographing processing as photographing processing at the step S506, the digital
camera 1 carries out the fourth photographing processing to carry out an image-pickup process by executing the 3><3 binning function of the image-pickup device 4 to produce Nc taken images at a high speed corresponding to an exposure time of Ta/Nc per taken image and at the diaphragm value F determined at the step S502 along with the actual exposure time Ta. Then, the digital camera 1 carries out the image generation processing by using the Nc taken images to generate an output image.
It is to be noted that the output image generated in the first photographing processing of course has the same number of pixels as the pixel count W*H of the image-pickup device 4. As a matter of fact, the output image generated in the second, third, or fourth photographing processing also has the same number of pixels as the pixel count W*H of the image-pickup device 4 as well.
After the process of the step S507 is completed, the flow of the processing goes on to the next step S508 at which the control circuit 18 carries out a process to produce a result of determination as to whether or not the state of the release button employed in the input device 16 is a half-pressed state or a completely pressed state. If the determination result produced at the step
S508 indicates that the state of the release button is a half-pressed state or a completely pressed state, the process of the step S508 is carried out repeatedly till the determination result indicates that the state of the release button is neither a half-pressed state nor a completely pressed state. As the determination result produced at the step S508 indicates that the state of the release button is neither a half-pressed state nor a completely pressed state, that is, as the photographer releases its finger from the release button to release the release button, the flow of the processing goes back to the first step S501 to prepare for the next photographing operation.
The processing of the steps S501 to S508 described above is carried out on the assumption that the power supply of the digital camera 1 has been turned on. If the power supply is turned off, the processing is suspended. In addition, the operation carried out by the photographer on the exposure correction dial is effective only while the release button is being in a released state. Thus, the process carried out at the step S502 to determine the actual exposure time Ta uses the exposure correction value, which is set by the exposure correction dial after the determination result produced at the step
S501 immediately preceding the step S502 indicates that the release button is in a half-pressed state.
Next, the following description explains the photographing-mode determination processing carried out at the step S506 of the flowchart shown in FIG. 44. Before the photographing-mode determination processing is explained, however, the first to fourth photographing processings are described more as follows.
In the first photographing processing, the image-pickup device 4 obtains one taken image at a diaphragm value F and an actual exposure time of Ta seconds, supplying the taken image to the image processing circuit 17. The image processing circuit 17 generates an output image from the taken image. That is to say, the image processing circuit 17 outputs the taken image as the output image. In other words, in the first photographing processing, the image-pickup device 4 obtains a taken image without executing a binning function to sum up pixel values and provides the image processing circuit 17 with the taken image having a pixel count equal to the number of pixels composing the image-pickup device 4. Then, the image processing circuit 17 generates an output image having a pixel count equal to the number of pixels composing the image-pickup device 4. It is to be noted
that the control circuit 18 controls the first photographing processing. In addition, the first photographing processing to generate an output image from a taken image is virtually the same as the photographing processing carried out by an ordinary digital camera.
In the second photographing processing, the image-pickup device 4 consecutively produces a plurality of taken images, say, Na taken images at a high speed corresponding to an exposure time of Ta/Na per taken image and at a diaphragm value F, supplying each of the taken images to the image processing circuit 17. Then, the image processing circuit 17 carries out the image generation processing described above in order to generate an output image from the Na taken images.
To put it in detail, in the second photographing processing, the image-pickup device 4 produces the taken images without execution of the binning function to sum up pixel values and supplies each of the taken images each having a pixel count equal to the number of pixels composing the image-pickup device 4 to the image processing circuit 17. The image processing circuit 17 adjusts the positions of the Na taken images received from the image-pickup device 4 and carries out the image generation processing applying a weighted addition
process to the Na taken images with their positions adjusted in order to generate a sharp output image having a pixel count egual to the number of pixels composing the image-pickup device 4. It is to be noted that, since the image-pickup device 4 consecutively produces Na taken images at a high speed corresponding to an exposure time of Ta/Na per taken image in the second photographing processing, the exposure time it takes to obtain all the Na taken images is (Ta/Na) xNa (= Ta) . The control circuit 18 also controls the second photographing processing.
In the third photographing processing, the image-pickup device 4 consecutively produces a plurality of taken images, say, Nb taken images at a high speed corresponding to an exposure time of Ta/Nb per taken image and at a diaphragm value F by execution of the 2*2 binning function, supplying each of the taken images to the image processing circuit 17. Then, the image processing circuit 17 carries out the image generation processing described above in order to generate an output image from the Nb taken images.
To put it in detail, in the third photographing processing, the image-pickup device 4 produces the taken images by execution of the 2x2 binning function as described above to sum up pixel values and supplies each
of the taken images each having a pixel count equal to one-fourth of the number of pixels composing the image-pickup device 4 to the image processing circuit 17. The image processing circuit 17 adjusts the positions of the Nb taken images received from the image-pickup device 4 and carries out the image generation processing applying a weighted addition process to the Nb taken images with their positions adjusted in order to- generate a sharp output image having a pixel count equal to the number of pixels composing the image-pickup device 4. It is to be noted that, since the image-pickup device 4 consecutively produces Nb taken images at a high speed corresponding to an exposure time of Ta/Nb per taken image in the third photographing processing, the exposure time it takes to obtain all the Nb taken images is (Ta/Nb) xNb (= Ta) . The control circuit 18 also controls the third photographing processing.
In the fourth photographing processing, the image-pickup device 4 consecutively produces a plurality of taken images, say, Nc taken images at a high speed corresponding to an exposure time of Ta/Nc per taken image and at a diaphragm value F by execution of the 3x3 binning function, supplying each of the taken images to the image processing circuit 17. Then, the image
processing circuit 17 carries out the image generation processing described above in order to generate an output image from the Nc taken images.
To put it in detail, in the fourth photographing processing, the image-pickup device 4 produces the taken images by execution of the 3*3 binning function as described above to sum up pixel values and supplies each of the taken images each having a pixel count equal to one-ninth of the number of pixels composing the image-pickup device 4 to the image processing circuit 17. The image processing circuit 17 adjusts the positions of the Nc taken images received from the image-pickup device 4 and carries out the image generation processing applying a weighted addition process to the Nc taken images with their positions adjusted in order to generate a sharp output image having a pixel count equal to the number of pixels composing the image-pickup device 4. It is to be noted that, since the image-pickup device 4 consecutively produces Nc taken images at a high speed corresponding to an exposure time of Ta/Nc per taken image in the fourth photographing processing, the exposure time it takes to obtain all the Nc taken images is (Ta/Nc) *NC (= Ta) . The control circuit 18 also controls the fourth photographing processing.
Next, by referring to a flowchart shown in FIG. 45, the following description explains the photographing-mode determination processing carried out at the step S506 of the flowchart shown in FIG. 44. It is to be noted that the control circuit 18 is assumed to have recognized information on a focal distance of the lens 2, the darkness limit l/Mmax and a continuous image-pickup interval limit to of the image-pickup device 4. If a zoom lens is employed, the information on a focal distance of the lens 2 is information on a focal distance in a state in which an attempt is made to carry out a photographing operation.
The flowchart shown in FIG. 45 begins with a step S521 at which the control circuit 18 carries out first determination processing to produce a result of determination as to whether or not a proper output image can be obtained on the basis of a condition for obtaining a proper output image by execution of the first photographing processing on the assumption that a decision has been made to perform the first photographing processing as the photographing processing. Then, the flow of the photographing-mode determination processing goes on to the next step S522. Details of the first determination processing will be described later by
referring to a flowchart shown in FIG. 46.
At the step S522, the control circuit 18 examines the result of the first determination processing carried out at the step S521. If the examination carried out at the step S522 reveals the fact that the result of the first determination processing indicates that a proper output image can be obtained by carrying out the first photographing processing, the flow of the photographing-mode determination processing goes on to a step S523 at which the control circuit 18 makes a decision to carry out the first photographing processing. Then, control of execution is returned to the calling program.
If the examination carried out at the step S522 reveals the fact that the result of the first determination processing indicates that a proper output image cannot be obtained by carrying out the first photographing processing, on the other hand, the flow of the photographing-mode determination processing goes on . to a step S524 at which the control circuit 18 carries out second determination processing to produce a result of determination as to whether or not a proper output image can be obtained on the basis of a condition for obtaining a proper output image by execution of the second photographing processing on the assumption that a
decision has been made to perform the second photographing processing as the photographing processing. Then, the flow of the photographing-mode determination processing goes on to the next step S525. Details of the second determination processing will be described later by referring to a flowchart shown in FIG. 47. If the second determination processing carried out at the step S524 produces a result indicating that a proper output image can be obtained by carrying out the second photographing processing, the second determination processing also sets the taken-image count Na at the same step.
At the step S525, the control circuit 18 examines the result of the second determination processing carried out at the step S524. If the examination carried out at the step S525 reveals the fact that the result of the second determination processing indicates that a proper output image can be obtained by carrying out the second photographing processing, the flow of the photographing-mode determination processing goes on to a step S526 at which the control circuit 18 makes a decision to carry out the second photographing processing. Then, control of execution is returned to the calling program.
If the examination carried out at the step S525
reveals the fact that the result of the second determination processing indicates that a proper output image cannot be obtained by carrying out the second photographing processing, on the other hand, the flow of the photographing-mode determination processing goes on to a step S527 at which the control circuit 18 carries out third determination processing to produce a result of determination as to whether or not a proper output image can be obtained on the basis of a condition for obtaining a proper output image by execution of the third photographing processing on the assumption that a decision has been made to perform the third photographing processing as the photographing processing. Then, the flow of the photographing-mode determination processing goes on to the next step S528. Details of the third determination processing will be described later by referring to a flowchart shown in FIG. 48. If the third determination processing carried out at the step S527 produces a result indicating that a proper output image can be obtained by carrying out the third photographing processing, the third determination processing also sets the taken-image count Nb at the same step.
At the step S528, the control circuit 18 examines the result of the third determination processing carried
out at the step S527. If the examination carried out at the step S528 reveals the fact that the result of the third determination processing indicates that a proper output image can be obtained by carrying out the third photographing processing, the flow of the photographing-mode determination processing goes on to a step S529 at which the control circuit 18 makes a decision to carry out the third photographing processing. Then, control of execution is returned to the calling program.
If the examination carried out at the step S528 reveals the fact that the result of the third determination processing indicates that a proper output image cannot be obtained by carrying out the third photographing processing, on the other hand, the flow of the photographing-mode determination processing goes on to a step S530 at which the control circuit 18 carries out fourth determination processing to produce a result of determination as to whether or not a proper output image can be obtained on the basis of a condition for obtaining a proper output image by execution of the fourth photographing processing on the assumption that a decision has been made to perform the fourth photographing processing as the photographing processing. Then, the flow of the photographing-mode determination
processing goes on to the next step S531. Details of the fourth determination processing will be described later by referring to a flowchart shown in FIG. 49. If the fourth determination processing carried out at the step S530 produces a result indicating that a proper output image can be obtained by carrying out the fourth photographing processing, the fourth determination processing also sets the taken-image count Nc at the same step.
At the step S531, the control circuit 18 examines the result of the fourth determination processing carried out at the step S530. If the examination carried out at the step S531 reveals the fact that the result of the fourth determination processing indicates that a proper output image can be obtained by carrying out the fourth photographing processing, the flow of the photographing-mode determination processing goes on to a step S532 at which the control circuit 18 makes a decision to carry out the fourth photographing processing. Then, control of execution is returned to the calling program.
If the examination carried out at the step S531 reveals the fact that the result of the fourth determination processing indicates that a proper output image cannot be obtained by carrying out the fourth
photographing processing, on the other hand, the flow of the photographing-mode determination processing goes on to a step S533 at which the control circuit 18 makes a decision to carry out the first photographing processing. Then, control of execution is returned to the calling program. This is because, in this case, it is difficult to generate a proper output image without regard to whether the first photographing processing, the second photographing processing, the third photographing processing or the fourth photographing processing is carried out. For this reason, while recognizing the existence of hand trembling, the control circuit 18 makes a decision to carry out the first photographing processing in the same way as the photographing processing carried out by an ordinary digital camera. It is to be noted that, when the control circuit 18 makes a decision to carry out the first photographing processing at the step S533, a warning message stating: "There is a risk of hand-trembling occurrence" may be displayed on the monitor 11 or the like.
The photographing-mode determination processing represented by the flowchart shown in FIG. 45 is carried out to determine whether or not a proper output image can be obtained in the first, second, third, or fourth
determination processing in an order starting with the first determination processing followed by the second determination processing followed by the third determination processing followed by the fourth determination processing. This is because an output image with a good picture quality can be obtained in an order starting with the first photographing processing followed by the second followed by the third followed by the fourth. That is to say, an output image generated in the ordinary photographing mode has a picture quality better than an output image generated in the hand-trembling correction photographing mode. In addition, an output image generated in the hand-trembling correction photographing mode without execution of a binning function has a picture quality better than an output image generated in the hand-trembling correction photographing mode by execution of a binning function. Furthermore, an output image generated in the hand-trembling correction photographing mode by execution of a binning function of a small scale has a picture quality better than an output image generated in the hand-trembling correction photographing mode by execution of a binning function of a large scale. The scale of the binning function is defined as the number of pixel valuessummed up by the function.
It is to be noted that the input device 16 can be provided with a button for forcibly excluding the photographing processing to be carried out in the hand-trembling correction photographing mode. As described above, the photographing processing to be carried out in the hand-trembling correction photographing mode can be the second, third, or fourth photographing processing. Thus, when this button is pressed, the photographing-mode determination processing represented by the flowchart shown in FIG. 45 is not carried out at the step S506 of the flowchart shown in FIG. 44. Instead, the first photographing processing is carried out by force.
In addition, the input device 16 can be provided with a button for forcibly excluding the binning function, That is to say, this button is pressed in order to forcibly exclude the third and the fourth photographing processing so that, as photographing processing, only either the first or the second photographing processing is selected.
This is because, when the binning function is executed, a taken image output by the image-pickup device 4 has a pixel count smaller than the number of pixels on the image-pickup device 4. Thus, in some cases, an output
image generated from such taken images has poor brightness to a certain degree in comparison with an output image generated without execution of a binning function. For this reason, by allowing the photographer to determine whether or not the third and the fourth photographing processing are to be excluded by force, the convenience offered by the digital camera 1 can be enhanced. It is to be noted that, when the button for excluding the third and the fourth photographing processing by force is pressed, in the photographing-mode determination processing represented by the flowchart shown in FIG. 45, the flow of the processing goes on from the step S525 to the step S533 instead of going on from the step S525 to the step S527.
As described above, at the step S521 of the flowchart shown in FIG. 45, the first determination processing is carried out to determine as to whether or not a proper output image can be obtained on the basis of a condition for obtaining a proper output image by execution of the first photographing processing. By the same token, at the step S524, the second determination processing is carried out by execution of the second photographing processing, at the step S527, the third determination processing is carried out by execution of
the third photographing processing, and at the step S530, the fourth determination processing is carried out by execution of the fourth photographing processing. The following description explains each of the conditions for obtaining a proper output image.
In order for an output image to be regarded as a proper output image, first of all, the output image is required to have no blurring portion caused by hand trembling.
In order to prevent a blurring portion from being generated by hand trembling on an output image, in the case of the first photographing processing, it is necessary to prevent a blurring portion from being generated by hand trembling on the one taken image obtained in one photographing operation. In the case of the second, third, and fourth photographing processings, on the other hand, it is necessary to prevent a blurring portion from being generated by hand trembling on each of N taken images each obtained at a high speed in the photographing operation.
That is to say, if any of the taken images includes
v
a blurring portion resulting from hand trembling, for example, an output image generated from the taken images including a taken image with a blurring portion will also
include a blurring portion. Thus, a taken image obtained by the image-pickup device 4 must not be an image including a blurring portion without regard to whether the taken image is obtained by carrying out the first, second, third or fourth photographing processing as the photographing processing.
In order to prevent a blurring portion from being generated by hand trembling on a taken image (or, strictly speaking, one taken image) obtained by the image-pickup device 4, the exposure time for obtaining the one taken image must be short in comparison with the time it takes to move the digital camera 1 by a certain distance due to hand trembling.
In general, in a photographing operation carried out by using a hand-held camera, if a 35 mm camera is taken as a model in the computation of the exposure time, an exposure time (expressed in terms of seconds) shorter than I/focal distance (expressed in terms of mm) is said to be an exposure time resulting in a taken image including no blurring portion caused by hand trembling. Thus, let us assume for example that the focal distance of a lens employed in the 35 mm camera taken as a model in the computation of the exposure time is 30 mm. In this case, a taken image obtained at an exposure time not
exceeding 1/30 seconds does not include a blurring portion caused by hand trembling.
Now, let us transform the focal distance of the lens 2 employed in the digital camera 1 into the focal distance of a lens employed in the 35mm camera taken as a model in the computation of the exposure time. In this case, the expression I/(focal distance) expressing the reciprocal of the focal distance is referred to as a threshold value Tbiur serving as the upper limit of exposure times that do not result in a blurring portion caused by hand trembling on taken images. Thus, since an output image generated from such taken images also does not include a blurring portion caused by hand trembling, a proper output image can be obtained. In the following description, a condition requiring that the exposure time for obtaining a single taken image be equal to or shorter than the threshold value Tbiur is appropriately referred to as a first condition for obtaining a proper output image.
By setting the threshold value Tbiur at the reciprocal of the focal distance of a lens employed in the 35mm camera taken as a model in the computation of the exposure time as described above, in an actual photographing operation, it is possible to determine whether or not an image without an effect of hand
trembling can be obtained without checking whether or not hand trembling-actually occurs (or whether or not actually occurring hand trembling is within a range of tolerance) . To put it concretely, if a single taken image is obtained at an exposure time longer than the threshold value Tbiur set as described above, that is, if a single taken image is obtained at a relatively low shutter speed, it is possible to determine that an output image generated from the taken image will include a blurring portion caused by hand trembling. If a single taken image is obtained at an exposure time shorter than the threshold value Tbiur set as described above, that is, if a single taken image is obtained at a relatively high shutter speed, on the other hand, it is possible to determine that an output image generated from the taken image will include no blurring portion caused by hand trembling.
It is to be noted that, since the degree of hand trembling varies from person to person, the input device 16 may be provided with a button for changing the magnitude of the threshold value Tbiur • Thus, the photographer may operate this button to change the magnitude of the threshold value Tbiur to a value according to the operation carried out by the photographer. In
addition, as described earlier, the input device 16 also provides the control circuit 18 with information on a focal distance set by an operation carried out on a zoom button employed in the input device 16. By using the information on a focal distance, the control circuit 18 carries out processes such conversion into a 35 mm camera used as a model in computation of an exposure time and calculation of the reciprocal of the focal distance.
If each of taken images obtained consecutively in the second, third, or fourth photographing processing carried out in the hand-trembling correction photographing mode is extremely too dark so that the photographing object on the taken images is unavoidably buried under noises, an output image generated from such taken images in image generation processing will also be inevitably dark and buried under noises. Thus, it is not possible to obtain a proper output image or a sharp output image including no blurring portion caused by hand trembling.
Thus, in order to obtain a proper output image, each of a plurality of taken images obtained consecutively must not be such too dark that the photographing object is buried under noises.
In order to obtain taken images that are not such
too dark, as described earlier, l/Mmax of the brightness of a single taken image obtained at the proper exposure time Tp is used as the upper limit of darkness levels causing such a too dark image that the photographing object on the image is unavoidably buried under noises. By setting the upper limit of darkness levels at l/Mmax of the brightness of a single taken image in this way, if the exposure time of a taken image is at least equal to the l/Mraax of the proper exposure time Tp, a taken image obtained at the exposure time will never become such too dark that the photographing object on the taken image is unavoidably buried under noises.
As described above, if the exposure time of a taken image is at least equal to the value of Tp/Mmax computed as a product resulting from multiplication of the proper exposure time Tp by l/Mmax representing the upper limit of darkness levels, a taken image obtained at the exposure time will never become such too dark that the photographing object on the taken image is unavoidably buried under noises. In the following description, a condition requiring that the exposure time for obtaining a taken image be equal to or longer than the value of Tp/Mmax computed as a product resulting from multiplication of the proper exposure time Tp by l/Mmax representing the
upper limit of darkness levels is appropriately referred to as a second condition for obtaining a proper output image.
Next, a shortest period of time from the start of a process carried out by the image-pickup device 4 to output a taken image to the end of the operation is referred to as an output minimum time. That is to say, the image-pickup interval of the highest-speed continuous image-pickup time of the image-pickup device 4 is referred to as an output minimum time. In this case, if the exposure time of each of taken images obtained consecutively in the second, third, or fourth photographing processing carried out in the hand-trembling correction photographing mode at a high speed is shorter than the output minimum time, that is, if the exposure time of a taken image is shorter than the output minimum time, a blanking period is generated in the highspeed image-pickup process to obtain a plurality of taken images. A blanking period is a period between the end of an exposure for obtaining a specific taken image and the beginning of an exposure for obtaining a taken image immediately following the specific taken image. During a blanking period, there is no image-pickup (or exposure) process.Details of an effect of a blanking period on an output image will be explained later. In a few words, if a blanking period is generated and the object of photographing moves, the movement of the photographing object during the blanking period is not reflected on the output image. Thus, an output image is generated as an unnatural image showing a discontinuous movement of the photographing object. As a result, a proper output image or a sharp image including no blurring portion caused by hand trembling cannot be generated.
Thus, in order to obtain a proper output image, the exposure time for obtaining a taken image in a high-speed image-pickup process must be at least equal to the output minimum time. In the following description, a condition requiring that the exposure time for obtaining a taken image be at least equal to the output minimum time is appropriately referred to as a third condition for obtaining a proper output image.
It is to be noted that, if the binning function is not executed, the output minimum time is equal to the aforementioned continuous image-pickup interval limit to of the image-pickup device 4. If the 2x2 binning function is executed in the image-pickup device 4, the output minimum time is equal to to/4 and, if the 3*3 binning
function is executed in the image-pickup device 4, the output minimum time is equal to to/9.
Next, by referring to a flowchart shown in FIG. 46, the following description explains the first determination processing carried out at the step S521 of the flowchart shown in FIG. 45 to produce a result of determination as to whether or not a proper output image can be obtained by execution of the first photographing processing. It is to be noted that the control circuit 18 controls the execution of the first photographing processing.
The flowchart shown in FIG. 46 begins with a step S541 at which the control circuit 18 produces a result of determination as to whether or not the first condition is satisfied, that is, whether or not the actual exposure time Ta is equal to or shorter than the threshold value Tbiur- In this case, the actual exposure time Ta has been determined at the step S502 of the flowchart shown in FIG, 44. On the other hand, the threshold value Tbiur is a quantity found by the control circuit 18 from information on a focal distance as described earlier.
If the determination result produced at the step S541 indicates that the actual exposure time Ta is equal to or shorter than the threshold value Tbiur, the flow of
the first determination processing goes on to a step S542 at which the control circuit 18 produces a determination result of the first determination photographing as a result indicating that a proper output image can be obtained by execution of the first photographing processing. Finally, control of the processing execution is returned to the calling program.
If the determination result produced at the step S541 indicates that the actual exposure time Ta is longer than the threshold value Tbiur, on the other hand, the flow of the first determination processing goes on to a step S543 at which the control circuit 18 produces a determination result of the first determination photographing as a result indicating that a proper output image cannot be obtained by execution of the first photographing processing. Finally, control of the processing execution is returned to the calling program.
If the actual exposure time Ta is equal to or shorter than the threshold value Tbiur, an image-pickup process carried out at the actual exposure time Ta seconds will result in a taken image including no blurring portion caused by hand trembling and, thus, a sharp output image can be generated from such a taken image. Conversely speaking, if the actual exposure time Ta is
longer than the threshold value Tbiur, on the other hand, a taken image obtained at such a long exposure time may include blurring portions each caused by hand trembling. Thus/ it is quite within the bounds of possibility that a sharp output image cannot be generated from such a taken image.
Next, by referring to a flowchart shown in FIG. 47, the following description explains the second determination processing carried out at the step S524 of the flowchart shown in FIG. 45 to produce a result of determination as to whether or not a proper output image can be obtained from a plurality of taken images obtained by carrying out the second photographing processing without execution of a binning function. It is to be noted that the control circuit 18 controls the execution of the second photographing processing.
The flowchart shown in FIG. 47 begins with a step S551 at which the control circuit 18 uses the larger one of two threshold values, i.e., the threshold value Tp/Mmax and the threshold value to, as an exposure time Ttmp for obtaining a taken image in an image-pickup process carried out at a high speed. The threshold value Tp/Mmax is a value computed as a product resulting from multiplication of the proper exposure time Tp by l/Mmax
representing the upper limit of darkness levels. On the other hand, the threshold value to is equal to the output minimum time for a case in which no binning function is executed. The threshold value to is a threshold value of image-pickup intervals of a continuous image-pickup process carried out at a highest speed. It is to be noted that, as described earlier, the value of l/Mmax representing the upper limit of darkness levels is a value determined in accordance with the performance of the digital camera 1 as a value for finding the threshold value Tp/Mmax- By the same token, the threshold value t0 representing the limit of image-pickup intervals of a continuous image-pickup process carried out at a highest speed is also a value determined in accordance with the performance of the digital camera 1. It is assumed that the control circuit 18 has recognized the value of l/Mmax and the threshold value to. The proper exposure time Tp for finding the threshold value Tp/Mmax has been found at the step S505 of the flowchart shown in FIG. 44.
By setting the exposure time Ttmp for obtaining a taken image in an image-pickup process carried out at a high speed at the larger one of the two threshold values, i.e., the threshold value Tp/Mmax and the threshold value to, the exposure time Ttmp satisfies the second and third
conditions simultaneously. As described above, the second condition is a condition requiring that the exposure time Ttmp for obtaining a taken image be equal to or longer than the value of Tp/Mmax. On the other hand, the third condition is a condition requiring that the exposure time Ttmp for obtaining a taken image be at least equal to the output minimum time to. That is to say, in this case, the
second condition is expressed by the relation Ttmp = Tp/Mmax whereas the third condition is expressed by the
relation TtmP = t0.
Let us assume for example that the relation to < Tp/Mmax holds true. In this case, at the step S551, the exposure time Ttmp is set at the value of Tp/Mmax or Ttmp = Tp/Mmax. Thus, the exposure time Ttmp simultaneously satisfies the relation Ttmp = Tp/Mmax expressing the second condition and the relation Ttmp = to expressing the third condition. If the relation Tp/Mmax < t0 holds true, on the other hand, at the step S551, the exposure time Ttmp is set at the value of T0 or Ttmp = T0. Thus, the exposure time Ttmp also simultaneously satisfies the relation Ttmp ^ Tp/Mmax expressing the second condition and the relation
Ttmp ^ to expressing the third condition.
After the process of the step S551 is completed, the flow of the second determination processing goes on
to the next step S552 at which the control circuit 18 produces a result of determination as to whether or not the exposure time Ttmp is equal to or shorter than the threshold value Tbiur. The fact that the exposure time Ttmp is equal to or shorter than the threshold value Tbiur indicates that, for the exposure time Ttmp, the first condition is satisfied. The fact that the exposure time Ttmp is neither equal to nor shorter than the threshold value Tbiur indicates that, for the exposure time Ttrnp, the first condition is not satisfied.
It is to be noted that, in this case, at the step S551, the exposure time Ttmp has been set at the larger one of two threshold values, i.e., the threshold value Tp/Mmax and the threshold value t0. That is to say, the exposure time Ttmp has been set at a minimum value satisfying both the second and third conditions. Then, at the step S552, the exposure time Ttmp is examined to produce a result of determination as to whether or not the exposure time Ttmp is equal to or shorter than the threshold value Tbiur, that is, whether or not the exposure
time Ttmp satisfies the relation Ttmp ^ TbiUr expressing the first condition. The fact that the exposure time Ttmp set at the step S551 does not satisfy the first condition indicates that, if a smaller exposure time Ttmp is used as
a value satisfying the first condition, the smaller exposure time Ttmp will definitely not satisfy the second or third condition. Thus, with the exposure time Ttmp set at the step S551 at the larger one of two threshold values, i.e., the threshold value Tp/Mmax and the threshold value to, the process carried out at the step S552 following the step S551 to produce a result of determination as to whether or not the exposure time Ttmp
satisfies the relation Ttmp = TbiUr expressing the first condition can be seen from another point of view as a process equivalent to a process to produce a result of determination as to whether or not the exposure time Ttmp satisfying the first condition also satisfies the second and third conditions as well.
If the determination result produced at the step S552 indicates that the exposure time Ttmp is equal to or shorter than the threshold value Tbiurr the flow of the second determination processing goes on to a step S553 at which the control circuit 18 computes a taken-image count Na of the second photographing processing by dividing the actual exposure time Ta by the exposure time Ttmp of one taken image. That is to say, the control circuit 18 computes the value of Na as a quotient Ta/Ttmp- The taken-image count Na represents the number of taken images
obtained in a high-speed image-pickup process of the second photographing processing. The actual exposure time Ta is the exposure time of photographing processing carried out in the ordinary photographing mode. In other words, the taken-image count Na satisfying the relation Na = Ta/Ttmp is determined by using the exposure time Ttmp set at the step S551 and found at the step S552 to be an exposure time satisfying the first to third conditions. Thus, the exposure time Ttmp of each of the Na taken images obtained at a high image-pickup speed in the second photographing processing can be expressed by a quotient Ta/Na, which satisfies the first to third conditions. That is to say, the taken-image count Na has a value varying in accordance with the exposure time Ta in this way. It is to be noted that the taken-image count Na is obtained as a result of a division operation by typically truncating the fraction part following the decimal point of the result.
After the process of the step S553 is completed, the flow of the second determination processing goes on to a step S554 at which the control circuit 18 produces a determination result of the second determination photographing as a result indicating that a proper output image can be obtained by execution of the second
photographing processing. Finally, control of the processing execution is returned to the calling program.
If the determination result produced at the step S552 indicates that the exposure time Ttmp is longer than the threshold value Tbiur/- on the other hand, the flow of the second determination processing goes on to a step S555 at which the control circuit 18 produces a determination result of the second determination photographing as a result indicating that a proper output image cannot be obtained by execution of the second photographing processing. Finally, control of the processing execution is returned to the calling program.
Next, by referring to a flowchart shown in FIG. 48, the following description explains the third determination processing carried out at the step S527 of the flowchart shown in FIG. 45 to produce a result of determination as to whether or not a proper output image can be obtained from a plurality of taken images obtained by carrying out the third photographing processing with execution of the 2x2 binning function. It is to be noted that the control circuit 18 controls the execution of the third photographing processing.
The flowchart shown in FIG. 48 begins with a step S561 at which the control circuit 18 uses the larger one
of 2 threshold values, i.e., the threshold value Tp/Mmax and the threshold value to/4, as an exposure time Ttmp for obtaining a taken image in an image-pickup process carried out at a high speed. The threshold value Tp/Mmax is a value computed as a product resulting from multiplication of the proper exposure time Tp by l/Mmax representing the upper limit of darkness levels. On the other hand, the threshold value to/4 is equal to the output minimum time for a case in which the 2x2 binning function is executed. It is to be noted that, as described earlier, the value of l/Mmax representing the upper limit of darkness levels is a value determined in accordance with the performance of the digital camera 1 as a value for finding the threshold value Tp/Mmax. By the same token, the threshold value to representing the limit of image-pickup intervals of a continuous image-pickup process carried out at a highest speed is also a value determined in accordance with the performance of the digital camera 1. It is assumed that the control circuit 18 has recognized the value of l/Mmax and the threshold value to. The proper exposure time Tp for finding the threshold value Tp/Mmax has been found at the step S505 of the flowchart shown in FIG. 44.
After the process of the step S561 is completed,
the flow of the second determination processing goes on to the next step S562 at which the control circuit 18 produces a result of determination as to whether or not the exposure time Ttmp is equal to or shorter than the threshold value TbiUr-
An exposure time Ttmp not exceeding the threshold value Tbiur satisfies the first condition applied to the exposure time Ttmp- On the other hand, an exposure time Ttmp exceeding the threshold value Tbiur does not satisfy the first condition applied to the exposure time Ttmp •
With the exposure time Ttmp set at the step S561 at the larger one of two threshold values, i.e., the threshold value Tp/Mmax and the threshold value to/4, the process carried out at the step S562 following the step S561 to produce a result of determination as to whether or not the exposure time Ttmp satisfies the relation Ttmp = Tbiur expressing the first condition can be regarded as a process said to be equivalent to a process to produce a result of determination as to whether or not the exposure time Ttmp satisfying the first condition also satisfies the second and third conditions as well, as is the case with the second determination processing represented by the flowchart shown in FIG. 47.
If the determination result produced at the step
S562 indicates that the exposure time Ttmp is equal to or shorter than the threshold value Tbiur, the flow of the third determination processing goes on to a step S563 at which the control circuit 18 computes a taken-image count Nb of the third photographing processing by dividing the actual exposure time Ta by the exposure time Ttmp of one taken image. That is to say, the control circuit 18 computes the value of Nb as a quotient Ta/Ttmp- The taken-image count Nb represents the number of taken images obtained in. a high-speed image-pickup process of the third photographing processing. In other words, the taken-image count Nb satisfying the relation Nb = Ta/Ttmp is determined by using the exposure time Ttmp set at the step S561 and found at the step S562 to be an exposure time satisfying the first to third conditions. Thus, the exposure time Ttmp of each of the Nb taken images obtained at a high image-pickup speed in the third photographing processing can be expressed by a quotient Ta/Nb, which satisfies the first to third conditions. That is to say, the taken-image count Nb has a value varying in accordance with the exposure time Ta in this way. It is to be noted that the taken-image count Nb is obtained as a result of a division operation by typically truncating the fraction part following the decimal point of the result.
After the process of the step S563 is completed, the flow of the second determination processing goes on to a step S564 at which the control circuit 18 produces a determination result of the third determination photographing as a result indicating that a proper output image can be obtained by execution of the third photographing processing. Finally, control of the processing execution is returned to the calling program.
If the determination result produced at the step S562 indicates that the exposure time Ttmp is longer than the threshold value Tbiurr on the other hand, the flow of the third determination processing goes on to a step S565 at which the control circuit 18 produces a determination result of the third determination photographing as a result indicating that a proper output image cannot be obtained by execution of the third photographing processing. Finally, control of the processing execution is returned to the calling program.
Next, by referring to a flowchart shown in FIG. 49, the following description explains the fourth determination processing carried out at the step S530 of the flowchart shown in FIG. 45 to produce a result of determination as to whether or not a proper output image can be obtained from a plurality of taken images obtained
by carrying out the fourth photographing processing with execution of the 3x3 binning function. It is to be noted that the control circuit 18 controls the execution of the fourth photographing processing.
The flowchart shown in FIG. 49 begins with a step S571 at which the control circuit 18 uses the larger one of two threshold values, i.e., the threshold value Tp/Mmax and the threshold value to/9, as an exposure time Ttmp for obtaining a taken image in an image-pickup process carried out at a high speed. The threshold value Tp/Mmax is a value computed as a product resulting from multiplication of the proper exposure time Tp by l/Mmax representing the upper limit of darkness levels. On the other hand, the threshold value to/9 is equal to the output minimum time for a case in which the 3X3 binning function is executed. It is to be noted that, as described earlier, the value of l/Mmax representing the upper limit of darkness levels is a value determined in accordance with the performance of the digital camera 1 as a value for finding the threshold value Tp/Mmax. By the same token, the threshold value to representing the limit of image-pickup intervals of a continuous image-pickup process carried out at a highest speed is also a value determined in accordance with the performance of the
digital camera I. It is assumed that the control circuit 18 has recognized the value of l/Mmax and the threshold value to- The proper exposure time Tp for finding the threshold value Tp/Mmax has been found at the step S505 of the flowchart shown in FIG. 44.
After the process of the step S571 is completed, the flow of the second determination processing goes on to the next step S572 at which the control circuit 18 produces a result of determination as to whether or not the exposure time Ttmp is equal to or shorter than the threshold value Tbiur.
An exposure time Ttmp not exceeding the threshold value Tbiur satisfies the first condition applied to the exposure time Ttmp. On the other hand, an exposure time Ttmp exceeding the threshold value Tblur does not satisfy the first condition applied to the exposure time Ttmp-
With the exposure time Ttmp set at the step S571 at the larger one of two threshold values, i.e., the threshold value Tp/Mmax and the threshold value to/9, the process carried out at the step S572 following the step S571 to produce a result of determination as to whether or not the exposure time Ttmp satisfies the first condition can be regarded as a process said to be equivalent to a process to produce a result of
determination as to whether or not the exposure time Ttmp satisfying the first condition also satisfies the second and third conditions as well as is the case with the second determination processing represented by the flowchart shown in FIG. 47.
If the determination result produced at the step S572 indicates that the exposure time Ttmp is equal to or shorter .than the threshold value Tblur? the flow of the fourth determination processing goes on to a step S573 at which the control circuit 18 computes a taken-image count Nc of the fourth photographing processing by dividing the actual exposure time Ta by the exposure time Ttmp of one taken image. That is to say, the control circuit 18 computes the value of Nc as a quotient Ta/Ttmp- The taken-image count Nc represents the number of taken images obtained in a high-speed image-pickup process of the fourth photographing processing. In other words, the taken-image count Nc satisfying the relation Nc = Ta/Ttmp is determined by using the exposure time TtmP set at the step S571 and found at the step S572 to be an exposure time satisfying the first to third conditions. Thus, the exposure time Ttmp of each of the Nc taken images obtained at a high image-pickup speed in the fourth photographing processing can be expressed by a quotient Ta/Nc, which
satisfies the first to third conditions. That is to say, the taken-image count Nc has a value varying in accordance with the exposure time Ta in this way. It is to be noted that the taken-image count Nc is obtained as a result of a division operation by typically truncating the fraction part following the decimal point of the result.
After the process of the step S573 is completed, the flow of the second determination processing goes on to a step S574 at which the control circuit 18 produces a determination result of the fourth determination photographing as a result indicating that a proper output image can be obtained by execution of the fourth photographing processing. Finally, control of the processing execution is returned to the calling program.
If the determination result produced at the step S572 indicates that the exposure time Ttmp is longer than the threshold value Tbiur/ on the other hand, the flow of the fourth determination processing goes on to a step S575 at which the control circuit 18 produces a determination result of the fourth determination photographing as a result indicating that a proper output image cannot be obtained by execution of the fourth photographing processing. Finally, control of the processing execution is returned to the calling program.
At the step S506 of the flowchart shown in FIG. 44, on the basis of the result of one of the first, second, third, and the fourth determination processings, which are explained earlier by referring to the flowcharts shown in FIGS. 46 to 49 respectively, the control circuit 18 carries out the photographing-mode determination processing to make a decision as to which of the first, second, third, and the fourth photographing processings are to be carried out as photographing processings that can be performed to generate a sharp output image.
The first to third conditions are explained more as follows.
First of all, if the exposure time Ttmp (which can be Ta, Ta/Na, Ta/Nb or Ta/Nc) of one taken image satisfies
the first condition, that is, if the relation Ttmp = Tbiur holds true, even though each taken image obtained as a result of an image-pickup process carried out at a high speed becomes a dark image or an image of an under-brightness exposure, it is possible to obtain taken images each having no effects of hand trembling. As a result, a sharp output image can be generated from such taken images.
That is to say, if a taken image includes a blurring portion caused by hand trembling, an output
image generated from such a taken image also inevitably includes a blurring portion caused by hand trembling. If the first condition is satisfied, however, a taken image does not include a blurring portion caused by hand trembling. Thus, an output image generated from such a taken image also does not include a blurring portion caused by hand trembling.
In other words, by determining whether or not the exposure time Ttmp is equal to or shorter than the threshold value Tbiur serving as an upper limit of exposure times causing no effects of hand trembling, it is possible to select (or make a decision to carry out) photographing processing capable of obtaining a taken image without an effect of hand trembling. As described above, the exposure time Ttmp can be the actual exposure time Ta of a taken image obtained on the assumption that the first photographing processing is carried out, the actual exposure time Ta/Na of each of Na taken images obtained on the assumption that the second photographing processing is carried out, the actual exposure time Ta/Nb of each of Nb taken images obtained on the assumption that the third photographing processing is carried out or the actual exposure time Ta/Nc of each of Nc taken images obtained on the assumption that the fourth photographing
processing is carried out.
Then, if the exposure time Ttmp (which can be Ta/Na, Ta/Nb or Ta/Nc) of each taken image satisfies the second
condition, that is, if the relation Ttmp = Tp/Mmax holds true, even though each taken image obtained as a result of an image-pickup process carried out at a high speed becomes a dark image or an image of an under-brightness exposure, it is possible to obtain taken images, which are each not so dark that the image is buried under noises. As a result, a sharp output image can be generated from such taken images.
That is to say, if each taken image is extremely too dark, the pixel values of the taken image are unavoidably buried under noises so that an output image generated from such taken images also inevitably becomes an image with many noises. If the second condition is satisfied, however, each taken image becomes an image, which is not too dark so that a sharp output image can be generated from such taken images.
In other words, by determining whether or not the exposure time Ttmp is equal to or longer than the threshold value Tp/Mmax computed as a product resulting from multiplication of the proper exposure time Tp by xf it is possible to select (or make a decision to
carry out) photographing processing capable of obtaining a taken image, which is not too dark. The threshold value Tp/Mmax is a lower limit of exposure times not causing an image to be buried under noises. By definition, the threshold value Tp/Mmax is an exposure time generating an image with a brightness level equal to l/Mmax of the brightness level of an image taken at the proper exposure time Tp. As described above, the exposure time Ttmp can be the actual exposure time Ta/Na of each of Na taken images obtained on the assumption that the second photographing processing is carried out, the actual exposure time Ta/Nb of each of Nb taken images obtained on the assumption that the third photographing processing is carried out or the actual exposure time Ta/Nc of each of Nc taken images obtained on the assumption that the fourth photographing processing is carried out.
If the relation Ta/Na ^ t0, Ta/Nb ^ t0/4 or Ta/Nc ^ to/9 used as the third condition holds true, a blanking period can be prevented from resulting during an image-pickup process carried out at a high speed. Thus, a proper (or natural) output image can be generated.
That is to say, if the exposure time Ttmp of each taken image obtained in an image-pickup process carried out at a high speed exceeds the output minimum time, a
blanking period is generated. As described above, the exposure time Ttmp can be Ta/Na, Ta/Nb or Ta/Nc. The output minimum time is a shortest period of time during which the image-pickup device 4 is capable of outputting a taken image. A blanking period is a period between the end of an exposure for obtaining a specific taken image and the beginning of an exposure for obtaining a taken image immediately following the specific taken image. During a blanking period, there is no image-pickup process. To put it concretely, if the image-pickup device 4 does not execute a binning function, a blanking period of {t0-(Ta/Na)} seconds is generated. If the image-pickup device 4 executes the 2*2 binning function, a blanking period of {to/4-(Ta/Nb) } seconds is generated. If the image-pickup device 4 executes the 3x3 binning function, a blanking period of {t0/9-(Ta/Nc) } seconds is generated. If the object of the photographing is moving, the movement made by the object of the photographing during a blanking period is not reflected on the taken image at all. Thus, if an output image is generated from such a taken image, the output image inevitably becomes an image with discontinuities. By enforcing the third condition, however, a proper output image displaying continuities can be generated.
That is to say, by verifying that the exposure time Ta/Na is at least equal to t0, verifying that the exposure time Ta/Nb is at least equal to to/4 or verifying that the exposure time Ta/Nc is at least equal to to/9, it is possible to select (or make a decision to carry out) photographing processing, that generates no blanking period in an image-pickup process carried out at a high speed and is capable of generating a proper output image. As described earlier, the exposure time Ta/Na is the exposure time of each of Na taken images obtained on the assumption that the second photographing processing is carried out. Likewise, the actual exposure time Ta/Nb is the exposure time of each of Nb taken images obtained on the assumption that the third photographing processing is carried out. Similarly, the actual exposure time Ta/Nc is the exposure time of each of Nc taken images obtained on the assumption that the fourth photographing processing is carried out. On the other hand, notation to denotes the continuous image-pickup interval limit of the image-pickup device 4.
Effects of a blanking period on an output image are explained by referring to FIG. 50 as follows.
FIG. 50 is a diagram showing a case in which an output image is generated from four taken images each
obtained as a result of an image-pickup process carried out at a high speed at an exposure time Ttmp (= Ta/4) as a process to take a picture of a photographing object moving at a constant velocity V in the vertical direction (or the downward direction). It is to be noted that time 0 denotes the start time of the image-pickup process carried out at a high speed and position 0 denotes the position of the photographing object at time 0. In this case, the position of the photographing object at time t can be expressed by the expression t><5) XV to position (3xTa/4+2x5)*V. On the taken image 501D, the image of the moving photographing object is taken in the form of a movement-blurring portion in a range from position (3*Ta/4 + 3x5) xv to position (Ta+3*5)xV.
On the output image 501E generated from these taken images 501A to 501D, the image of the moving photographing object is taken in the form of a discontinuous movement-blurring portion in a range from position 0 to position (Ta+3x5)xV. The output image 501E generated in the form of an image including a discontinuous movement-blurring portion as described above is unnatural and such an unnatural image arouses a feeling of incompatibility in the photographer.
As described above, if a blanking period is generated, the output image becomes unnatural. In order to solve this problem, this embodiment selects (or makes a decision to carry out) photographing processing that satisfies the third condition so that a blanking period is not generated. In this way, the embodiment is capable of generating a proper (or natural) output image. It is to be noted that, if no moving photographing object exists, no movement-blurring results. Thus, it is not necessary to determine whether or not the third condition is satisfied. It is possible to determine whether or not a moving photographing object exists by for example taking an image of the photographing object by using the image-pickup device 4 and displaying the image on the monitor 11 as a through face, which is then subjected to a movement detection process to recognize any movement on the face.
As describe above, the diaphragm value F is fixed and an actual exposure time Ta is found from the proper exposure time Tp on the basis of an exposure correction value set by the exposure correction dial employed in the input device 16. It is to be noted, however, that the actual exposure Ta can also be fixed at the proper exposure time Tp and it is the diaphragm value F that be
adjusted in accordance with an exposure correction value set by the exposure correction dial.
That is to say, in accordance with what is described above, in general, the exposure is corrected by adoption of a method known as a diaphragm prioritizing mode in which the exposure is corrected by keeping the diaphragm value F at a constant value and varying the actual exposure time Ta (or the shutter speed). However, the exposure can also be corrected by for example adoption of a method referred to as a shutter-speed prioritizing mode. In the shutter-speed prioritizing mode, the actual exposure time Ta (or the shutter speed) is fixed at the proper exposure time Tp and not changed and, when the exposure correction dial is operated to set an exposure correction value, the diaphragm value F is adjusted in accordance with the set exposure correction value.
At a glance, the shutter-speed prioritizing mode that keeps the actual exposure time Ta at the proper exposure time Tp as described above is perceived as a mode unsuitable for the digital camera 1 shown in FIG. 1 as a camera in which the actual exposure time Ta is set at a value different from the proper exposure time Tp. As described below, however, the shutter-speed prioritizing
mode can be applied to the digital camera 1 shown in FIG. 1.
That is to say, in the shutter-speed prioritizing mode, let us assume for example that the diaphragm value found from a value measured by the exposure meter 19 as the value of the brightness of the photographing object is F' and the proper exposure time is Tp' .
At the photographing object F' and the proper exposure time is Tp' , for a first case in which the exposure correction value set by the exposure correction dial is 0, at the step S505 of the flowchart shown in FIG, 44 as a flowchart representing the photographing-mode determination processing, the diaphragm value F, the proper exposure time Tp and the actual exposure time Ta are determined at F', Tp' and Tp' respectively.
For a second case in which the exposure correction value set by the exposure correction dial is the one-stage under brightness, at the step S505 of the flowchart shown in FIG. 44, the diaphragm value F, the proper exposure time Tp and the actual exposure time Ta are determined at a diaphragm value lower than F' by one stage, 2TP' and Tp' respectively.
That is to say, the amount of light hitting the image-pickup device 4 in an image-pickup process with the
diaphragm value F set at F' and the proper exposure time Tp set at Tp' is equal to the amount of light hitting the image-pickup device 4 in an image-pickup process with the diaphragm value F set at a diaphragm value lower than F' by one stage and the proper exposure time Tp set at two times the exposure time Tp' . Now, as a one-stage under process of the image-pickup process with the diaphragm value F set at F' and the proper exposure time Tp set at Tp', let us consider a one-stage under process of the image-pickup process with the diaphragm value F set at a diaphragm value lower than F' by one stage and the proper exposure time Tp set at two times the exposure time Tp'. In this case, the one-stage under process of the image-pickup process with the diaphragm value F set at a diaphragm value lower than F' by one stage and the proper exposure time Tp set at two times the exposure time Tp' is an image-pickup process with the diaphragm value F set at a diaphragm value lower than F' by one stage and the actual exposure time Ta set at (1/2) (Tp) , which is equal to Tp' .
The one-stage under process of the image-pickup process with the diaphragm value F set at F' and the proper exposure time Tp set at Tp' is equivalent to the image-pickup process with the diaphragm value F set at a
value lower than F' by one stage and the actual exposure time Ta set Tp' . That is to say, the one-stage under process of the image-pickup process with the diaphragm value F set at F' and the proper exposure time Tp set at Tp' is equivalent to a one-stage under process of the image-pickup process with the diaphragm value F set at a value lower than F' by one stage and the proper exposure time Tp set at two times the exposure time Tp' . The proper exposure time Tp for such a process is 2TP' . An image-pickup process with the diaphragm value F set at a diaphragm value lower than F' by one stage and the actual proper exposure time Ta set at the exposure time Tp' is a one-stage under process carried out in the shutter-speed prioritizing mode for an image-pickup process with the diaphragm value F set at F' and the proper exposure time Tp set at Tp' .
Thus, when the exposure correction value set by the exposure correction dial represents the one-stage under brightness, in the shutter-speed prioritizing mode, an image-pickup process with the diaphragm value F set at a diaphragm value lower than F' by one stage and the actual proper exposure time Ta set at the exposure time Tp' is carried out. In the photographing-mode determination processing, the processing is carried out by setting the
diaphragm value F, the actual exposure time Ta and the proper exposure time Tp at a value lower than F' by one stage, a value fixed at Tp' and 2TP' respectively.
For a second case in which the exposure correction value set by the exposure correction dial is the two-stage under brightness, at the step S505 of the flowchart shown in FIG. 44, the diaphragm value F, the proper exposure time Tp and the actual exposure time Ta are determined at a diaphragm value lower than F' by two stages, 4TP' and a Tp' respectively.
That is to say, the amount of light hitting the image-pickup device 4 in an image-pickup process with the diaphragm value F set at F' and the proper exposure time Tp set at Tp' is egual to the amount of light hitting the image-pickup device 4 in an image-pickup process with the diaphragm value F set at a diaphragm value lower than F' by two stages and the proper exposure time Tp set at four times the exposure time Tp' . Now, as a two-stage under process of the image-pickup process with the diaphragm value F set at F' and the proper exposure time Tp set at Tp', let us consider a two-stage under process of the image-pickup process with the diaphragm value F set at a diaphragm value lower than F' by one stage and the proper exposure time Tp set at four times the exposure time Tp' .
In this case, the two-stage under process of the image-pickup process with the diaphragm value F set at a diaphragm value lower than F' by two stages and the proper exposure time Tp set at four times the exposure time Tp' is an image-pickup process with the diaphragm value F set at a diaphragm value lower than F' by two stages and the actual exposure time Ta set at (1/4) (Tp) , which is equal to Tp' .
The two-stage under process of the image-pickup process with the diaphragm value F set at F' and the proper exposure time Tp set at Tp' is equivalent to the image-pickup process with the diaphragm value F set at a value lower than F' by two stages and the actual exposure time Ta set at Tp' . That is to say, the two-stage under process of the image-pickup process with the diaphragm value F set at F' and the proper exposure time Tp set at Tp' is equivalent to a two-stage under process of the image-pickup process with the diaphragm value F set at a value lower than F' by two stages and the proper exposure time Tp set at four times the exposure time Tp' . The proper exposure time Tp for such a process is 4TP' . An image-pickup process with the diaphragm value F set at a diaphragm value lower than F' by two stages and the actual proper exposure time Ta set at the exposure time
Tp' is a two-stage under process carried out in the shutter-speed prioritizing mode for an image-pickup process with the diaphragm value F set at F' and the proper exposure time Tp set at Tp' .
Thus, when the exposure correction value set by the exposure correction dial represents the two-stage under brightness, in the shutter-speed prioritizing mode, an image-pickup process with the diaphragm value F set at a diaphragm value lower than F' by two stages and the actual proper exposure time Ta set at the exposure time Tp' is carried out. In the photographing-mode determination processing, the processing is carried out by setting the diaphragm value F, the actual exposure time Ta and the proper exposure time Tp at a value lower than F' by two stages, a value fixed at Tp' and 4TP' respectively.
As for the three-stage under brightness or the one-stage over brightness, the two-stage over brightness and the three-stage over brightness, in the shutter-speed prioritizing mode, the photographing-mode determination processing is carried out in the same way as the one-stage under brightness and the two-stage under brightness, which have been described above.
It is to be noted that, when a binning function is
executed, a pixel value output by the image-pickup device 4 increases to a value larger than a pixel value output without execution of a binning function due to summation of a plurality of pixel values, and larger pixel values cause a brighter image. Thus, the limit l/Mmax of darkness levels that do not cause an image to be inevitably buried under noises can be reduced, that is, the value of Mmax can be increased. That is to say, when a binning function is executed, the threshold value Tp/Mmax of the second condition can be set at a value smaller than the value for an image-pickup process without execution of a binning function. This is because a taken image is hardly buried under noises even if an image is taken as an image on the side darker than an image taken without execution of the binning function.
In the processing represented by the flowchart shown in FIG. 44, at the step S505, the proper exposure time Tp is determined and, then, the proper exposure time Tp is used in the following processes. However, the process of the step S505 does not have to be carried out. In this case, the proper exposure time Tp determined at the step S502 is used in the processes.
In addition, in accordance with this embodiment, the digital camera 1 shown in FIG. 1 is provided with a
release button having a half-press function. However, the digital camera 1 can also be a digital camera without a release button having a half-press function. In this case, first of all, a process is carried out in order to determine whether or not the release button has been completely pressed. Then, the diaphragm value F, the proper exposure time Tp and the actual exposure time Ta are determined in a process similar to the process carried out at the step S502. Subsequently, the photographing-mode determination processing of the step S506 is carried out.
In the example described above, the image-pickup device 4 employed in the image-pickup device 4 is a single-plate sensor. From a chrominance signal output by each pixel of the image-pickup device 4, an output image having three chrominance signals per pixel is generated. It is to be noted, however, that the image-pickup device 4 does not have to be a single-plate sensor. For example, it is also possible to employ a triple-plate image-pickup device for outputting three' colors, i.e., the R, G and B colors, per pixel.
In addition, the present invention can be adopted in not only a digital still camera as described above, but also other apparatus such as a digital video camera
by increasing the processing speed.
Processes of steps composing each of the flowcharts described above can of course be carried out along the time axis in accordance with an order in which the steps are described. It is to be noted, however, that the processes do not have to be carried out along the time axis. For example, the processes can also be carried out concurrently and individually. In addition, a portion or all of each flowchart can be carried out by execution of a program in a computer (or the CPU 15) or dedicated hardware.
Finally, the range of claims of the present invention is described. The present invention relates to the photographing-mode determination processing carried out at the step S506 of the flowchart shown in FIG. 44. As is obvious from the flowchart shown in FIG. 44, the photographing-mode determination processing of the step S506 can be carried out if the proper exposure time Tp, the actual exposure time Ta and the information on a focal distance have been obtained prior to the step S506. As described before, the information on a focal distance is information for finding the threshold value Tbiur serving as the limit of exposure times causing no effects of hand trembling.
There are many conventional exposure determination methods, which are methods for obtaining the proper exposure time Tp and the actual exposure time Ta as well as obtaining the information on a focal distance as information for finding the threshold value Tbiur serving as the limit of exposure times causing no effects of hand trembling. Any of these methods can be adopted for obtaining the proper exposure time Tp, the actual exposure time Ta and the information on a focal distance. That is to say, a method for obtaining the proper exposure time Tp, the actual exposure time Ta and the information on a focal distance is not specified in particular.
As described above, the present invention relates to the photographing-mode determination processing for automatically determining which photographing processing is to be carried out to photograph an object of photographing. The photographing processing can be the first photographing processing carried out in the ordinary photographing mode in the same way as the photographing processing carried out by the ordinary digital camera. The photographing processing can also be the second, third or fourth photographing processing carried out in the hand-trembling correction photographing mode. In the second, third or fourth
photographing processing, a continuous image-pickup process is carried out at a high speed to obtain a plurality of taken images successively. Each of the taken images obtained in this way is darker than an image taken at the proper exposure time. Then, a sharp output image is generated by for example superposing or interpolating the taken images.
That is to say, the first processing carried out at the step S521 of the flowchart shown in FIG. 45 is a process of importance to the present invention. To be more specific, the first determination processing is important processing to produce a result of determination as to whether or not it is quite within the bounds of possibility that a burring portion is produced by hand trembling on an output image generated on the assumption that a decision has been made to perform the first photographing processing as the photographing processing in the ordinary photographing mode. This first determination processing leads to selection of (a decision to select) the first photographing processing to be carried out as photographing processing in the ordinary photographing mode or the second, third or fourth photographing processing to be carried out as photographing processing in the hand-trembling correction
photographing mode. As described above, in the second, third or fourth photographing processing, a continuous image-pickup process is carried out at a high speed to obtain a plurality of taken images successively. Each of the taken images obtained in this way is darker than an image taken at the proper exposure time. Then, a sharp output image is generated by for example superposing or interpolating the taken images. This point is a point key to the present invention.
Thus, a method (or processing) to generate a sharp output image from a plurality of taken images obtained by carrying out an continuous image-pickup process at a high speed as images each darker than an image taken at the proper exposure time is not limited to the image generation processing described above. Instead, it is possible to adopt any other method including any of the conventional methods.
[Name of Document] Claims[Claim 1]A control method provided for a photographingapparatus as a control method having a first mode forgenerating an output image by taking one input image and a second mode forgenerating an output image by taking a plurality ofinput images successively wherebyat least one of a first exposure time, which is an exposure time of a photographing operation to take an input image on the assumption that said photographing operation is to be carried out in said first mode, and a second exposure time, which is an exposure time of a photographing operation to take a plurality of input images on the assumption that said photographing operation is to be carried out in said second mode, is compared with a predetermined threshold valuein order to produce a result of determination as to whether to carry out a photographing operation in said first mode selected as a photographing mode or carry out a photographing operation in said second mode selected as said photographing mode. [Claim 2]A control method provided for a photographing apparatus in accordance with claim 1, said control methodcomprising:a first determination step of producing a result of determination as to whether or not said first exposure time is equal to or shorter than a first threshold value; a first decision step of making a decision to take an image in said first mode if said determination result produced at said first determination step indicates that said first exposure time is equal to or shorter than said first threshold value; anda second decision step of making a decision to take an image in said second mode if said determination result produced at said first determination step indicates that said first exposure time is neither equal to nor shorter than said first threshold value. [Claim 3]A control method provided for a photographing apparatus in accordance with claim 2 wherein said first threshold value is a limit of exposure times not causing effects of hand trembling on said input image taken in said first mode. [Claim 4]A control method provided for a photographing apparatus in accordance with claim 2 wherein said first threshold value is a value based on a focal distance usedat an image-pickup time to take said input image. [Claim 5]A control method provided for a photographing apparatus in accordance with claim 2 wherein:there is further included a second determination step of producing a result of determination as to whether or not said second exposure time is equal to or shorter than a second threshold value if said determination result produced at said first determination step indicates that said first exposure time is neither equal to nor shorter than said first threshold value; andsaid second decision step is executed to make a decision to take an image in said second mode if said determination result produced at said second determination step indicates that said second exposure time is equal to or shorter than said second threshold value. [Claim 6]A control method provided for a photographing apparatus in accordance with claim 5 wherein said second threshold value is a limit of exposure times not causing effects of hand trembling on said input image taken in said second mode. [Claim 7]A control method provided for a photographing apparatus in accordance with claim 5 wherein said second threshold value is a value based on a focal distance used at an image-pickup time to take said input image. [Claim 8]A control method provided for a photographing apparatus in accordance with claim 5, said control method further includinga third decision step of making a decision to take an image in said first mode if said determination result produced at said first determination step indicates that said first exposure time is neither equal to nor shorter than said first threshold value and said determination result produced at said second determination step indicates that said second exposure time is neither equal to nor shorter than said second threshold value. [Claim 9]A control method provided for a photographing apparatus in accordance with claim 5 wherein:said second determination step further includes a step of producing a result of determination as to whether or not said second exposure time is equal to or longer than the larger one of a threshold value based on a proper exposure time found from the brightness of a
photographing object and a threshold value based on an image-pickup interval of successive image-pickup operations carried out by said photographing apparatus at a highest speed; andsaid second decision step is a step of making a decision to take an image in said second mode if said determination result produced at said second determination step indicates that said second exposure time is equal to or shorter than said second threshold value and that said second exposure time is equal to or longer than said larger one of said threshold value based on a proper exposure time found from the brightness of said photographing object and said threshold value based on said image-pickup interval of successive image-pickup operations carried out by said photographing apparatus at a highest speed. [Claim 10]A control method provided for a photographing apparatus in accordance with claim 5 wherebyat said second determination step, the larger one of a threshold value based on a proper exposure time found from the brightness of a photographing object and a threshold value based on an image-pickup interval of successive image-pickup operations carried out by said
photographing apparatus at a highest speed is used as said second threshold value, and said second exposure time is compared with said threshold value in order to produce a result of determination as to whether or not said second exposure time is equal to or smaller than said threshold value. [Claim 11]A control method provided for a photographing apparatus in accordance with claim 1 whereinsaid photographing apparatus has an image-pickup means for taking said input image and in said second mode:positional relations among positions of a plurality of said input images taken by said image-pickup means are detected;pixels of said input images are identified on the basis of said detected positional relations as pixels to be used for inference of a pixel value at the position of every pixel on said output image; andsaid output image is generated by inference of a pixel value at the position of every pixel on said output image on the basis of pixel values of said identified pixels on said input images. [Claim 12]A control method provided for a photographing apparatus in accordance with claim 11 whereby, in said second mode, said photographing apparatus detects positional relations among positions of a plurality of said input images by using one of said input images as a reference wherein said input image used as a reference is an input image taken at a middle time in the middle of times at which said input images are taken or an input image taken at a time close to said middle time. [Claim 13]A control method provided for a photographing apparatus in accordance with claim 11 whereby, in said second mode, as said output image, said photographing apparatus generates an image, which is observed at a central portion included in a range of a photographing object projected on said input images as a portion excluding peripheral portions of said range, and has a pixel pitch smaller than the pixel pitch of each of said input images. [Claim 14]A control method provided for a photographing apparatus in accordance with claim 13 whereby, in said second mode, an output image is generated as an image having a pixel count equal to the pixel count of saidimage-pickup means. [Claim 15]A control method provided for a photographing apparatus in accordance with claim 11 whereby,in said second mode, said image-pickup means employed in said photographing apparatus treats a plurality of pixels owned by said image-pickup means as a group in order to output an output image having a pixel count smaller than the number of pixels owned by said image-pickup means. [Claim 16]A control method provided for a photographing apparatus in accordance with claim 11 whereby, in said second mode, said photographing apparatus infers three pixel values for the position of every pixel on said output image having three pixel values per pixel on the basis of pixel values of pixels on a plurality of input images each having one pixel value per pixel. [Claim 17]A control method provided for a photographing apparatus in accordance with claim 1, said control method comprising:a determination step of producing a result of determination as to whether or not said second exposuretime is equal to or longer than a threshold value based on a proper exposure time found from the brightness of a photographing object;a first decision step of making a decision to take an image in said second mode if said determination result produced at said determination step indicates that said second exposure time is equal to or longer than said threshold value; anda second decision step of making a decision to take an image in said first mode if said determination result produced at said determination step indicates that said second exposure time is neither equal to nor longer than said threshold value. [Claim 18]A control method provided for a photographing apparatus in accordance with claim 1, said control method comprising:a determination step of producing a result of determination as to whether or not said second exposure time is equal to or longer than a threshold value based on an image-pickup interval of successive image-pickup operations carried out by said photographing apparatus at a highest speed;a first decision step of making a decision to takean image in said second mode if said determination result produced at said determination step indicates that said second exposure time is equal to or longer than said threshold value; anda second decision step of making a decision to take an image in said first mode if said determination result produced at said determination step indicates that said second exposure time is neither equal to nor longer than said threshold value. [Claim 19]A control apparatus provided for a photographing apparatus having a first mode for generating an output image by taking one input image and a second mode for generating an output image by taking a plurality of input images successively whereinat least one of a first exposure time, which is an exposure time of a photographing operation to take an input image on the assumption that said photographing operation is to be carried out in said first mode, and a second exposure time, which is an exposure time of a photographing operation to take a plurality of input images on the assumption that said photographing operation is to be carried out in said second mode, is compared with a predetermined threshold valuein order to produce a result of determination as to whether to carry out a photographing operation in said first mode selected as a photographing mode or carry out a photographing operation in said second mode selected as said photographing mode. [Claim 20]A control program to be executed by a computer as a program for controlling a photographing apparatus having a first mode for generating an output image by taking one input image and a second mode for generating an output image by taking a plurality of input images successively wherebyat least one of a first exposure time, which is an exposure time of a photographing operation to take an input image on the assumption that said photographing operation is to be carried out in said first mode, and a second exposure time, which is an exposure time of a photographing operation to take a plurality of input images on the assumption that said photographing operation is to be carried out in said second mode, is compared with a predetermined threshold valuein order to produce a result of determination as to whether to carry out a hotographing operation in said first mode selected as a photographing mode or carry outa photographing operation in said second mode selected as said photographing mode.