FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION
(See section 10, rule 13)
“IMAGING DEVICE AND METHOD, AND IMAGE PROCESSING
DEVICE AND METHOD”
SONY CORPORATION, of 1-7-1, Konan, Minato-ku,
Tokyo 108-0075, Japan
The following specification particularly describes the invention and the manner in which it is to
be performed.
2
DESCRIPTION
IMAGING APPARATUS AND METHOD, AND IMAGE PROCESSING
APPARATUS AND METHOD
5 TECHNICAL FIELD
[0001]
The present technology relates to an imaging
apparatus and method, and an image processing apparatus
and method, and more particularly, to an imaging
10 apparatus and method, and an image processing apparatus
and method enabled to control a resolution of a detection
image.
BACKGROUND ART
15 [0002]
Conventionally, an imaging element is generally
used in combination with an imaging lens that focuses
light on the imaging element. The imaging lens guides
the light from a subject surface to each pixel of the
20 imaging element to reproduce a light intensity
distribution of the subject surface, whereby the imaging
element can obtain a detection signal of a level
corresponding to the light intensity distribution in each
pixel, and can obtain a captured image of the subject as
25 a whole.
[0003]
However, in this case, the physical size becomes
large. Thus, an imaging element has been devised that
does not use an imaging lens (for example, see Patent
30 Document 1, Patent Document 2, and Non-Patent Document 1).
3
CITATION LIST
PATENT DOCUMENT
[0004]
Patent Document 1: International Publication No.
5 2016/123529
Patent Document 2: PCT Japanese Translation Patent
Publication No. 2016-510910
NON-PATENT DOCUMENT
[0005]
10 Non-Patent Document 1: M. Salman Asif and four others,
“Flatcam: Replacing lenses with masks and computation”,
“2015 IEEE International Conference on Computer Vision
Workshop (ICCVW)”, 2015, pages 663-666
15 SUMMARY OF THE INVENTION
PROBLEMS TO BE SOLVED BY THE INVENTION
[0006]
However, with the method described in Patent
Document 1, only a detection image including information
20 obtained from all pixels of the imaging element can be
obtained, and it has not been possible to obtain a
detection image having a desired resolution.
[0007]
The present disclosure has been made in view of
25 such a situation, and makes it possible to control a
resolution of a detection image.
SOLUTIONS TO PROBLEMS
[0008]
30 An imaging apparatus according to one aspect of the
present technology is an imaging apparatus including: an
4
imaging element including a plurality of pixel output
units that receives incident light entering without
passing through either an imaging lens or a pinhole, and
each outputs one detection signal indicating an output
pixel value modulated by an incident 5 angle of the
incident light; and a read control unit that selectively
reads the output pixel value of each of the pixel output
units of the imaging element.
[0009]
10 The read control unit can select some pixel unit
outputs among the plurality of pixel output units of the
imaging element, and read output pixel values of the
pixel output units selected.
[0010]
15 The read control unit can select some pixel output
units at arbitrary positions among the plurality of pixel
output units of the imaging element.
[0011]
The read control unit can select the pixel output
20 units such that, regarding an incident angle directivity
of the output pixel value indicating a directivity with
respect to an incident angle of incident light from a
subject, a whole of the pixel output units selected has
the incident angle directivity equivalent to an incident
25 angle directivity of all pixel output units of the
imaging element.
[0012]
The read control unit can select some pixel output
units in a positional relationship having a predetermined
30 regularity among the plurality of pixel output units of
the imaging element.
5
[0013]
Regarding an incident angle directivity of the
output pixel value indicating a directivity with respect
to an incident angle of incident light from a subject, a
whole of the some pixel output units 5 of the imaging
element in the positional relationship having the
regularity selected by the read control unit can be made
to have the incident angle directivity equivalent to an
incident angle directivity of all pixel output units of
10 the imaging element.
[0014]
The read control unit can select a pixel output
unit formed in one partial region of a region in which
the plurality of pixel output units of the imaging
15 element is formed.
[0015]
Regarding an incident angle directivity of the
output pixel value indicating a directivity with respect
to an incident angle of incident light from a subject, a
20 whole of pixel output units of the imaging element formed
in the partial region selected by the read control unit
can be made to have the incident angle directivity
equivalent to an incident angle directivity of all pixel
output units of the imaging element.
25 [0016]
The read control unit can read the output pixel
values from all pixel output units of the imaging element,
and select some of the output pixel values read.
[0017]
30 The read control unit can read output pixel values
of all pixel output units of the imaging element, and add
6
the read output pixel values together for each
predetermined number.
[0018]
The read control unit can add together output pixel
values of pixel output units, the output 5 pixel values
having mutually similar incident angle directivities each
indicating a directivity with respect to an incident
angle of incident light from a subject.
[0019]
10 The read control unit can add together output pixel
values of pixel output units close to each other.
[0020]
The plurality of pixel output units can have a
configuration in which an incident angle directivity of
15 the output pixel value indicating a directivity with
respect to an incident angle of incident light from a
subject is settable independently for each of the pixel
output units.
[0021]
20 The plurality of pixel output units can be made to
have a configuration in which an incident angle
directivity indicating a directivity with respect to an
incident angle of incident light from a subject is
settable independently for each of the pixel output units.
25 [0022]
The plurality of pixel output units can be made to
have a configuration in which an incident angle
directivity of the output pixel value indicating a
directivity with respect to an incident angle of incident
30 light from a subject is settable independently for each
of the pixel output units by making photo diodes (PDs)
7
that contribute to output different from each other.
[0023]
An imaging method according to one aspect of the
present technology is an imaging method including:
imaging a subject by an imaging element 5 including a
plurality of pixel output units that receives incident
light entering without passing through either an imaging
lens or a pinhole, and each outputs one detection signal
indicating an output pixel value modulated by an incident
10 angle of the incident light; and selectively reading the
output pixel value of each of the pixel output units of
the imaging element.
[0024]
An image processing apparatus according to another
15 aspect of the present technology is an image processing
apparatus including: a resolution setting unit that sets
a resolution; and a restoration matrix setting unit that
sets a restoration matrix including coefficients used
when a restored image is restored from output pixel
20 values of a plurality of pixel output units, of an
imaging element including the plurality of pixel output
units that receives incident light entering without
passing through either an imaging lens or a pinhole, and
each outputs one detection signal indicating an output
25 pixel value modulated by an incident angle of the
incident light, depending on the resolution set by the
resolution setting unit.
[0025]
The resolution setting unit can set the resolution
30 by selecting output pixel values of some of the pixel
output units.
8
[0026]
The resolution setting unit can set the resolution
by adding the output pixel values of the pixel output
units together for each predetermined number.
5 [0027]
An image processing method according to the other
aspect of the present technology is an image processing
method including: setting a resolution; and setting a
restoration matrix including coefficients used when a
10 restored image is restored from output pixel values of a
plurality of pixel output units, of an imaging element
including the plurality of pixel output units that
receives incident light entering without passing through
either an imaging lens or a pinhole, and each outputs one
15 detection signal indicating an output pixel value
modulated by an incident angle of the incident light,
depending on the resolution set.
[0028]
In the imaging apparatus and method according to
20 one aspect of the present technology, a subject is imaged
by an imaging element including a plurality of pixel
output units that receives incident light entering
without passing through either an imaging lens or a
pinhole, and each outputs one detection signal indicating
25 an output pixel value modulated by an incident angle of
the incident light, and an output pixel value of each of
the pixel output units of the imaging element is
selectively read.
[0029]
30 In the image processing apparatus and method
according to the other aspect of the present technology,
9
a resolution is set, and a restoration matrix is set
including coefficients used when a restored image is
restored from output pixel values of a plurality of pixel
output units, of an imaging element including the
plurality of pixel output units that 5 receives incident
light entering without passing through either an imaging
lens or a pinhole, and each outputs one detection signal
indicating an output pixel value modulated by an incident
angle of the incident light, depending on the resolution
10 set.
EFFECTS OF THE INVENTION
[0030]
According to the present technology, a subject can
15 be imaged, or an image can be processed. Furthermore,
according to the present technology, a resolution of a
detection image can be controlled.
BRIEF DESCRIPTION OF DRAWINGS
20 [0031]
Fig. 1 is a block diagram illustrating a main
configuration example of an imaging apparatus.
Fig. 2 is a diagram illustrating the principle of
imaging in an imaging apparatus to which the technology
25 according to the present disclosure is applied.
Fig. 3 is a diagram illustrating a difference in
configuration between a conventional imaging element and
an imaging element according to the present disclosure.
Fig. 4 is a diagram illustrating a first
30 configuration example of the imaging element.
Fig. 5 is a diagram illustrating the first
10
configuration example of the imaging element.
Fig. 6 is a diagram illustrating the principle of
generation of incident angle directivity.
Fig. 7 is a diagram illustrating a change in
incident angle directivity using 5 an on-chip lens.
Fig. 8 is a diagram illustrating design of incident
angle directivity.
Fig. 9 is a diagram illustrating a relationship
between a subject distance and coefficients expressing
10 incident angle directivity.
Fig. 10 is a diagram illustrating a relationship
between a narrow angle-of-view pixel and a wide angle-ofview
pixel.
Fig. 11 is a diagram illustrating the relationship
15 between the narrow angle-of-view pixel and the wide
angle-of-view pixel.
Fig. 12 is a diagram illustrating the relationship
between the narrow angle-of-view pixel and the wide
angle-of-view pixel.
20 Fig. 13 is a diagram illustrating a modification.
Fig. 14 is a diagram illustrating a modification.
Fig. 15 is a diagram illustrating a modification.
Fig. 16 is a diagram illustrating an example in
which an angle of view is changed by applying the
25 modification.
Fig. 17 is a diagram illustrating an example of
combining pixels having a respective plurality of angles
of view when the angle of view is changed by applying the
modification.
30 Fig. 18 is a diagram illustrating a modification.
Fig. 19 is a diagram illustrating a modification.
11
Fig. 20 is a diagram illustrating a modification.
Fig. 21 is a diagram illustrating an example of a
mask pattern by a light-shielding film.
Fig. 22 is a diagram for explaining an outline of a
method of reading 5 from all pixels.
Fig. 23 is a diagram for explaining an outline of a
method of reading from some arbitrary pixels.
Fig. 24 is a diagram for explaining an outline of a
method of regularly reading from some pixels.
10 Fig. 25 is a diagram for explaining an outline of a
method of reading from pixels in a partial region.
Fig. 26 is a diagram for explaining an outline of a
method of performing pixel addition.
Fig. 27 is a diagram for comparing and explaining
15 each method.
Fig. 28 is a diagram for explaining a method of
making directivities equivalent to each other.
Fig. 29 is a diagram for explaining a restoration
matrix.
20 Fig. 30 is a diagram for explaining a restoration
matrix of the method of reading from all pixels.
Fig. 31 is a diagram for explaining the restoration
matrix of the method of reading from all pixels.
Fig. 32 is a diagram for explaining a restoration
25 matrix of the method of reading from some arbitrary
pixels.
Fig. 33 is a diagram for explaining the restoration
matrix of the method of reading from some arbitrary
pixels.
30 Fig. 34 is a diagram for explaining a restoration
matrix of the method of regularly reading from some
12
pixels.
Fig. 35 is a diagram for explaining the restoration
matrix of the method of regularly reading from some
pixels.
Fig. 36 is a diagram for explaining 5 a restoration
matrix of the method of reading from pixels in a partial
region.
Fig. 37 is a diagram for explaining the restoration
matrix of the method of reading from pixels in a partial
10 region.
Fig. 38 is a diagram for explaining a restoration
matrix of the method of performing pixel addition.
Fig. 39 is a diagram for explaining the restoration
matrix of the method of performing pixel addition.
15 Fig. 40 is a diagram illustrating a reason why the
amount of calculation and the memory capacity are reduced
by providing rules for a light-shielding range in each of
the horizontal direction and the vertical direction.
Fig. 41 is a diagram illustrating the reason why
20 the amount of calculation and the memory capacity are
reduced by providing rules for the light-shielding range
in each of the horizontal direction and the vertical
direction.
Fig. 42 is a diagram illustrating the reason why
25 the amount of calculation and the memory capacity are
reduced by providing rules for the light-shielding range
in each of the horizontal direction and the vertical
direction.
Fig. 43 is a diagram illustrating the reason why
30 the amount of calculation and the memory capacity are
reduced by providing rules for the light-shielding range
13
in each of the horizontal direction and the vertical
direction.
Fig. 44 is a flowchart illustrating an example of a
flow of imaging processing.
Fig. 45 is a diagram for explaining 5 an outline of a
reading method in the case of a color image.
Fig. 46 is a block diagram illustrating a main
configuration example of an image processing apparatus.
Fig. 47 is a flowchart illustrating an example of a
10 flow of image processing.
Fig. 48 is a diagram illustrating a main
configuration example of the imaging element.
Fig. 49 is a diagram illustrating a case where a
black-and-white pattern mask is used.
15 Fig. 50 is a diagram illustrating a case where an
optical interference mask is used.
Fig. 51 is a diagram illustrating a modification of
the imaging element.
20 MODE FOR CARRYING OUT THE INVENTION
[0032]
The following is a description of a mode for
carrying out the present disclosure (the mode will be
hereinafter referred to as the embodiment). Note that,
25 description will be made in the following order.
1. First embodiment (imaging apparatus)
2. Second embodiment (image processing apparatus)
3. Third embodiment (other configuration examples
of imaging element, imaging apparatus, and image
30 processing apparatus)
4. Others
14
[0033]
<1. First embodiment>

Fig. 1 is a diagram illustrating a main
configuration example of an imaging apparatus 5 that is an
embodiment of an imaging apparatus or an image processing
apparatus to which the present technology is applied. An
imaging apparatus 100 illustrated in Fig. 1 is an
apparatus that images a subject and obtains electronic
10 data regarding a captured image of a subject.
[0034]
As illustrated in Fig. 1, the imaging apparatus 100
includes a control unit 101, an input unit 111, an output
unit 112, a storage unit 113, a communication unit 114,
15 and a recording/reproducing unit 115. Furthermore, the
imaging apparatus 100 includes an imaging element 121, a
read control unit 122, a restoration matrix setting unit
123, a restoration unit 124, an associating unit 125, and
a sensor unit 126. The processing units and the like are
20 connected to each other via a bus 110, and can exchange
information, commands, and the like with each other.
[0035]
Note that, the imaging element 121 and the read
control unit 122 may be integrated together as an imaging
25 unit 120. The imaging unit 120 may be realized by any
physical configuration. For example, the imaging unit
120 may be realized as a processor as a system large
scale integration (LSI) or the like. Furthermore, the
imaging unit 120 may be realized as, for example, a
30 module using a plurality of processors, a unit using a
plurality of modules and the like, or a set obtained by
15
further adding other functions to a unit, and the like
(in other words, a partial configuration of the
apparatus). Furthermore, the imaging unit 120 may be
realized as an apparatus.
5 [0036]
The control unit 101 is configured to perform
processing related to control of the processing units and
the like in the imaging apparatus 100. For example, the
control unit 101 includes a central processing unit (CPU),
10 a read only memory (ROM), a random access memory (RAM),
and the like, and performs the above-described processing
by executing a program by using the CPU and the like.
[0037]
The input unit 111 is configured to perform
15 processing related to input of information. For example,
the input unit 111 includes input devices such as an
operation button, a dial, a switch, a touch panel, a
remote controller, and a sensor, and an external input
terminal. For example, the input unit 111 accepts an
20 instruction (information corresponding to input
operation) from the outside such as a user with these
input devices. Furthermore, for example, the input unit
111 acquires arbitrary information (program, command,
data, and the like) supplied from an external apparatus
25 via the external input terminal. Furthermore, for
example, the input unit 111 supplies the accepted
information (acquired information) to other processing
units and the like via the bus 110.
[0038]
30 Note that, the sensor included in the input unit
111 may be any sensor as long as it can accept the
16
instruction from the outside such as the user, for
example, an acceleration sensor or the like. Furthermore,
the input device included in the input unit 111 is
arbitrary, and the number of them is also arbitrary. The
input unit 111 may include a plurality 5 of types of input
devices. For example, the input unit 111 may include
some of the examples described above, or may include the
whole. Furthermore, the input unit 111 may include an
input device other than the examples described above.
10 Moreover, for example, the input unit 111 may acquire
control information regarding the input unit 111 (input
device or the like) supplied via the bus 110, and operate
on the basis of the control information.
[0039]
15 The output unit 112 is configured to perform
processing related to output of information. For example,
the output unit 112 includes an image display device such
as a monitor, an image projection device such as a
projector, a sound output device such as a speaker, an
20 external output terminal, and the like. For example, the
output unit 112 outputs information supplied from other
processing units and the like via the bus 110 by using
those output devices and the like. For example, the
output unit 112 displays a captured image (restored image
25 described later) on a monitor, projects a captured image
(restored image described later) from a projector,
outputs sound (for example, sound corresponding to an
input operation, a processing result, or the like), or
outputs arbitrary information (program, command, data,
30 and the like) to the outside (another device).
[0040]
17
Note that, the output device and the like included
in the output unit 112 are arbitrary, and the number of
them is also arbitrary. The output unit 112 may include
a plurality of types of output devices and the like. For
example, the output unit 112 may include 5 some of the
examples described above, or may include the whole.
Furthermore, the output unit 112 may include an output
device and the like other than the examples described
above. Moreover, for example, the output unit 112 may
10 acquire control information regarding the output unit 112
(output device or the like) supplied via the bus 110, and
operate on the basis of the control information.
[0041]
The storage unit 113 is configured to perform
15 processing related to storage of information. For
example, the storage unit 113 includes an arbitrary
storage medium such as a hard disk or a semiconductor
memory. For example, the storage unit 113 stores
information (program, command, data, and the like)
20 supplied from other processing units and the like via the
bus 110 in the storage medium. Furthermore, the storage
unit 113 may store arbitrary information (program,
command, data, and the like) at the time of shipment.
Furthermore, the storage unit 113 reads information
25 stored in the storage medium at an arbitrary timing or in
response to a request from other processing units and the
like, and supplies the read information to the other
processing units and the like via the bus 110.
[0042]
30 Note that, the storage medium included in the
storage unit 113 is arbitrary, and the number of them is
18
also arbitrary. The storage unit 113 may include a
plurality of types of storage media. For example, the
storage unit 113 may include some of the examples of the
storage medium described above, or may include the whole.
Furthermore, the storage unit 113 may 5 include a storage
medium and the like other than the examples described
above. Furthermore, for example, the storage unit 113
may acquire control information regarding the storage
unit 113 supplied via the bus 110, and operate on the
10 basis of the control information.
[0043]
The communication unit 114 is configured to perform
processing related to communication with other
apparatuses. For example, the communication unit 114
15 includes a communication device that performs
communication for exchanging information such as programs
and data with an external apparatus via a predetermined
communication medium (for example, an arbitrary network
such as the Internet). For example, the communication
20 unit 114 communicates with another apparatus, and
supplies information (program, command, data, and the
like) supplied from other processing units and the like
via the bus 110 to the other apparatus that is a
communication partner. Furthermore, for example, the
25 communication unit 114 communicates with another
apparatus, acquires information supplied from the other
apparatus that is a communication partner, and supplies
the information to the other processing units and the
like via the bus 110.
30 [0044]
The communication device included in the
19
communication unit 114 may be any communication device.
For example, the communication device may be a network
interface. A communication method and a communication
standard are arbitrary. For example, the communication
unit 114 may be made to perform wired 5 communication,
wireless communication, or both. Furthermore, for
example, the communication unit 114 may acquire control
information regarding the communication unit 114
(communication device or the like) supplied via the bus
10 110, and operate on the basis of the control information.
[0045]
The recording/reproducing unit 115 is configured to
perform processing related to recording and reproduction
of information using a recording medium 116 mounted to
15 the recording/reproducing unit 115. For example, the
recording/reproducing unit 115 reads information (program,
command, data, and the like) recorded on the recording
medium 116 mounted to the recording/reproducing unit 115,
and supplies the information to other processing units
20 and the like via the bus 110. Furthermore, for example,
the recording/reproducing unit 115 acquires information
supplied from the other processing units and the like via
the bus 110, and writes (records) the information in the
recording medium 116 mounted to the recording/reproducing
25 unit 115. Note that, for example, the
recording/reproducing unit 115 may acquire control
information regarding the recording/reproducing unit 115
supplied via the bus 110, and operate on the basis of the
control information.
30 [0046]
Note that, the recording medium 116 may be any
20
recording medium. For example, the recording medium may
be a magnetic disk, an optical disk, a magneto-optical
disk, a semiconductor memory, or the like.
[0047]
The imaging element 121 is configured 5 to perform
processing related to imaging of a subject. For example,
the imaging element 121 images the subject, and obtains
data (electronic data) regarding the captured image. At
that time, the imaging element 121 can image the subject
10 without using an imaging lens, an optical filter such as
a diffraction grating, and the like, or a pinhole or the
like, and obtain data regarding the captured image. For
example, the imaging element 121 images the subject and
obtains data (detection signals and the like) that makes
15 it possible to obtain data of the captured image by a
predetermined calculation.
[0048]
Note that, the captured image is an image that is
configured by values of pixels on which a subject image
20 is formed, and can be visually recognized by the user.
On the other hand, an image (referred to as a detection
image) configured by a detection signal that is a
detection result of incident light in the pixel unit
output of the imaging element 121 cannot be recognized as
25 an image even when viewed by the user (that is, the
subject cannot be visually recognized) since the subject
image is not formed. That is, the detection image is an
image different from the captured image. However, as
described above, by performing the predetermined
30 calculation on the data of the detection image, it is
possible to restore the captured image, in other words,
21
an image on which the subject image is formed and that
can be recognized as an image when viewed by the user
(that is, the subject can be visually recognized). This
restored captured image is referred to as a restored
image. That is, the detection 5 image is an image
different from the restored image.
[0049]
Note that, an image constituting the restored image,
and before synchronization processing, color separation
10 processing, or the like (for example, demosaic processing
or the like) is referred to as a Raw image. Similarly to
the captured image, the Raw image is also an image that
can be visually recognized by the user (that is, the
subject can be visually recognized). In other words, the
15 detection image is an image according to an arrangement
of color filters, but is an image different from the Raw
image.
[0050]
However, in a case where the imaging element 121
20 has sensitivity only to invisible light, for example,
infrared light, ultraviolet light, or the like, the
restored image (Raw image or captured image) becomes an
image that cannot be recognized as an image when viewed
by the user (the subject cannot be visually recognized).
25 However, since this depends on a wavelength range of
detected light, the restored image can be an image in
which the subject can be visually recognized, by
converting the wavelength range to a visible light range.
On the other hand, since the subject image is not formed,
30 the detection image cannot be an image in which the
subject can be visually recognized, only by converting
22
the wavelength range. Thus, even in a case where the
imaging element 121 has sensitivity only to the invisible
light, the image obtained by performing the predetermined
calculation on the detection image as described above is
referred to as the restored image. 5 Note that, in the
following, the present technology will be described by
using an example case where the imaging element 121
receives visible light basically, unless otherwise
specified.
10 [0051]
That is, the imaging element 121 can image a
subject, and obtain data regarding the detection image.
For example, the imaging element 121 can supply the data
regarding the detection image to the restoration unit 124
15 via the read control unit 122, and cause the restored
image to be generated. Furthermore, for example, the
imaging element 121 can supply the data regarding the
detection image to the associating unit 125 and the like
via the read control unit 122, and cause metadata and the
20 like to be associated. Of course, the imaging element
121 can supply the data regarding the detection image to
an arbitrary processing unit or the like. Furthermore,
for example, the imaging element 121 may acquire control
information regarding the imaging element 121 supplied
25 via the bus 110, and operate on the basis of the control
information.
[0052]
The read control unit 122 is configured to perform
processing related to data read control from the imaging
30 element 121, and control a resolution of the detection
image. For example, the read control unit 122 controls
23
reading of the detection image from the imaging element
121, and selectively reads the detection signal that is
an output from each of the pixel output units of the
imaging element 121.
5 [0053]
For example, the read control unit 122 can read
detection signals from all pixel output units of the
imaging element 121 and select the detection signals of
all pixel output units read, as detection signals to be
10 included in the detection image.
[0054]
For example, the read control unit 122 can select
some pixel unit outputs among the plurality of pixel
output units of the imaging element 121, and read
15 detection signals from the pixel output units selected.
Furthermore, for example, the read control unit 122 can
read detection signals from all pixel output units of the
imaging element 121, and select some of the read
detection signals of the respective pixel output units,
20 as detection signals to be included in the detection
image.
[0055]
For example, the read control unit 122 can select
some pixel output units at an arbitrary position among
25 the plurality of pixel output units of the imaging
element 121. That is, for example, the read control unit
122 can select some pixel unit outputs at an arbitrary
position among the plurality of pixel output units of the
imaging element 121, and read detection signals from the
30 pixel output units selected. Furthermore, for example,
the read control unit 122 can read detection signals from
24
all pixel output units of the imaging element 121, and
select detection signals read from some pixel output
units at an arbitrary position among the read detection
signals of the respective pixel output units, as
detection signals to be included in the 5 detection image.
[0056]
For example, the read control unit 122 can select
some pixel output units in a positional relationship
having a predetermined regularity among the plurality of
10 pixel output units of the imaging element 121. That is,
for example, the read control unit 122 can select some
pixel unit outputs in the positional relationship having
the predetermined regularity among the plurality of pixel
output units of the imaging element 121, and read
15 detection signals from the pixel output units selected.
Furthermore, for example, the read control unit 122 can
read detection signals from all pixel output units of the
imaging element 121, and select detection signals read
from some pixel output units in the positional
20 relationship having the predetermined regularity among
the read detection signals of the respective pixel output
units, as detection signals to be included in the
detection image.
[0057]
25 For example, the read control unit 122 can select
pixel output units formed in one partial region of a
region in which the plurality of pixel output units of
the imaging element 121 is formed. That is, for example,
the read control unit 122 can select the pixel unit
30 outputs formed in the above-described partial region, and
read detection signals from the pixel output units
25
selected. Furthermore, for example, the read control
unit 122 can read detection signals from all pixel output
units of the imaging element 121, and select detection
signals read from the pixel output units formed in the
above-described partial region among 5 the read detection
signals of the respective pixel output units, as
detection signals to be included in the detection image.
[0058]
For example, the read control unit 122 can read
10 detection signals from all pixel output units of the
imaging element 121, add the read detection signals of
the respective pixel output units together for each
predetermined number, and set a detection signal group
after the addition as the detection image.
15 [0059]
Selecting a detection signal to be adopted as the
detection image also means selecting a non-adopted
detection signal. That is, the read control unit 122
controls (sets) the resolution of the detection image by
20 selecting detection signals (including a case where all
detection signals are selected). For example, the read
control unit 122 controls (sets) the resolution of the
detection image by reading detection signals of all
pixels from the imaging element 121, reading detection
25 signals from the imaging element 121 by thinning out the
detection signals, thinning out detection signals read
from the imaging element 121, or adding detection signals
read from the imaging element 121 together for each
predetermined number.
30 [0060]
The read control unit 122 supplies the read
26
detection image (whose resolution is set) (in a case
where thinning, addition, or the like is performed, the
detection image after processing) via the bus 110 to
other processing units and the like (for example, the
restoration matrix setting unit 123, the 5 restoration unit
124, the associating unit 125, and the like).
[0061]
The restoration matrix setting unit 123 is
configured to perform processing related to setting of a
10 restoration matrix. The detection image can be converted
into the restored image by performing the predetermined
calculation. Although details will be described later,
the predetermined calculation is to multiply detection
signals included in the detection image by predetermined
15 coefficients and add them together. That is, the
detection image can be converted into the restored image
by performing a predetermined matrix operation. In this
specification, a matrix including the above-described
coefficients used for the matrix operation is referred to
20 as a restoration matrix.
[0062]
For example, the restoration matrix setting unit
123 sets a restoration matrix corresponding to the
detection image whose resolution is set by the read
25 control unit 122 (a restoration matrix used when the
restored image is restored from the detection signals
selectively read by the read control unit 122). That is,
the restoration matrix corresponds to the resolution of
the detection image to be processed. For example, the
30 restoration matrix setting unit 123 supplies the set
restoration matrix to other processing units and the like
27
(for example, the restoration unit 124, the associating
unit 125, and the like) via the bus 110.
[0063]
Note that, in the predetermined matrix operation
for converting the detection image 5 into the restored
image, the detection image may be converted into the
restored image having an arbitrary resolution. In that
case, the restoration matrix setting unit 123 is only
required to set a restoration matrix having the number of
10 rows and the number of columns depending on the
resolution of the detection image and a target resolution
of the restored image.
[0064]
Note that, for example, the restoration matrix
15 setting unit 123 may acquire control information
regarding the restoration matrix setting unit 123
supplied via the bus 110, and operate on the basis of the
control information.
[0065]
20 The restoration unit 124 is configured to perform
processing related to generation of the restored image.
For example, the restoration unit 124 generates the
restored image from data (detection signals and the like)
regarding the detection image supplied from the imaging
25 element 121 by performing the predetermined calculation.
Furthermore, the restoration unit 124 supplies data
(pixel values and the like) regarding the generated
restored image to other processing units and the like via
the bus 110.
30 [0066]
Note that, in the imaging element 121, a detection
28
image in which a plurality of color components is mixed
is obtained by using color filters, for example, and a
Raw image in which the plurality of color components is
mixed may be obtained by performing the predetermined
calculation on the detection image 5 by the restoration
unit 124. Then, the restoration unit 124 may supply the
Raw image in which the plurality of color components is
mixed as the restored image to other processing units and
the like, or may perform synchronization processing,
10 color separation processing, or the like (for example,
demosaic processing or the like) on the Raw image, and
supply the image subjected to the processing as the
restored image to the other processing units and the like.
Of course, in the imaging element 121, a monochrome
15 detection image or a detection image for each color is
obtained, and synchronization processing, color
separation processing, or the like (for example, demosaic
processing or the like) may be unnecessary.
[0067]
20 Furthermore, the restoration unit 124 may perform,
on the restored image, arbitrary image processing, for
example, gamma correction (γ correction), white balance
adjustment, or the like, and supply data regarding the
restored image after image processing to other processing
25 units and the like. Moreover, the restoration unit 124
may convert the format of data of the restored image, or
compress the data with, for example, a predetermined
compression method such as joint photographic experts
group (JPEG), tagged image file format (TIFF), graphics
30 interchange format (GIF), or the like, and supply the
data after the conversion (compression) to the other
29
processing units and the like.
[0068]
Note that, for example, the restoration unit 124
may acquire control information regarding the restoration
unit 124 supplied via the bus 110, 5 and operate on the
basis of the control information.
[0069]
The associating unit 125 is configured to perform
processing related to data association. For example, the
10 associating unit 125 associates data (for example,
coefficients and the like) used for the predetermined
calculation for generating the restored image with data
(detection signals and the like) regarding the detection
image supplied from the imaging element 121 or the like.
15 [0070]
Here, the term “associate” means that, for example,
in processing of one information (data, command, program,
and the like), the other information is made to be usable
(linkable). That is, the pieces of information
20 associated with each other may be combined into one file
or the like, or may be individual pieces of information.
For example, information B associated with information A
may be transmitted on a transmission path different from
that for the information A. Furthermore, for example,
25 the information B associated with the information A may
be recorded on a recording medium different from that for
the information A (or another recording area of the same
recording medium). Note that, this “association” may be
for part of information, not the entire information. For
30 example, an image and information corresponding to the
image may be associated with each other in an arbitrary
30
unit such as a plurality of frames, one frame, or a
portion within a frame.
[0071]
Furthermore, for example, the associating unit 125
supplies the associated data to other 5 processing units
and the like via the bus 110. Note that, for example,
the associating unit 125 may acquire control information
regarding the associating unit 125 supplied via the bus
110, and operate on the basis of the control information.
10 [0072]
The sensor unit 126 is configured to perform
processing related to detection. For example, the sensor
unit 126 includes an arbitrary sensor, and detects a
predetermined parameter. For example, the sensor unit
15 126 detects a parameter related to a peripheral state of
the imaging apparatus 100, a parameter related to a state
of the imaging apparatus 100, and the like. For example,
the sensor unit 126 detects a parameter related to a
state of the imaging element 121. Furthermore, for
20 example, the sensor unit 126 supplies the detected
information to other processing unit and the like via the
bus 110. Note that, for example, the sensor unit 126 may
acquire control information regarding the sensor unit 126
supplied via the bus 110, and operate on the basis of the
25 control information.
[0073]

Next, the imaging element 121 will be described
with reference to Figs. 2 to 20.
30 [0074]

31
In this specification, the present technology will
be described by using the term “pixel” (or “pixel output
unit”). In this specification, the “pixel” (or “pixel
output unit”) refers to a division unit including at
least one physical configuration capable 5 of receiving
light independently from other pixels, of a region (also
referred to as a pixel region) in which physical
configurations for receiving incident light of the
imaging element 121 are formed. The physical
10 configuration capable of receiving light is, for example,
a photoelectric conversion element, and is, for example,
a photodiode (photo diode (PD)). The number of physical
configurations (for example, photodiodes) formed in one
pixel is arbitrary, and may be singular or plural. The
15 physical configuration's type, size, shape, and the like
are also arbitrary.
[0075]
Furthermore, in addition to the above-described
“physical configuration capable of receiving light”, the
20 physical configuration of the “pixel” unit includes all
physical configurations related to reception of incident
light, for example, an on-chip lens, a light-shielding
film, a color filter, a planarization film, an antireflection
film, and the like. Moreover, a configuration
25 such as a read circuit may be included. That is, the
physical configuration of the pixel unit may be any
configuration.
[0076]
Furthermore, a detection signal read from the
30 “pixel” (that is, the physical configuration of the pixel
unit) may be referred to as a “detection signal of a
32
pixel unit (or pixel output unit)” or the like. Moreover,
the detection signal of the pixel unit (or pixel output
unit) is also referred to as a “pixel unit detection
signal (or pixel output unit detection signal)”.
Furthermore, the pixel unit detection 5 signal is also
referred to as “pixel output”. Moreover, a value of the
pixel output is also referred to as “output pixel value”.
[0077]
A value (output pixel value) of a detection signal
10 of a pixel unit of the imaging element 121 can have an
incident angle directivity indicating a directivity with
respect to an incident angle of incident light from a
subject, independently of the others. That is, each
pixel unit (pixel output unit) of the imaging element 121
15 has a configuration in which the incident angle
directivity of the output pixel value indicating the
directivity with respect to the incident angle of the
incident light from the subject is settable independently.
For example, in the imaging element 121, output pixel
20 values of at least two pixel units can respectively have
different incident angle directivities indicating the
directivity with respect to the incident angle of the
incident light from the subject.
[0078]
25 Note that, as described above, since the number of
the “physical configurations capable of receiving light”
included in the “pixel (or pixel output unit)” is
arbitrary, the pixel unit detection signal may be a
detection signal obtained by a single “physical
30 configuration capable of receiving light”, or may be a
detection signal obtained by a plurality of the “physical
33
configurations capable of receiving light”.
[0079]
Furthermore, a plurality of the pixel unit
detection signals (output pixel values) can also be
combined into one at an arbitrary stage. 5 For example,
output pixel values of a plurality of pixels may be added
together in the analog signal state, or may be added
together after being converted into digital signals.
[0080]
10 Furthermore, after the detection signal is read
from the imaging element 121, in other words, in the
detection image, a plurality of detection signals can be
combined into a single signal, or a single detection
signal can be converted into a plurality of signals.
15 That is, the resolution (number of data) of the detection
image is variable.
[0081]
By the way, in the following, for convenience of
description, a description will be given assuming that
20 the imaging element 121 includes a pixel region in which
a plurality of pixels is arranged in a matrix (a pixel
array is formed), unless otherwise specified. Note that,
the arrangement pattern of pixels (or pixel output units)
of the imaging element 121 is arbitrary, and is not
25 limited to this example. For example, the pixels (or
pixel output units) may be arranged in a honeycomb
structure. Furthermore, for example, the pixels (or
pixel output units) may be arranged in one row (or one
column). That is, the imaging element 121 may be a line
30 sensor.
[0082]
34
Note that, the wavelength range in which the
imaging element 121 (pixels thereof) has sensitivity is
arbitrary. For example, the imaging element 121 (pixels
thereof) may have sensitivity to visible light, may have
sensitivity to invisible light such as 5 infrared light or
ultraviolet light, or may have sensitivity to both
visible light and invisible light. For example, in a
case where the imaging element detects far-infrared light
that is invisible light, a thermograph (an image
10 representing a heat distribution) can be generated by
using a captured image obtained in the imaging element.
However, in the case of an imaging element with an
imaging lens, glass is difficult to transmit far-infrared
light, so that an imaging lens including an expensive
15 special material is required, and there is a possibility
that manufacturing costs increase. Since the imaging
element 121 can image a subject without using an imaging
lens and the like and obtain data regarding the captured
image, an increase in manufacturing costs can be
20 suppressed by enabling the pixel to detect far-infrared
light. That is, imaging of far-infrared light can be
performed at lower cost (a thermograph can be obtained at
lower cost). Note that, in a case where the imaging
element 121 (pixels thereof) has sensitivity to invisible
25 light, the restored image does not become an image in
which the user can visually recognize the subject but
becomes an image in which the user cannot visually
recognize the subject. In other words, the restored
image may be an image of visible light, or may be an
30 image of invisible light (for example, (far) infrared
light, ultraviolet light, or the like).
35
[0083]

The imaging element 121 includes a plurality of
pixel output units that receives incident light entering
without passing through either an 5 imaging lens or a
pinhole, and each outputs one detection signal indicating
an output pixel value modulated by an incident angle of
the incident light. For example, the imaging element 121
has a configuration for causing incident angle
10 directivities each indicating the directivity with
respect to the incident angle of the incident light from
the subject of output pixel values of at least two pixel
output units among the plurality of pixel output units to
be different characteristics from each other. That is,
15 in that case, the imaging element 121 can obtain
detection signals for the plurality of pixel output units
(a plurality of pixel output unit detection signals), and
incident angle directivities each indicating the
directivity with respect to the incident angle of the
20 incident light from the subject of at least two pixel
output unit detection signals among the plurality of
pixel output unit detection signals are different from
each other.
[0084]
25 Here, “incident angle directivity” refers to a
light-receiving sensitivity characteristic depending on
an incident angle of incident light, in other words,
detection sensitivity with respect to the incident angle
of the incident light. For example, even when incident
30 light has the same light intensity, the detection
sensitivity may change depending on the incident angle.
36
Such a deviation in detection sensitivity (including a
case where there is no deviation) is referred to as
“incident angle directivity”.
[0085]
For example, when incident light 5 beams having the
same light intensity as each other enter physical
configurations of the two pixel output units at the same
incident angle as each other, signal levels (detection
signal levels) of detection signals of the pixel output
10 units can be different values from each other. The
imaging element 121 (each pixel output unit thereof) has
a physical configuration having such a feature.
[0086]
The incident angle directivity may be realized by
15 any method. For example, the incident angle directivity
may be realized by providing a light-shielding film, for
example, in front (light incident side) of a
photoelectric conversion element (photodiode or the like)
of an imaging element having a basic structure similar to
20 that of, for example, a general complementary metal oxide
semiconductor (CMOS) image sensor or the like.
[0087]
When imaging is performed only with a general
imaging element including pixels having the same incident
25 angle directivity as each other, light beams of
substantially the same light intensity enter all pixels
of the imaging element, and an image of the subject
formed cannot be obtained. Thus, in general, an imaging
lens or a pinhole is provided in front (light incident
30 side) of the imaging element. For example, by providing
the imaging lens, light from the subject surface can be
37
formed as the image on the imaging surface of the imaging
element. Thus, the imaging element can obtain a
detection signal of a level corresponding to the image of
the subject formed at each pixel (that is, a captured
image of the subject formed can be obtained). 5 However,
in this case, the size is physically increased, and there
has been a possibility that downsizing of the apparatus
becomes difficult. Furthermore, in a case where the
pinhole is provided, downsizing becomes possible as
10 compared with the case where the imaging lens is provided,
but the amount of light entering the imaging element is
reduced, so that measures are essential such as
increasing the exposure time or increasing the gain, and
there has been a possibility that blurring is likely to
15 occur in imaging of a high-speed subject, or natural
color expression is lost.
[0088]
On the other hand, the imaging element 121 has a
configuration for causing incident angle directivities of
20 output pixel values of at least two pixel output units
among the plurality of pixel output units to be different
characteristics from each other, for example. With such
a configuration, for example, the imaging element 121 has
incident angle directivity in which detection
25 sensitivities of the respective pixels are different from
each other. That is, the light-receiving sensitivity
characteristic depending on the incident angle of
incident light is different for each pixel. However, it
is not necessary that the light-receiving sensitivity
30 characteristics of all the pixels are completely
different from each other, and some pixels may have the
38
same light-receiving sensitivity characteristic, and some
pixels may have different light-receiving sensitivity
characteristics.
[0089]
For example, in Fig. 2, in 5 a case where it is
assumed that a light source constituting a subject
surface 131 is a point light source, in the imaging
element 121, light beams having the same light intensity
emitted from the same point light source are incident on
10 all pixels, but incident at different incident angles on
respective pixels. Then, since the pixels of the imaging
element 121 respectively have incident angle
directivities different from each other, the light beams
having the same light intensity are detected with
15 respective sensitivities different from each other. That
is, a detection signal is detected having a different
signal level for each pixel.
[0090]
In more detail, the sensitivity characteristic
20 depending on the incident angle of the incident light
received at each pixel of the imaging element 121, in
other words, the incident angle directivity depending on
the incident angle at each pixel is expressed by a
coefficient representing light-receiving sensitivity
25 depending on the incident angle, and the signal level of
the detection signal depending on the incident light in
each pixel (also referred to as a detection signal level)
is obtained by multiplication by a coefficient set
corresponding to the light-receiving sensitivity
30 depending on the incident angle of the incident light.
[0091]
39
More specifically, as illustrated in the upper left
part of Fig. 2, detection signal levels DA, DB, and DC at
positions Pa, Pb, and Pc are expressed by the following
equations (1) to (3), respectively.
5 [0092]
DA = α1 × a + β1 × b + γ1 × c
··· (1)
DB = α2 × a + β2 × b + γ2 × c
··· (2)
10 DC = α3 × a + β3 × b + γ3 × c
··· (3)
[0093]
Here, α1 is a coefficient set depending on an
incident angle of a light beam from a point light source
15 PA on the subject surface 131 to be restored at the
position Pa on the imaging element 121. Furthermore, β1
is a coefficient set depending on an incident angle of a
light beam from a point light source PB on the subject
surface 131 to be restored at the position Pa on the
20 imaging element 121. Moreover, γ1 is a coefficient set
depending on an incident angle of a light beam from a
point light source PC on the subject surface 131 to be
restored at the position Pa on the imaging element 121.
[0094]
25 As indicated in the equation (1), the detection
signal level DA at the position Pa is expressed by the
sum (composite value) of a product of a light intensity
“a” of the light beam from the point light source PA at
the position Pa and the coefficient α1, a product of a
30 light intensity “b” of the light beam from the point
light source PB at the position Pa and the coefficient β1,
40
and a product of a light intensity “c” of the light beam
from the point light source PC at the position Pa and the
coefficient γ1. In the following, coefficients αx, βx,
and γx (x is a natural number) are collectively referred
to as 5 a coefficient set.
[0095]
Similarly, a coefficient set α2, β2, and γ2 of the
equation (2) is a coefficient set that is set depending
on incident angles of light beams from the point light
10 sources PA, PB, and PC on the subject surface 131 to be
restored at the position Pb on the imaging element 121.
That is, as in the above-described equation (2), the
detection signal level DB at the position Pb is expressed
by the sum (composite value) of a product of the light
15 intensity “a” of the light beam from the point light
source PA at the position Pb and the coefficient α2, a
product of the light intensity “b” of the light beam from
the point light source PB at the position Pb and the
coefficient β2, and a product of the light intensity “c”
20 of the light beam from the point light source PC at the
position Pb and the coefficient γ2. Furthermore,
coefficients α3, β3, and γ3 in the equation (3) are a
coefficient set that is set depending on incident angles
of light beams from the point light sources PA, PB, and
25 PC on the subject surface 131 to be restored at the
position Pc on the imaging element 121. That is, as in
the above-described equation (3), the detection signal
level DC at the position Pc is expressed by the sum
(composite value) of a product of the light intensity “a”
30 of the light beam from the point light source PA at the
position Pc and the coefficient α3, a product of the
41
light intensity “b” of the light beam from the point
light source PB at the position Pc and the coefficient β3,
and a product of the light intensity “c” of the light
beam from the point light source PC at the position Pc
and the 5 coefficient γ3.
[0096]
As described above, these detection signal levels
are different from those in which the subject image is
formed since the light intensities of the light beams
10 emitted from the point light sources PA, PB, and PC are
mixed. That is, the detection signal level illustrated
in the upper right part of Fig. 2 is not the detection
signal level corresponding to the image (captured image)
in which the subject image is formed, so that the
15 detection signal level is different from the pixel value
illustrated in the lower right part of Fig. 2 (generally
they do not match).
[0097]
However, by configuring simultaneous equations
20 using the coefficient set α1, β1, and γ1, coefficient set
α2, β2, and γ2, coefficient set α3, β3, and γ3, and
detection signal levels DA, DB, and DC, and solving the
simultaneous equations of the above-described equations
(1) to (3) using a, b, and c as variables, it is possible
25 to obtain the pixel values at the respective positions Pa,
Pb, and Pc as illustrated in the lower right part of Fig.
2. As a result, a restored image is restored that is a
set of pixel values (an image in which the subject image
is formed).
30 [0098]
With such a configuration, the imaging element 121
42
can output one detection signal indicating an output
pixel value modulated by an incident angle of incident
light, in each pixel, without requiring an imaging lens,
an optical filter including a diffraction grating or the
like, a pinhole, or the like. As a result, 5 the imaging
lens, the optical filter including the diffraction
grating or the like, the pinhole, or the like is not an
essential configuration, so that it is possible to reduce
the height of the imaging apparatus, in other words, to
10 reduce the thickness in the light incident direction in a
configuration that realizes an imaging function.
[0099]

The left part of Fig. 3 illustrates a front view of
15 a part of a pixel array unit of a general imaging element,
and the right part of Fig. 3 illustrates a front view of
a part of a pixel array unit of the imaging element 121.
Note that, Fig. 3 illustrates an example in which the
pixel array unit has a configuration in which the number
20 of pixels in the horizontal direction × vertical
direction is 6 pixels × 6 pixels; however, the
configuration of the number of pixels is not limited to
this.
[0100]
25 The incident angle directivity can be formed by a
light-shielding film, for example. As illustrated in the
example of the left part of Fig. 3, in a general imaging
element 151, pixels 151a having the same incident angle
directivity are arranged in an array. On the other hand,
30 the imaging element 121 in the example of the right part
of Fig. 3 is provided with a light-shielding film 121b
43
that is one of modulation elements to cover a part of the
light-receiving region of the photodiode for each of
pixels 121a, and incident light entering each pixel 121a
is optically modulated depending on an incident angle.
Then, for example, by providing the light-5 shielding film
121b in a different range for each pixel 121a, the lightreceiving
sensitivity with respect to the incident angle
of the incident light differs for each pixel 121a, and
each pixel 121a has a different incident angle
10 directivity.
[0101]
For example, a pixel 121a-1 and a pixel 121a-2 have
different ranges of pixels shielded by a light-shielding
film 121b-1 and a light-shielding film 121b-2 provided
15 (at least one of light-shielding region (position) or
light-shielding area differs). In other words, in the
pixel 121a-1, the light-shielding film 121b-1 is provided
to shield a part of the left side in the light-receiving
region of the photodiode by a predetermined width, and in
20 the pixel 121a-2, the light-shielding film 121b-2 is
provided to shield a part of the right side in the lightreceiving
region by a width wider in the horizontal
direction than the light-shielding film 121b-1.
Similarly, in the other pixels 121a, the light-shielding
25 film 121b is provided so that a different range in the
light-receiving region is shielded for each pixel, and is
randomly arranged in the pixel array.
[0102]
Note that, since the amount of light that can be
30 received decreases as the ratio of covering the lightreceiving
region of each pixel increases, the range of
44
the light-shielding film 121b is desirably set to an area
that can secure a desired amount of light, and the area
of the light-shielding film 121b may be configured with a
limitation, for example, up to about 3/4 of a range
capable of receiving light at most. 5 In this way, it
becomes possible to secure the amount of light of greater
than or equal to the desired amount. However, if an
unshielded range is provided having a width corresponding
to the wavelength of light to be received, for each pixel,
10 it is possible to receive a minimum amount of light. In
other words, for example, in the case of a blue pixel (B
pixel), the wavelength is about 500 nm, and it is
possible to receive the minimum amount of light if the
pixel is not shielded from light of greater than or equal
15 to a width corresponding to this wavelength.
[0103]

With reference to Fig. 4, a configuration example
will be described of the imaging element 121 in this case.
20 The upper part of Fig. 4 is a side cross-sectional view
of the imaging element 121, and the middle part of Fig. 4
is a top view of the imaging element 121. Furthermore,
the side cross-sectional view in the upper part of Fig. 4
is an AB cross section in the middle part of Fig. 4.
25 Moreover, the lower part of Fig. 4 is a circuit
configuration example of the imaging element 121.
[0104]
The imaging element 121 having the configuration
illustrated in Fig. 4 includes a plurality of pixel
30 output units that receives incident light entering
without passing through either an imaging lens or a
45
pinhole, and each outputs one detection signal indicating
an output pixel value modulated by an incident angle of
the incident light. For example, the imaging element 121
has a configuration for causing incident angle
directivities each indicating the 5 directivity with
respect to the incident angle of the incident light from
the subject of output pixel values of at least two pixel
output units among the plurality of pixel output units to
be different characteristics from each other.
10 Furthermore, in this case, in the imaging element 121,
the plurality of pixel output units has a configuration
in which an incident angle directivity indicating the
directivity with respect to the incident angle of the
incident light from the subject is settable independently
15 for each of the pixel output units.
[0105]
In the imaging element 121 in the upper part of Fig.
4, the incident light enters from the upper side to the
lower side in the figure. Adjacent pixels 121a-15 and
20 121a-16 are of a so-called back-illuminated type in which
a wiring layer Z12 is provided in the lowermost layer in
the figure, and a photoelectric conversion layer Z11 is
provided thereon.
[0106]
25 Note that, in a case where it is not necessary to
distinguish the pixels 121a-15 and 121a-16, the pixels
121a-15 and 121a-16 are simply referred to as the pixel
121a, and the other configurations are also referred to
similarly. Furthermore, Fig.4 illustrates a side view
30 and a top view of two pixels constituting the pixel array
of the imaging element 121; however, needless to say, a
46
larger number of pixels 121a are arranged but
illustration thereof is omitted.
[0107]
Moreover, the pixels 121a-15 and 121a-16 include
photodiodes 121e-15 and 121e-16 in 5 the photoelectric
conversion layer Z11, respectively. Furthermore, on the
photodiodes 121e-15 and 121e-16, on-chip lenses 121c-15
and 121c-16, and color filters 121d-15 and 121d-16 are
formed from above, respectively.
10 [0108]
The on-chip lenses 121c-15 and 121c-16 focus the
incident light on the photodiodes 121e-15 and 121e-16.
[0109]
The color filters 121d-15 and 121d-16 are, for
15 example, optical filters that transmit light of specific
wavelengths such as red, green, blue, infrared, and white.
Note that, in the case of white, the color filters 121d-
15 and 121d-16 may be transparent filters, or do not have
to exist.
20 [0110]
In the photoelectric conversion layer Z11 of the
pixels 121a-15 and 121a-16, light-shielding films 121p-15
to 121p-17 are respectively formed at boundaries between
pixels, and crosstalk between adjacent pixels is
25 suppressed.
[0111]
Furthermore, the light-shielding films 121b-15 and
121b-16, which are one of the modulation elements, shield
a part of a light-receiving surface S as illustrated in
30 the upper part and the middle part of Fig. 4. A part of
the light-receiving surface S is shielded by the light47
shielding film 121b, whereby the incident light entering
the pixel 121a is optically modulated depending on the
incident angle. Since the pixel 121a detects the
optically modulated incident light, the pixel 121a has an
incident angle directivity. On the 5 light-receiving
surface S of the photodiodes 121e-15 and 121e-16 in the
pixels 121a-15 and 121a-16, different ranges are
respectively shielded by the light-shielding films 121b-
15 and 121b-16, whereby a different incident angle
10 directivity is set for each pixel. However, not limited
to a case where the ranges shielded from light are
different from each other in all the pixels 121a of the
imaging element 121, some pixels 121a may exist in which
the same range is shielded from light.
15 [0112]
With the configuration illustrated in the upper
part of Fig. 4, the right end of the light-shielding film
121p-15 and the upper end of the light-shielding film
121b-15 are connected together, and the left end of the
20 light-shielding film 121b-16 and the upper end of the
light-shielding film 121p-16 are connected together, and
they are configured to have an L shape when viewed from
the side.
[0113]
25 Moreover, the light-shielding films 121b-15 to
121b-17 and the light-shielding films 121p-15 to 121p-17
include metal, for example, tungsten (W), aluminum (Al),
or an alloy of Al and copper (Cu). Furthermore, the
light-shielding films 121b-15 to 121b-17 and the light30
shielding films 121p-15 to 121p-17 may be formed at the
same time with the same metal as wiring, in the same
48
process as a process of forming wiring in a semiconductor
process. Note that, the film thicknesses of the lightshielding
films 121b-15 to 121b-17 and the lightshielding
films 121p-15 to 121p-17 do not have to be the
same depending 5 on the position.
[0114]
Furthermore, as illustrated in the lower part of
Fig. 4, the pixel 121a includes a photodiode 161
(corresponding to the photodiode 121e), a transfer
10 transistor 162, a floating diffusion (FD) portion 163, a
selection transistor 164, an amplification transistor 165,
and a reset transistor 166, and is connected to a current
source 168 via a vertical signal line 167.
[0115]
15 The photodiode 161 is configured such that the
anode electrode is individually grounded, and the cathode
electrode is individually connected to the gate electrode
of the amplification transistor 165 via the transfer
transistor 162.
20 [0116]
The transfer transistors 162 are individually
driven in accordance with a transfer signal TG. For
example, when the transfer signal TG supplied to the gate
electrode of the transfer transistor 162 becomes the high
25 level, the transfer transistor 162 is turned on.
Therefore, charges accumulated in the photodiode 161 are
transferred to the FD portion 163 via the transfer
transistor 162.
[0117]
30 The amplification transistor 165 is an input unit
of a source follower that is a read circuit that reads a
49
signal obtained by photoelectric conversion in the
photodiode 161, and outputs a pixel signal of a level
corresponding to the charges accumulated in the FD
portion 163 to the vertical signal line 23. In other
words, the amplification transistor 165, 5 in which a drain
terminal is connected to a power supply voltage VDD and a
source terminal is connected to the vertical signal line
167 via the selection transistor 164, configures the
source follower together with the current source 168
10 connected to one end of the vertical signal line 167.
[0118]
The floating diffusion (FD) portion 163 is a
floating diffusion region including a charge capacitance
C1 provided between the transfer transistor 162 and the
15 amplification transistor 165, and temporarily accumulates
the charges transferred from the photodiode 161 via the
transfer transistor 162. The FD portion 163 is a charge
detection unit that converts charges into a voltage, and
the charges accumulated in the FD portion 163 are
20 converted into a voltage in the amplification transistor
165.
[0119]
The selection transistor 164 is driven in
accordance with a selection signal SEL, and is turned on
25 when the selection signal SEL supplied to the gate
electrode becomes the high level, and connects the
amplification transistor 165 and the vertical signal line
167 together.
[0120]
30 The reset transistor 166 is driven in accordance
with a reset signal RST. For example, the reset
50
transistor 166 is turned on when the reset signal RST
supplied to the gate electrode becomes the high level,
discharges the charges accumulated in the FD portion 163
to the power supply voltage VDD, and resets the FD
5 portion 163.
[0121]
With the circuit configuration described above, the
pixel circuit illustrated in the lower part of Fig. 4
operates as follows.
10 [0122]
In other words, as the first operation, the reset
transistor 166 and the transfer transistor 162 are turned
on, the charges accumulated in the FD portion 163 are
discharged to the power supply voltage VDD, and the FD
15 portion 163 is reset.
[0123]
As the second operation, the reset transistor 166
and the transfer transistor 162 are turned off, and an
exposure period is started, and charges corresponding to
20 the amount of light of the incident light are accumulated
by the photodiode 161.
[0124]
As the third operation, the reset transistor 166 is
turned on and the FD portion 163 is reset, and then the
25 reset transistor 166 is turned off. By this operation,
the FD portion 163 is reset, and set to a reference
potential.
[0125]
As the fourth operation, the potential of the FD
30 portion 163 in a reset state is output from the
amplification transistor 165 as the reference potential.
51
[0126]
As the fifth operation, the transfer transistor 162
is turned on, and the charges accumulated in the
photodiode 161 are transferred to the FD portion 163.
5 [0127]
As the sixth operation, the potential of the FD
portion 163 to which the charges of the photodiode are
transferred is output from the amplification transistor
165 as a signal potential.
10 [0128]
Through the above processing, the reference
potential is subtracted from the signal potential, and is
output as a detection signal by correlated double
sampling (CDS). A value of the detection signal (output
15 pixel value) is modulated depending on the incident angle
of the incident light from the subject, and the
characteristic (directivity) varies depending on the
incident angle (has incident angle directivity).
[0129]
20 As described above, the pixel 121a in the case of
Fig. 4 is provided with one photodiode 121e for each
pixel, and a different range for each pixel 121a is
shielded by the light-shielding film 121b, and by optical
modulation using the light-shielding film 121b, a
25 detection signal for one pixel of a detection image
having incident angle directivity can be expressed by one
pixel 121a.
[0130]

30 Furthermore, the incident angle directivity can be
formed by, for example, the position, size, shape, and
52
the like in a pixel of a light receiving element (for
example, a photodiode). Pixels having different
parameters have different sensitivities to incident light
having the same light intensity from the same direction.
That is, by setting these parameters for 5 each pixel, the
incident angle directivity can be set for each pixel.
[0131]
For example, a plurality of light receiving
elements (for example, photodiodes) may be provided in a
10 pixel and used selectively. In this way, the incident
angle directivity can be set for each pixel by selection
of the light receiving element.
[0132]
Fig. 5 is a diagram illustrating another
15 configuration example of the imaging element 121. The
upper part of Fig. 5 illustrates a side cross-sectional
view of the pixel 121a of the imaging element 121, and
the middle part of Fig. 5 illustrates a top view of the
imaging element 121. Furthermore, the side cross20
sectional view of the upper part of Fig. 5 is an AB cross
section in the middle part of Fig. 5. Moreover, the
lower part of Fig. 5 is a circuit configuration example
of the imaging element 121.
[0133]
25 The imaging element 121 having the configuration
illustrated in Fig. 5 includes a plurality of pixel
output units that receives incident light entering
without passing through either an imaging lens or a
pinhole, and each outputs one detection signal indicating
30 an output pixel value modulated by an incident angle of
the incident light. For example, the imaging element 121
53
has a configuration for causing incident angle
directivities each indicating the directivity with
respect to the incident angle of the incident light from
the subject of output pixel values of at least two pixel
output units among the plurality of pixel 5 output units to
be different characteristics from each other.
Furthermore, in the imaging element 121 of this case, the
plurality of pixel output units can set the incident
angle directivity of the output pixel value indicating
10 the directivity with respect to the incident angle of the
incident light from the subject independently for each
pixel output unit, by making photo diodes (PDs) that
contribute to output different from each other.
[0134]
15 As illustrated in Fig. 5, the imaging element 121
has a configuration different from that of the imaging
element 121 of Fig. 5 in that four photodiodes 121f-1 to
121f-4 are formed in the pixel 121a, and a lightshielding
film 121p is formed in a region that separates
20 the photodiodes 121f-1 to 121f-4 from each other. In
other words, in the imaging element 121 of Fig. 5, the
light-shielding film 121p is formed in a “+” shape when
viewed from the top. Note that, common components are
denoted by the same reference signs, and a detailed
25 description thereof will be omitted.
[0135]
In the imaging element 121 configured as
illustrated in Fig. 5, the photodiodes 121f-1 to 121f-4
are separated by the light-shielding film 121p, whereby
30 an electrical and optical crosstalk between the
photodiodes 121f-1 to 121f-4 can be prevented. In other
54
words, similarly to the light-shielding film 121p of the
imaging element 121 of Fig. 4, the light-shielding film
121p of Fig. 5 is for preventing the crosstalk, and is
not for providing the incident angle directivity.
5 [0136]
Although details will be described later, the
photodiodes 121f-1 to 121f-4 have different incident
angles at which the light-receiving sensitivity
characteristics increase. That is, a desired incident
10 angle directivity can be given to the output pixel value
of the pixel 121a depending on which of the photodiodes
121f-1 to 121f-4 is used to read charges. That is, it is
possible to control the incident angle directivity of the
output pixel value of the pixel 121a.
15 [0137]
In the configuration example of the imaging element
121 of Fig. 5, one FD portion 163 is shared by four
photodiodes 121f-1 to 121f-4. The lower part of Fig. 5
illustrates a circuit configuration example in which one
20 FD portion 163 is shared by four photodiodes 121f-1 to
121f-4. Note that, in the lower part of Fig. 5,
descriptions will be omitted of the same components as
those of the lower part of Fig. 4.
[0138]
25 The lower part of Fig. 5 differs from the circuit
configuration of the lower part of Fig. 4 in that
photodiodes 161-1 to 161-4 (corresponding to photodiodes
121f-1 to 121f-4 in the upper part of Fig. 5) and
transfer transistors 162-1 to 162-4 are provided instead
30 of the photodiode 161 and the transfer transistor 162,
and the FD portion 163 is shared.
55
[0139]
In the circuit illustrated in the lower part of Fig.
5, the photodiodes 161-1 to 161-4 are referred to as the
photodiode 161 in a case where it is not necessary to
distinguish them from each other. 5 Furthermore, the
transfer transistors 162-1 to 162-4 are referred to as
the transfer transistors 162 in a case where it is not
necessary to distinguish them from each other.
[0140]
10 In the circuit illustrated in the lower part of Fig.
5, when any of the transfer transistors 162 is turned on,
the charges of the photodiode 161 corresponding to the
transfer transistor 162 is read, and transferred to the
common FD portion 163. Then, a signal corresponding to a
15 level of the charges held in the FD portion 163 is read
as a detection signal in a pixel output unit. That is,
the charges of each photodiode 161 can be read
independently of each other, and it is possible to
control which of the photodiodes 161 is used to read the
20 charges depending on which transfer transistor 162 is
turned on. In other words, it is possible to control the
degree of contribution to the output pixel value by each
photodiode 161 depending on which transfer transistor 162
is turned on. For example, the photodiodes 161 that
25 contribute to the output pixel value can be made
different from each other by making the photodiodes 161
that read the charges different from each other between
at least two pixels. That is, by selection of the
photodiode 161 that reads the charges, a desired incident
30 angle directivity can be given to the output pixel value
of the pixel 121a. That is, the detection signal output
56
from each pixel 121a can be a value (output pixel value)
modulated depending on the incident angle of the incident
light from the subject.
[0141]
For example, in Fig. 5, 5 the charges of the
photodiodes 121f-1 and 121f-3 are transferred to the FD
portion 163, and the signals obtained by reading them are
added together, whereby incident angle directivity in the
horizontal direction in the figure can be given to the
10 output pixel value of the pixel 121a. Similarly, the
charges of the photodiode 121f-1 and the photodiode 121f-
2 are transferred to the FD portion 163, and the signals
obtained by reading them are added together, whereby
incident angle directivity in the vertical direction in
15 the figure can be given to the output pixel value of the
pixel 121a
[0142]
Note that, the signals obtained on the basis of the
charges of respective photodiodes 121f of the pixel 121a
20 of Fig. 5 may be added together after being read from the
pixel, or may be added together within the pixel (for
example, the FD portion 163).
[0143]
Furthermore, the combination is arbitrary of the
25 photodiodes 121f for adding the charges (or the signals
corresponding to the charges) together, and is not
limited to the above example. For example, the charges
(or the signals corresponding to the charges) of three or
more photodiodes 121f may be added together. Furthermore,
30 for example, the charges of one photodiode 121f may be
read without performing addition.
57
[0144]
Note that, for example, by resetting detection
values (charges) accumulated in the photodiode 161
(photodiode 121f) by using the electronic shutter
function before reading the charges to the 5 FD portion 163,
a desired incident angle directivity may be given to the
pixel 121a (detection sensitivity thereof).
[0145]
For example, in a case where the electronic shutter
10 function is used, if the reset of the charges of the
photodiode 121f is performed immediately before the
reading to the FD portion 163, the photodiode 121f can be
made to have no contribution to the detection signal
level of the pixel 121a, and if a time is given between
15 the reset and the reading to the FD portion 163, a
partial contribution can be made.
[0146]
As described above, each of the pixels 121a of Fig.
5 includes four photodiodes 121f, and, although the
20 light-shielding film 121b is not formed on the lightreceiving
surface, is divided into a plurality of regions
by the light-shielding film 121p, and the four
photodiodes 121f-1 to 121f-4 are formed, and expresses a
detection signal for one pixel of a detection image
25 having incident angle directivity. In other words, for
example, a range that does not contribute to output among
the photodiodes 121f-1 to 121f-4 functions similarly to a
region shielded from light, and expresses a detection
signal for one pixel of a detection image having incident
30 angle directivity. Note that, in a case where the
detection signal for one pixel is expressed by using the
58
photodiodes 121f-1 to 121f-4, since the light-shielding
film 121b is not used, the detection signal is not a
signal obtained by optical modulation.
[0147]
In the above, an example has 5 been described in
which four photodiodes are arranged in a pixel; however,
the number of photodiodes arranged in the pixel is
arbitrary and is not limited to the above example. That
is, the number of partial regions is also arbitrary in
10 which the photodiodes are arranged in the pixel.
[0148]
Furthermore, in the above description, the
photodiodes are arranged in four partial regions obtained
by equally dividing the inside of the pixel into four
15 regions; however, the partial regions do not have to be
equally divided. That is, the sizes and shapes of the
partial regions do not have to be unified (partial
regions having different sizes and shapes may be
included). Alternatively, the position (position in the
20 partial region), size, shape, and the like of the
photodiodes arranged in each partial region may be
different for each photodiode (for each partial region).
At that time, the sizes and shapes of the partial regions
may be all unified or may not be unified.
25 [0149]
Moreover, these parameters do not have to be
unified for all the pixels of the imaging element 121.
That is, in one or more pixels of the imaging element 121,
one or more of these parameters may be different from
30 those of other pixels.
[0150]
59
For example, the pixel group of the imaging element
121 may include a pixel in which a division position for
forming the partial region in which the photodiode is
arranged in the pixel is different from that of other
pixels. That is, the imaging element 121 5 may include one
or more pixels whose partial regions have different sizes
and shapes from those of other pixels. For example, by
making the division position different for each pixel,
even if only the upper left photodiode is used in a
10 plurality of pixels, the incident angle directivity of
the detection signal detected in each of the plurality of
pixels can be made different from each other.
[0151]
Furthermore, for example, the pixel group of the
15 imaging element 121 may include a pixel in which the
position, size, shape, and the like of a plurality of
photodiodes arranged in the pixel are different from
those of other pixels. That is, the imaging element 121
may include one or more pixels in which at least one of
20 the position, size, or shape of the plurality of
photodiodes arranged is different from that of other
pixels. For example, by making the position, size, shape,
and the like of the photodiode different for each pixel,
even if only the upper left photodiode is used in a
25 plurality of pixels, the incident angle directivity of
the detection signal detected in each of the plurality of
pixels can be made different from each other.
[0152]
Moreover, for example, one or more pixels may be
30 included in which both the parameters (size, shape) of
the partial region and the parameters (position, size,
60
shape) of the photodiode are different from those of
other pixels.
[0153]
Furthermore, for example, the pixel group of the
imaging element 121 may include a 5 pixel in which the
number of divisions for forming the partial region in
which the photodiode is arranged in the pixel is
different from that of other pixels. That is, the
imaging element 121 may include one or more pixels in
10 which the number of photodiodes arranged is different
from that of other pixels. For example, by making the
number of divisions (the number of photodiodes) different
for each pixel, the incident angle directivity can be set
more freely.
15 [0154]

The incident angle directivity of each pixel in the
imaging element 121 is generated on the basis of a
principle illustrated in Fig. 6, for example. Note that,
20 the upper left part and the upper right part of Fig. 6
are diagrams illustrating a generation principle of the
incident angle directivity in the imaging element 121 of
Fig. 4, and the lower left part and lower right part of
Fig. 6 are diagrams illustrating a generation principle
25 of the incident angle directivity in the imaging element
121 of Fig. 5.
[0155]
Furthermore, each of the pixels in the upper left
part and the upper right part of Fig. 6 includes one
30 photodiode 121e. On the other hand, each of the pixels
in the lower left part and the lower right part of Fig. 6
61
includes two photodiodes 121f. Note that, here, an
example is described in which one pixel includes two
photodiodes 121f; however, this is for convenience of
description, and the number of photodiodes 121f included
in one pixel may be 5 the other number.
[0156]
In the upper left part of Fig. 6, a light-shielding
film 121b-11 is formed to shield the right half of the
light-receiving surface of the photodiode 121e-11 when
10 incident light enters from the upper side to the lower
side in the figure. Furthermore, in the upper right part
of Fig. 6, a light-shielding film 121b-12 is formed to
shield the left half of the light-receiving surface of
the photodiode 121e-12. Note that, it is indicated that
15 one-dot chain lines in the figure are at the center
position in the horizontal direction in the figure of the
light-receiving surface of the photodiode 121e, and is in
the vertical direction with respect to the lightreceiving
surface.
20 [0157]
For example, in the case of the configuration
illustrated in the upper left part of Fig. 6, incident
light from the upper right direction in the figure
indicated by an arrow forming an incident angle θ1 with
25 respect to the one-dot chain line in the figure is easily
received in a left half range that is not shielded by the
light-shielding film 121b-11 of the photodiode 121e-11,
but incident light from the upper left direction in the
figure indicated by an arrow forming an incident angle θ2
30 with respect to the one-dot chain line in the figure is
not easily received in the left half range that is not
62
shielded by the light-shielding film 121b-11 of the
photodiode 121e-11. Thus, in the case of the
configuration illustrated in the upper left part of Fig.
6, an incident angle directivity is given such that the
light-receiving sensitivity characteristic 5 is high for
the incident light from the upper right in the figure,
and the light-receiving sensitivity characteristic is low
for the incident light from the upper left.
[0158]
10 On the other hand, for example, in the case of the
configuration illustrated in the upper right part of Fig.
6, incident light from the upper right direction in the
figure indicated by an arrow forming an incident angle
θ11 with respect to the one-dot chain line in the figure
15 is not easily received in a left half range that is
shielded by the light-shielding film 121b-12 of the photo
diode 121e-12, but incident light from the upper left
direction in the figure indicated by an arrow forming an
incident angle θ12 with respect to the one-dot chain line
20 in the figure is easily received in a right half range
that is not shielded by the light-shielding film 121b-12
of the photodiode 121e-12. Thus, in the case of the
configuration illustrated in the upper right part of Fig.
6, an incident angle directivity is given such that the
25 light-receiving sensitivity characteristic is low for the
incident light from the upper right in the figure, and
the light-receiving sensitivity characteristic is high
for the incident light from the upper left.
[0159]
30 Furthermore, in the case of the lower left part of
Fig. 6, the photodiodes 121f-1 and 121f-2 are provided on
63
the left and right in the figure, and the configuration
is made to have the incident angle directivity without
providing the light-shielding film 121b by reading one of
the detection signals.
5 [0160]
In other words, in a case where two photodiodes
121f-1 and 121f-2 are formed in the pixel 121a as
illustrated in the lower left part of Fig. 6, by making
the detection signal of the photodiode 121f-1 provided on
10 the left side in the figure contribute to the detection
signal level of the pixel 121a, it is possible to have
the incident angle directivity similar to that of the
configuration in the upper left part of Fig. 6. In other
words, incident light from the upper right direction in
15 the figure, indicated by an arrow forming an incident
angle θ21 with respect to the one-dot chain line in the
figure, enters the photodiode 121f-1 and is received, and
the detection signal is read and contributes to the
detection signal level of the pixel 121a. On the other
20 hand, incident light from the upper left direction in the
figure, indicated by an arrow forming an incident angle
θ22 with respect to the one-dot chain line in the figure,
enters the photodiode 121f-2, but the detection signal is
not read and does not contribute to the detection signal
25 level of the pixel 121a.
[0161]
Similarly, in a case where two photodiodes 121f-11
and 121f-12 are formed in the pixel 121a as illustrated
in the lower right part of Fig. 6, by making the
30 detection signal of the photodiode 121f-12 provided on
the left side in the figure contribute to the detection
64
signal level of the pixel 121a, it is possible to have
the incident angle directivity similar to that of the
configuration in the upper right part of Fig. 6. In
other words, incident light from the upper right
direction in the figure, indicated by an 5 arrow forming an
incident angle θ31 with respect to the one-dot chain line
in the figure, enters the photodiode 121f-11, but the
detection signal is not read and does not contribute to
the detection signal level of the pixel 121a. On the
10 other hand, incident light from the upper left direction
in the figure, indicated by an arrow forming an incident
angle θ32 with respect to the one-dot chain line in the
figure, enters the photodiode 121f-12 and is received,
and the detection signal is read and contributes to the
15 detection signal level of the pixel 121a.
[0162]
Note that, in Fig. 6, an example has been described
in which the one-dot chain line in the vertical direction
is at the center position in the horizontal direction in
20 the figure of the light-receiving surface of the
photodiode 121e; however, this is for convenience of
description, and the one-dot chain line may be at another
position. Different incident angle directivities can be
generated by the difference in the horizontal position of
25 the light-shielding film 121b indicated by the one-dot
chain line in the vertical direction.
[0163]

30 In the above, the principle of generation of the
incident angle directivity has been described; however,
65
here, a description will be given of the incident angle
directivity in the configuration including the on-chip
lens 121c.
[0164]
In other words, the incident angle 5 directivity of
each pixel in the imaging element 121 is set, for example,
as illustrated in Fig. 7, by using the on-chip lens 121c,
in addition to that by the above-described lightshielding
film 121b. In other words, in the middle left
10 part of Fig. 7, from the incident direction in the upper
part of the figure, an on-chip lens 121c-11 that focuses
incident light, a color filter 121d-11 that transmits
light of a predetermined wavelength, and the photodiode
121e-11 that generates a pixel signal by photoelectric
15 conversion are layered in this order, and in the middle
right part of Fig. 7, from the incident direction in the
upper part of the figure, an on-chip lens 121c-12, a
color filter 121d-12, and the photodiode 121e-12 are
arranged in this order.
20 [0165]
Note that, in a case where it is not necessary to
distinguish between the on-chip lenses 121c-11 and 121c-
12, between the color filters 121d-11 and 121d-12, and
between the photodiodes 121e-11 and 121e-12, they are
25 simply referred to as the on-chip lenses 121c, the color
filter 121d, and the photodiode 121e.
[0166]
The imaging element 121 is further provided with
the light-shielding films 121b-11 and 121b-12 that shield
30 part of the region that receives incident light, as
respectively illustrated in the middle left part and the
66
middle right part of Fig. 7.
[0167]
As illustrated in the middle left part of Fig. 7,
in a case where the light-shielding film 121b-11 is
provided that shields the right half 5 of the photodiode
121e-11 in the figure, the detection signal level of the
photodiode 121e-11 changes depending on an incident angle
θ of the incident light as indicated by the solid line
waveform in the upper part of Fig. 7.
10 [0168]
In other words, when the incident angle θ, which is
an angle formed by the incident light with respect to the
one-dot chain line that is at the center position of the
photodiode 121e and the on-chip lens 121c and vertical to
15 each of the photodiode 121e and the on-chip lens 121c,
increases (when the incident angle θ increases in the
positive direction (inclines to the right direction in
the figure)), the light is focused on a range where the
light-shielding film 121b-11 is not provided, whereby the
20 detection signal level of the photodiode 121e-11
increases. Conversely, as the incident angle θ decreases
(as the incident angle θ increases in the negative
direction (inclines to the left direction in the figure)),
the light is focused on a range where the light-shielding
25 film 121b-11 is provided, whereby the detection signal
level of the photodiode 121e-11 decreases.
[0169]
Note that, the incident angle θ here is defined as
0 degrees in a case where the direction of the incident
30 light coincides with the one-dot chain line, and the
incident angle θ on the incident angle θ21 side in the
67
middle left of Fig. 7, at which incident light from the
upper right in the figure enters, is defined as a
positive value, and the incident angle θ on the incident
angle θ22 side in the middle right of Fig.7 is defined as
a negative value. Thus, in Fig. 7, the 5 incident angle of
the incident light entering the on-chip lens 121c from
the upper right is greater than the incident angle of the
incident light entering from the upper left. That is, in
Fig. 7, the incident angle θ increases as a direction of
10 travel of the incident light inclines to the right
(increases in the positive direction), and decreases as
the direction of travel inclines to the left (increases
in the negative direction).
[0170]
15 Furthermore, as illustrated in the middle right
part of Fig. 7, in a case where the light-shielding film
121b-12 is provided that shields the left half of the
photodiode 121e-12 in the figure, the detection signal
level of the photodiode 121e-12 changes depending on the
20 incident angle θ of the incident light as indicated by
the dotted line waveform in the upper part of Fig. 7.
[0171]
In other words, as indicated by the dotted line
waveform in the upper part of Fig. 7, as the incident
25 angle θ, which is an angle formed by the incident light
with respect to the one-dot chain line that is at the
center position of the photodiode 121e and the on-chip
lens 121c and vertical to each of the photodiode 121e and
the on-chip lens 121c, increases (as the incident angle θ
30 increases in the positive direction), the light is
focused on a range where the light-shielding film 121b-12
68
is provided, whereby the detection signal level of the
photodiode 121e-12 decreases. Conversely, as the
incident angle θ decreases (as the incident angle θ
increases in the negative direction), the light enters a
range where the light-shielding film 5 121b-12 is not
provided, whereby the detection signal level of the
photodiode 121e-12 increases.
[0172]
Note that, in the upper part of Fig. 7, the
10 horizontal axis indicates the incident angle θ, and the
vertical axis indicates the detection signal level in the
photodiode 121e.
[0173]
Since the waveforms indicated by the solid line and
15 the dotted line indicating the detection signal level
depending on the incident angle θ illustrated in the
upper part of Fig. 7 can be changed depending on the
range of the light-shielding film 121b, thus it becomes
possible to give (set) incident angle directivities
20 different from each other in respective pixel units.
Note that, the solid line waveform in the upper part of
Fig. 7 corresponds to solid line arrows indicating that
the incident light in the middle left part and the lower
left part of Fig. 7 is focused with the incident angle θ
25 changed. Furthermore, the dotted line waveform in the
upper part of Fig. 7 corresponds to dotted arrows
indicating that the incident light in the middle right
part and the lower right part of Fig. 7 is focused with
the incident angle θ changed.
30 [0174]
The incident angle directivity here is a
69
characteristic (light-receiving sensitivity
characteristic) of the detection signal level of each
pixel depending on the incident angle θ, but in the case
of the example of the middle part of Fig. 7, it can also
be said that this is a characteristic 5 of a light
shielding value depending on the incident angle θ. In
other words, the light-shielding film 121b blocks
incident light in a specific direction at a high level,
but cannot sufficiently block incident light from
10 directions other than the specific direction. This
change in level of shielding from light causes different
detection signal levels depending on the incident angle θ
as illustrated in the upper part of Fig. 7. Thus, when a
direction in which each pixel can be shielded at the
15 highest level from light is defined as a light shielding
direction of each pixel, having incident angle
directivities different from each other in respective
pixel units is, in other words, having light shielding
directions different from each other in respective pixels.
20 [0175]
Moreover, with a configuration in which two
photodiodes 121f-1 and 121f-2 are provided for one onchip
lens 121c-11 (a pixel output unit includes two
photodiodes 121f-1 and 121f-2) as illustrated in the
25 lower left part of Fig. 7, by using only the detection
signal of the photodiode 121f-1 in the left part of the
figure, it is possible to obtain the same detection
signal level as that in a state where the right side of
the photodiode 121e-11 in the middle left part of Fig. 7
30 is shielded from light.
[0176]
70
In other words, when the incident angle θ, which is
an angle formed by the incident light with respect to the
one-dot chain line that is the center position of the onchip
lens 121c and vertical to each, increases (when the
incident angle θ increases in the positive 5 direction),
the light is focused on a range of the photodiode 121f-1
from which the detection signal is read, whereby the
detection signal level increases. Conversely, as the
incident angle θ decreases (as the incident angle θ
10 increases in the negative direction), the light is
focused on a range of the photodiode 121f-2 from which
the detection value is not read, whereby the detection
signal level decreases.
[0177]
15 Furthermore, similarly, with a configuration in
which two photodiodes 121f-11 and 121f-12 are provided
for one on-chip lens 121c-12 as illustrated in the lower
right part of Fig. 7, by using only the detection signal
of the photodiode 121f-12 in the right part of the figure,
20 it is possible to obtain a detection signal of an output
pixel unit of the same detection signal level as that in
a state where the left side of the photodiode 121e-12 in
the middle right part of Fig. 7 is shielded from light.
[0178]
25 In other words, when the incident angle θ, which is
an angle formed by the incident light with respect to the
one-dot chain line that is at the center position of the
on-chip lens 121c and vertical to each, increases (when
the incident angle θ increases in the positive direction),
30 the light is focused on a range of the photodiode 121f-11
in which the detection signal does not contribute to the
71
detection signal of the output pixel unit, whereby the
detection signal level of the detection signal of the
output pixel unit decreases. Conversely, as the incident
angle θ decreases (as the incident angle θ increases in
the negative direction), the light is 5 focused on a range
of the photodiode 121f-12 in which the detection signal
contributes to the detection signal of the output pixel
unit, whereby the detection signal level of the detection
signal of the output pixel unit increases.
10 [0179]
Note that, it is desirable that the incident angle
directivity has high randomness. This is because there
is a possibility that, for example, when adjacent pixels
have the same incident angle directivity, the above15
described equations (1) to (3) or equations (4) to (6)
described later become the same equations as each other,
and the relationship cannot be satisfied between the
number of equations and the number of unknowns that are
the solutions of the simultaneous equations, and the
20 pixel values constituting the restored image cannot be
obtained. Furthermore, in the configuration illustrated
in the middle part of Fig. 7, one photodiode 121e-11 and
one photodiode 121e-12 are formed in the pixel 121a. On
the other hand, in the configuration illustrated in the
25 lower part of Fig. 7, two photodiodes 121f-1 and 121f-2,
and two photodiodes 121f-11 and 121f-12 are formed in the
pixel 121a. Thus, for example, in the lower part of Fig.
7, a single photodiode 121f does not constitute one pixel.
[0180]
30 Furthermore, as illustrated in the lower part of
Fig. 7, in a case where one pixel output unit includes a
72
plurality of photodiodes 121f, it can be considered that
the output pixel value of the pixel output unit is
modulated depending on the incident angle. Thus, the
characteristic (incident angle directivity) of the output
pixel value can be made different in 5 pixel output unit,
and the incident angle directivity in one pixel output
unit is set. Moreover, in the case where one pixel
output unit includes the plurality of photodiodes 121f, a
configuration is essential of one on-chip lens 121c for
10 one pixel output unit, for generating incident angle
directivity in one pixel output unit.
[0181]
Furthermore, as illustrated in the upper part of
Fig. 7, in a case where one photodiode 121e-11 or one
15 photodiode 121e-12 individually constitutes one pixel
output unit, incident light to one photodiode 121e-11 or
one photodiode 121e-12 constituting one pixel output unit
is modulated depending on the incident angle, whereby the
output pixel value is modulated as a result. Thus, the
20 characteristics (incident angle directivities) of the
output pixel value can be made different from each other,
and the incident angle directivity in one pixel output
unit is set. Moreover, in a case where one photodiode
121e-11 or one photodiode 121e-12 individually
25 constitutes one pixel output unit, the incident angle
directivity is set independently by the light-shielding
film 121b provided for each one pixel output unit at the
time of manufacturing.
[0182]
30 Furthermore, as illustrated in the lower part of
Fig. 7, in the case where one pixel output unit includes
73
the plurality of photodiodes 121f, positions and the
number of the plurality of photodiodes 121f (the number
of divisions of the photodiodes 121f constituting one
pixel output unit) for setting the incident angle
directivity for each one pixel output 5 unit are set
independently in one pixel output unit at the time of
manufacturing, and moreover, regarding which photodiode
121f is used for setting the incident angle directivity
among the plurality of photodiodes 121f, it is possible
10 to switch at the time of imaging.
[0183]

For example, as illustrated in the upper part of
Fig. 8, a setting range of the light-shielding film 121b
15 is set as a range from the left end to a position A in
the horizontal direction in the pixel 121a, and a range
from the upper end to a position B in the vertical
direction.
[0184]
20 In this case, a weight Wx of from 0 to 1 in the
horizontal direction is set, which serves as an index of
incident angle directivity depending on an incident angle
θx (deg) from the center position in the horizontal
direction of each pixel. In more detail, in a case where
25 it is assumed that the weight Wx is 0.5 at the incident
angle θx = θa corresponding to the position A, a weight
Wh is set so that the weight Wx is 1 at the incident
angle θx < θa - α, and the weight Wx is (-(θx - θa)/2α +
1/2) at θa -α ≤ the incident angle θx ≤ θa + α, and the
30 weight Wx is 0 at the incident angle θx > θa + α. Note
that, here, an example will be described in which the
74
weight Wh is 0, 0.5, and 1; however, the weight Wh is 0,
0.5, and 1 when an ideal condition is satisfied.
[0185]
Similarly, a weight Wy of from 0 to 1 in the
vertical direction is set, which serves 5 as an index of
incident angle directivity depending on an incident angle
θy (deg) from the center position in the vertical
direction of each pixel. In more detail, in a case where
it is assumed that the weight Wv is 0.5 at the incident
10 angle θy = θb corresponding to the position B, a weight
Wy is set so that the weight Wy is 0 at the incident
angle θy < θb - α, the weight Wy is ((θy - θb)/2α + 1/2)
at θb -α ≤ the incident angle θy ≤ θb + α, and the weight
Wy is 1 at the incident angle θy > θb + α.
15 [0186]
Then, by using the weights Wx and Wy thus obtained,
the incident angle directivity of each pixel 121a, in
other words, coefficients (coefficient set) corresponding
to the light-receiving sensitivity characteristic can be
20 obtained.
[0187]
Furthermore, at this time, an inclination (1/2α)
indicating a change in weight in a range where the weight
Wx in the horizontal direction and the weight Wy in the
25 vertical direction are around 0.5 is set by using the onchip
lens 121c having a different focal length.
[0188]
In other words, different focal lengths can be
obtained by using on-chip lenses 121c having different
30 curvatures.
[0189]
75
For example, by using the on-chip lens 121c having
a different curvature, as indicated by the solid line in
the lower part of Fig. 8, when light is focused so that
the focal length is on the light-shielding film 121b, the
inclination (1/2α) becomes steep. In other 5 words, in the
upper part of Fig. 8, the weight Wx in the horizontal
direction and the weight Wy in the vertical direction
sharply change to 0 or 1 in the vicinity of boundaries of
the incident angle θx = θa in the horizontal direction
10 and the incident angle θy = θb in the vertical direction
where the weights are near 0.5.
[0190]
Furthermore, for example, by using the on-chip lens
121c having a different curvature, when the focal length
15 is focused on the photodiode 121e as indicated by the
dotted line in the lower part of Fig. 8, the inclination
(1/2α) becomes moderate. In other words, in the upper
part of Fig. 8, the inclination moderately changes to 0
or 1 in the vicinity of boundaries of the incident angle
20 θx = θa in the horizontal direction and the incident
angle θy = θb in the vertical direction where the weight
Wx in the horizontal direction and the weight Wy in the
vertical direction are near 0.5.
[0191]
25 As described above, different incident angle
directivities, in other words, different light-receiving
sensitivity characteristics can be obtained by using the
on-chip lenses 121c having different curvatures to make
different focal lengths.
30 [0192]
Thus, the incident angle directivity of the pixel
76
121a can be set to a different value by making the range
in which the photodiode 121e is shielded by the lightshielding
film 121b and the curvature of the on-chip lens
121c different. Note that, the curvature of the on-chip
lens may be the same for all pixels 5 in the imaging
element 121, or may be different for some pixels.
[0193]

As described above, the imaging element 121 does
10 not require an imaging lens. However, the on-chip lens
121c is necessary at least in a case where the incident
angle directivity is realized by using the plurality of
photodiodes in the pixel as described with reference to
Fig. 5. The on-chip lens 121c and the imaging lens have
15 different physical functions.
[0194]
The imaging lens has a focusing function for
causing incident light entering from the same direction
to enter a plurality of pixels adjacent to each other.
20 On the other hand, light passing through the on-chip lens
121c is incident only on the light-receiving surface of
the photodiode 121e or 121f constituting one
corresponding pixel. In other words, the on-chip lens
121c is provided for each pixel output unit, and focuses
25 subject light entering the on-chip lens 121c on only the
corresponding pixel output unit. In other words, the onchip
lens 121c does not have a focusing function for
causing diffused light emitted from a virtual point light
source to enter a plurality of pixels adjacent to each
30 other.
[0195]
77

Next, a relationship between the subject surface
and the distance to the imaging element 121 will be
described with reference 5 to Fig. 9.
[0196]
As illustrated in the upper left part of Fig. 9, in
a case where a subject distance between the imaging
element 121 and the subject surface 131 is a distance d1,
10 for example, when the point light sources PA, PB, and PC
on the subject surface 131 are set, it is assumed that
the detection signal levels DA, DB, and DC at the
corresponding positions Pa, Pb, and Pc on the imaging
element 121 can be expressed by the same equations as the
15 equations (1) to (3) described above.
[0197]
DA = α1 × a + β1 × b + γ1 × c
··· (1)
DB = α2 × a + β2 × b + γ2 × c
20 ··· (2)
DC = α3 × a + β3 × b + γ3 × c
··· (3)
[0198]
On the other hand, as illustrated in the lower left
25 part of Fig. 9, in the case of a subject surface 131’ in
which the subject distance to the imaging element 121 is
a distance d2 greater than the distance d1 by d, in other
words, in the case of the subject surface 131’ that is
behind the subject surface 131 when viewed from the
30 imaging element 121, the detection signal levels DA, DB,
and DC are all similar, as illustrated in the upper
78
center part and the lower center part of Fig. 9.
[0199]
However, in this case, the light beams having light
intensities a’, b’, and c’ from point light sources PA’,
PB’, and PC’ on the subject surface 131’ 5 are received by
each pixel of the imaging element 121. At this time,
since the incident angles of the light beams having the
light intensities a’, b’, and c’ received on the imaging
element 121 differ (change), respective different
10 coefficient sets are required, and the detection signal
levels DA, DB, and DC in the respective positions Pa, Pb,
and Pc are expressed as indicated in the following
equations (4) to (6), for example.
[0200]
15 DA = α11 × a' + β11 × b' + γ11 × c'
··· (4)
DB = α12 × a' + β12 × b' + γ12 × c'
··· (5)
DC = α13 × a' + β13 × b' + γ13 × c'
20 ··· (6)
[0201]
Here, a coefficient set group including a
coefficient set α11, β11, and γ11, a coefficient set α12,
β12, and γ12, and a coefficient set α13, β13, and γ13 is
25 a coefficient set group of the subject surface 131’
respectively corresponding to the coefficient set α1, β1,
and γ1, the coefficient set α2, β2, and γ2, and the
coefficient set α3, β3, and γ3 in the subject surface 131.
[0202]
30 Thus, by solving the equations (4) to (6) by using
the preset coefficient set group α11, β11, γ11, α12, β12,
79
γ12, α13, β13, and γ13, it becomes possible to obtain the
light intensity (a’, b’, c’) of the light beams from the
point light sources PA’, PB’, and PC’, as illustrated in
the lower right part of Fig. 9, with a method similar to
the method of obtaining the light intensity 5 (a, b, c) of
the light beams in the point light sources PA, PB, and PC
in the case of the subject surface 131 as illustrated in
the upper right part of Fig. 9, and as a result, it
becomes possible to obtain a restored image of the
10 subject on the subject surface 131’.
[0203]
In other words, in the imaging apparatus 100 of Fig.
1, a coefficient set group for each distance from the
imaging element 121 to a subject surface is stored in
15 advance, simultaneous equations are configured by
switching the coefficient set groups, and the configured
simultaneous equations are solved, whereby it becomes
possible to obtain a restored image of the subject
surface at various subject distances on the basis of one
20 detection image.
[0204]
That is, by simply capturing the detection image
once, the restored image is obtained by switching the
coefficient set groups depending on the distance to the
25 subject surface in subsequent processing, whereby it is
also possible to generate a restored image at an
arbitrary distance.
[0205]
Furthermore, in the case of image recognition or in
30 a case where it is desired to obtain characteristics of a
subject such as a visible image or other than the visible
80
image, it is also possible to perform image recognition
or the like by using a detection signal itself by
applying machine learning such as deep learning to the
detection signal of the imaging element, without
performing the image recognition 5 on the basis of a
restored image after the restored image is obtained.
[0206]
Furthermore, in a case where the subject distance
and the angle of view can be specified, a restored image
10 may be generated by using a detection image including
detection signals of respective pixels each having an
incident angle directivity suitable for imaging the
subject surface corresponding to the specified subject
distance and angle of view, without using all the pixels.
15 In this way, a restored image can be obtained by using a
detection signal of a pixel suitable for imaging the
subject surface corresponding to the specified subject
distance and angle of view.
[0207]
20 For example, pixels are considered, a pixel 121a
that is shielded by the light-shielding film 121b by a
width d1 from each end of four sides as illustrated in
the upper part of Fig. 10, and a pixel 121a’ that is
shielded by the light-shielding film 121b by a width d2
25 (> d1) from each end of four sides as illustrated in the
lower part of Fig. 10.
[0208]
The pixel 121a is used, for example, for restoring
an image I1 of Fig. 10 corresponding to an angle of view
30 SQ1 including the whole of a person H101 as a subject, as
illustrated in the upper part of Fig. 11. On the other
81
hand, the pixel 121a’ is used, for example, for restoring
an image I2 of Fig. 10 corresponding to an angle of view
SQ2 in which the periphery of the face of the person H101
as the subject is zoomed up, as illustrated in the upper
5 part of Fig. 11.
[0209]
This is because the pixel 121a of Fig. 10 has an
incident light angle range A with respect to the imaging
element 121 as illustrated in the left part of Fig.12,
10 whereby incident light can be received for a subject
width W1 in the horizontal direction on the subject
surface 131.
[0210]
On the other hand, since the pixel 121a’ of Fig. 10
15 has a wider range shielded from light than that of the
pixel 121a of Fig. 10, an incident light angle range with
respect to the imaging element 121 is B (< A) as
illustrated in the left part of Fig. 12, so that incident
light can be received for a subject width W2 (< W1) in
20 the horizontal direction on the subject surface 131.
[0211]
That is, the pixel 121a of Fig. 10 with a narrow
light-shielding range is a wide angle-of-view pixel
suitable for imaging a wide range on the subject surface
25 131, whereas the pixel 121a’ of Fig. 10 with a wide
light-shielding range is a narrow angle-of-view pixel
suitable for imaging a narrow range on the subject
surface 131. Note that, the wide angle-of-view pixel and
the narrow angle-of-view pixel here are expressions for
30 comparing both the pixels 121a and 121a’ of Fig. 10 with
each other, and are not limited to these when comparing
82
pixels having other angles of view.
[0212]
Note that, Fig. 12 illustrates a relationship
between positions on the subject surface 131 and the
incident angle of incident light from each 5 position, with
respect to the center position C1 of the imaging element
121. Furthermore, Fig. 12 illustrates the relationship
with respect to the horizontal direction between the
positions on the subject surface 131 and the incident
10 angle of incident light from each position on the subject
surface 131, but there is a similar relationship for the
vertical direction. Moreover, on the right part of Fig.
12, the pixels 121a and 121a’ of Fig. 10 are illustrated.
[0213]
15 With such a configuration, as illustrated in the
lower part of Fig. 11, in the case of a configuration in
which a predetermined number of pixels 121a of Fig. 10
are gathered in a range ZA surrounded by the dotted line,
and the predetermined number of pixels 121a’ of Fig. 10
20 are gathered in a range ZB surrounded by the one-dot
chain line, in the imaging element 121, when an image of
the angle of view SQ1 corresponding to the subject width
W1 is to be restored, the pixel 121a of Fig. 10 that
images the angle of view SQ1 is used, whereby an image of
25 the subject width W1 on the subject surface 131 can be
appropriately restored.
[0214]
Similarly, when an image of the angle of view SQ2
corresponding to the subject width W2 is to be restored,
30 the detection signal level of the pixel 121a’ of Fig. 10
that images the angle of view SQ2 is used, whereby an
83
image of the subject width W2 can be appropriately
restored.
[0215]
Note that, in the lower part of Fig. 11, a
configuration is illustrated in which 5 the predetermined
number of pixels 121a' are provided on the left side in
the figure, and the predetermined number of pixels 121a
are provided on the right side; however, this is
illustrated as an example for simplifying the description,
10 and the pixel 121a and the pixel 121a’ are desirably
arranged to be randomly mixed.
[0216]
As described above, the angle of view SQ2 is
narrower than the angle of view SQ1, so in a case where
15 the images of the angle of view SQ2 and the angle of view
SQ1 are to be restored with the same predetermined number
of pixels, a restored image with higher image quality can
be obtained by restoring the image of the angle of view
SQ2 having a narrower angle of view, than restoring the
20 image of the angle of view SQ1.
[0217]
That is, in a case where it is considered to obtain
a restored image by using the same number of pixels, a
restored image with higher image quality can be obtained
25 by restoring an image with a narrower angle of view.
[0218]
Note that, in a case where an image with a wide
angle of view is obtained as a restored image, all pixels
of the wide angle-of-view pixels may be used, or some of
30 the wide angle-of-view pixels may be used. Furthermore,
in a case where an image with a narrow angle of view is
84
obtained as a restored image, all pixels of the narrow
angle-of-view pixels may be used, or some of the narrow
angle-of-view pixels may be used.
[0219]
By using the imaging element 121 as 5 described above,
as a result, an imaging lens, an optical element
including a diffraction grating or the like, a pinhole,
or the like is unnecessary (becomes imaging lens free),
so that it becomes possible to increase the degree of
10 freedom in apparatus design, and also possible to realize
downsizing of the apparatus with respect to the incident
direction of the incident light, and possible to reduce
the manufacturing cost. Furthermore, a lens is also
unnecessary corresponding to an imaging lens for forming
15 an optical image, such as a focus lens.
[0220]
Moreover, by using the imaging element 121, only a
detection image is acquired, and thereafter, a restored
image is obtained by solving simultaneous equations
20 configured by selectively using a coefficient set group
corresponding to the subject distance and the angle of
view, whereby it becomes possible to generate restored
images having various subject distances and angles of
view.
25 [0221]
Moreover, since the imaging element 121 can have an
incident angle directivity in a pixel unit, it is
possible to realize a multi-pixel, compared to an optical
filter including a diffraction grating, a conventional
30 imaging element, and the like, and also it is possible to
obtain a restored image with high resolution and high
85
angular resolution. On the other hand, in an imaging
apparatus including an optical filter and a conventional
imaging element, it is difficult to realize a high
resolution of a restored image, and the like since it is
difficult to miniaturize the optical 5 filter even if the
pixels are miniaturized.
[0222]
Furthermore, since the imaging element 121 does not
require an optical filter including a diffraction grating,
10 or the like, it does not occur that the optical filter is
distorted by heat due to temperature rise of the use
environment. Thus, by using such an imaging element 121,
it becomes possible to realize an apparatus with high
environmental resistance.
15 [0223]

In the right part of Fig. 3, as the configuration
of the light-shielding film 121b in each pixel 121a of
the imaging element 121, an example has been described in
20 which the entire light shielding is performed in the
vertical direction, and the light shielding width and
position are changed in the horizontal direction, whereby
a difference is given in the incident angle directivity
in the horizontal direction; however, the configuration
25 of the light-shielding film 121b is not limited to this
example. For example, the entire light-shielding is
performed in the horizontal direction, and the width
(height) and position are changed in the vertical
direction, whereby a difference may be given in the
30 incident angle directivity in the vertical direction.
[0224]
86
Note that, the light-shielding film 121b that
shields the entire pixel 121a in the vertical direction,
and shields the pixel 121a with a predetermined width in
the horizontal direction as in the example illustrated in
the right part of Fig. 3, is referred 5 to as a horizontal
band type light-shielding film 121b. On the other hand,
the light-shielding film 121b that shields the entire
pixel 121a in the horizontal direction, and shields the
pixel 121a with a predetermined height in the vertical
10 direction, is referred to as a vertical band type lightshielding
film 121b.
[0225]
Furthermore, as illustrated in the example
illustrated in the left part of Fig. 13, the pixel 121a
15 may be provided with an L-shaped light-shielding film
121b obtained by combining the vertical band type and the
horizontal band type light-shielding films 121b. In the
left part of Fig. 13, a portion indicated in black is the
light-shielding film 121b. That is, light-shielding
20 films 121b-21 to 121b-24 are light-shielding films of
pixels 121a-21 to 121a-24, respectively.
[0226]
Each of these pixels (pixels 121a-21 to 121a-24)
has incident angle directivity as illustrated in the
25 right part of Fig. 13. The graph illustrated in the
right part of Fig. 13 illustrates light-receiving
sensitivity in each pixel. The horizontal axis
represents the incident angle θx in the horizontal
direction (x direction) of the incident light, and the
30 vertical axis represents the incident angle θy in the
vertical direction (y direction) of the incident light.
87
Then, light-receiving sensitivity within a range C4 is
higher than that outside the range C4, light-receiving
sensitivity within a range C3 is higher than that outside
the range C3, light-receiving sensitivity within a range
C2 is higher than that outside the range 5 C2, and lightreceiving
sensitivity within a range C1 is higher than
that outside the range C1.
[0227]
Thus, it is indicated that, for each pixel, a
10 detection signal level of the incident light that
satisfies conditions of the incident angle θx in the
horizontal direction (x direction) and the incident angle
θy in the vertical direction (y direction) that are
within the range C1, is the highest, and the detection
15 signal level decreases in the order of the conditions of
being within the range C2, the range C3, the range C4,
and the range other than the range C4. Such intensity of
light-receiving sensitivity is determined by the range
shielded by the light-shielding film 121b.
20 [0228]
Furthermore, in the left part of Fig. 13, an
alphabet in each pixel 121a indicates color of a color
filter (the alphabet is indicated for convenience of
description, and is not actually written). The pixel
25 121a-21 is a G pixel in which a green color filter is
arranged, the pixel 121a-22 is an R pixel in which a red
color filter is arranged, the pixel 121a-23 is a B pixel
in which a blue color filter is arranged, and the pixel
121a-24 is a G pixel in which a green color filter is
30 arranged. That is, these pixels form a Bayer array. Of
course, this is an example, and the arrangement pattern
88
of the color filters is arbitrary. The arrangement of
the light-shielding film 121b and the color filter are
irrelevant. For example, in some or all of the pixels, a
filter other than the color filter may be provided, or no
filter 5 may be provided.
[0229]
In the left part of Fig. 13, an example is
illustrated in which an “L-shaped” light-shielding film
121b shields the left side and the lower side in the
10 figure of the pixel 121a; however, the orientation of the
“L-shaped” light-shielding film 121b is arbitrary, and is
not limited to the example of Fig. 13. For example, the
“L-shaped” light-shielding film 121b may shield the lower
side and the right side in the figure of the pixel 121a,
15 may shield the right side and the upper side in the
figure of the pixel 121a, or may shield the upper side
and the left side in the figure of the pixel 121a. Of
course, the orientation of the light-shielding film 121b
can be set independently for each pixel. Note that, the
20 “L-shaped” light-shielding film 121b is also collectively
referred to as “L-shaped type light-shielding film 121b”.
[0230]
Although the light-shielding film has been
described above, the description of this example can also
25 be applied to a case where incident angle directivity is
given by selectively using a plurality of photodiodes
arranged in a pixel. That is, for example, by
appropriately setting the division position (size and
shape of each partial region), and the position, size,
30 shape, and the like of each photodiode, or appropriately
selecting the photodiode, an incident light directivity
89
can be realized equivalent to the incident light
directivity by the above-described L-shaped type lightshielding
film 121b.
[0231]

In the above, an example has been described in
which the horizontal band type, the vertical band type,
and the L-shaped type light-shielding films are arranged
in each pixel so that the range shielded from light
10 randomly changes; however, for example, as illustrated by
an imaging element 121’ of Fig. 14, a light-shielding
film 121b may be formed that shields a range (a range
indicated in black in the figure) other than a range in
the vicinity of a position where a light beam is received
15 in each pixel in a case where a rectangular opening is
provided.
[0232]
In other words, the light-shielding film 121b may
be provided so that an incident angle directivity is
20 given in which only a light beam transmitted through the
rectangular opening is received among light beams emitted
from a point light source constituting a subject surface
at a predetermined subject distance in a case where the
rectangular opening is provided for each pixel.
25 [0233]
Note that, in Fig. 14, for example, the horizontal
width of the light-shielding film 121b changes to the
widths dx1, dx2, ··· dxn with respect to the horizontal
pixel arrangement, and there is a relationship of dx1 <
30 dx2 < ··· < dxn. Similarly, the vertical height of the
light-shielding film 121b changes to the heights dy1, dy2
90
··· dym with respect to the vertical pixel arrangement,
and there is a relationship of dy1 < dy2 < ··· < dxm.
Furthermore, an interval of the change in each of the
horizontal width and the vertical width of the lightshielding
film 121b depends on the 5 subject resolution
(angular resolution) to be restored.
[0234]
In other words, it can be said that the
configuration of each pixel 121a in the imaging element
10 121’ of Fig. 14 has incident angle directivity in which a
range shielded from light is changed to correspond to the
pixel arrangement in the imaging element 121’ in the
horizontal direction and the vertical direction.
[0235]
15 In more detail, the light-shielding range of each
pixel 121a of Fig. 14 is determined in accordance with a
rule described by using the pixel 121a illustrated in the
left part of Fig. 15, for example.
[0236]
20 Note that, the right part of Fig. 15 illustrates
the configuration of the same imaging element 121’ as
that of Fig. 14. Furthermore, the left part of Fig. 15
illustrates the configuration of the pixel 121a of the
imaging element 121’ in the right part of Fig. 15 (same
25 as Fig. 14).
[0237]
As illustrated in the left part of Fig. 15, the
pixel is shielded by the light-shielding film 121b by the
widths dx1 from the ends of the upper side and the lower
30 side of the pixel 121a toward the inside of the pixel
121a, respectively, and shielded by the light-shielding
91
film 121b by the heights dy1 from the ends of the left
side and the right side toward the inside of the pixel
121a, respectively. Note that, in Figs. 15 and 16, the
light-shielding film 121b is in a range indicated in
5 black.
[0238]
In the left part of Fig. 15, a range shielded from
light by such formation of the light-shielding film 121b
is hereinafter referred to as a main light-shielding
10 portion Z101 (black portion in the left part of Fig. 15)
of the pixel 121a, and a rectangular range other than
that is referred to as a range Z102.
[0239]
In the pixel 121a, a rectangular opening Z111 not
15 shielded by the light-shielding film 121b is provided in
the range Z102. Thus, in the range Z102, a range other
than the rectangular opening Z111 is shielded by the
light-shielding film 121b.
[0240]
20 In the pixel arrangement in the imaging element
121′ of Fig. 14, as illustrated in the right part of Fig.
15 (same as Fig. 14), the pixel 121a-1 at the upper left
end has a configuration in which the rectangular opening
Z111 is arranged so that its left side is at a distance
25 of the width dx1 from the left side of the pixel 121a,
and its upper side is at a distance of the dy1 from the
upper side of the pixel 121a.
[0241]
Similarly, the pixel 121a-2 on the right side of
30 the pixel 121a-1 has a configuration in which the
rectangular opening Z111 is arranged so that its left
92
side is at a distance of the width dx2 from the left side
of the pixel 121a, and its upper side is at a distance of
the height dy1 from the upper side of the pixel 121a, and
the range other than the rectangular opening Z111 is
shielded by the light-shielding 5 film 121b.
[0242]
Similarly, in the pixel 121a adjacent in the
horizontal direction, as the arrangement proceeds to the
right side in the figure, the right side of the
10 rectangular opening Z111 moves to the widths dx1, dx2 ···
dxn from the right side of the pixel 121a. Note that,
the dotted line rectangular portion of the upper right
part in the range Z102 of Fig. 15 illustrates a state in
which the rectangular opening Z111 is arranged so that
15 its left side is at a distance of the width dxn from the
left side of the pixel 121a, and its upper side is at a
distance of the height dy1 from the upper side of the
pixel 121a. Furthermore, each interval between the
widths dx1, dx2 ··· dxn is a value obtained by dividing
20 the width obtained by subtracting the width of the
rectangular opening Z111 from the horizontal width of the
range Z102 by the number of pixels n in the horizontal
direction. In other words, the interval of the change in
the horizontal direction is determined by division by the
25 number of pixels n in the horizontal direction.
[0243]
Furthermore, the horizontal position of the
rectangular opening Z111 in the pixel 121a in the imaging
element 121’ is the same in the pixels 121a having the
30 same horizontal position in the imaging element 121’
(pixels 121a in the same column).
93
[0244]
Moreover, the pixel 121a-3 immediately below the
pixel 121a-1 has a configuration in which the rectangular
opening Z111 is arranged so that its left side is at a
distance of the width dx1 from the left 5 side of the pixel
121a, and its upper side is at a distance of the height
dy2 from the upper side of the pixel 121a, and the range
other than the rectangular opening Z111 is shielded by
the light-shielding film 121b.
10 [0245]
Similarly, in the pixel 121a adjacent in the
vertical direction, as the arrangement proceeds to the
lower side in the figure, the upper side of the
rectangular opening Z111 moves to the heights dy1, dy2,
15 ··· dyn from the upper side of the pixel 121a. Note that,
the dotted line rectangular portion of the lower left
part in the range Z102 of Fig. 15 illustrates a state in
which the rectangular opening Z111 is arranged so that
its left side is at a distance of the width dx1 from the
20 left side of the pixel 121a, and its upper side is at a
distance of the height dym from the upper side of the
pixel 121a. Furthermore, each interval between the
heights dy1, dy2, ··· dym is a value obtained by dividing
the height obtained by subtracting the height of the
25 rectangular opening Z111 from the vertical height of the
range Z102 by the number of pixels m in the vertical
direction. In other words, the interval of the change in
the vertical direction is determined by division by the
number of pixels m in the vertical direction.
30 [0246]
Furthermore, the vertical position of the
94
rectangular opening Z111 in the pixel 121a in the imaging
element 121’ is the same in the pixels 121a having the
same vertical position in the imaging element 121’
(pixels 121a in the same row).
5 [0247]
Moreover, the angle of view can be changed by
changing the main light-shielding portion Z101 and the
rectangular opening Z111 of each pixel 121a constituting
the imaging element 121’ illustrated in Fig. 15 (Fig. 14).
10 [0248]
The right part of Fig. 16 illustrates a
configuration of the imaging element 121’ in a case where
the angle of view is wider than the imaging element 121’
of Fig. 15 (Fig. 14). Furthermore, the left part of Fig.
15 16 illustrates a configuration of the pixel 121a of the
imaging element 121’ in the right part of Fig. 16.
[0249]
In other words, as illustrated in the left part of
Fig. 16, for example, in the pixel 121a, a main light20
shielding portion Z151 (black portion in the left part of
Fig. 16) is set having a light-shielding range narrower
than that of the main light-shielding portion Z101 in Fig.
15, and a range other than that is set to a range Z152.
Moreover, in the range Z152, a rectangular opening Z161
25 is set having a wider opening area than that of the
rectangular opening Z111.
[0250]
In more detail, as illustrated in the left part of
Fig. 16, the pixel is shielded by the light-shielding
30 film 121b by the widths dx1’ (< dx1) from the ends of the
upper side and the lower side of the pixel 121a toward
95
the inside of the pixel 121a, respectively, and shielded
by the light-shielding film 121b by the heights dy1’ (<
dy1) from the ends of the left side and the right side
toward the inside of the pixel 121a, respectively,
whereby the rectangular opening 5 Z161 is formed.
[0251]
Here, as illustrated in the right part of Fig. 16,
the pixel 121a-1 at the upper left end has a
configuration in which the rectangular opening Z161 is
10 arranged so that its left side is at a distance of the
width dx1’ from the left side of the pixel 121a, and its
upper side is at a distance of the height dy1’ from the
upper side of the pixel 121a, and a range other than the
rectangular opening Z161 is shielded by the light15
shielding film 121b.
[0252]
Similarly, the pixel 121a-2 on the right side of
the pixel 121a-1 has a configuration in which the
rectangular opening Z161 is arranged so that its left
20 side is at a distance of the width dx2’ from the left
side of the pixel 121a, and its upper side is at a
distance of the height dy1’ from the upper side of the
pixel 121a, and the range other than the rectangular
opening Z161 is shielded by the light-shielding film 121b.
25 [0253]
Similarly, in the pixel 121a adjacent in the
horizontal direction, as the arrangement proceeds to the
right side in the figure, the right side of the
rectangular opening Z161 moves to the widths dx1’, dx2’
30 ··· dxn’ from the right side of the pixel 121a. Here,
each interval between the widths dx1’, dx2’ ··· dxn’ is a
96
value obtained by dividing the width obtained by
subtracting the horizontal width of the rectangular
opening Z161 from the horizontal width of the range Z152
by the number of pixels n in the horizontal direction.
In other words, the interval of 5 the change in the
vertical direction is determined by division by the
number of pixels n in the horizontal direction. Thus,
the interval of the change between the widths dx1’, dx2’
··· dxn’ is greater than the interval of the change
10 between the widths dx1, dx2 ··· dxn.
[0254]
Furthermore, the horizontal position of the
rectangular opening Z161 in the pixel 121a in the imaging
element 121’ of Fig. 16 is the same in the pixels 121a
15 having the same horizontal position in the imaging
element 121’ (pixels 121a in the same column).
[0255]
Moreover, the pixel 121a-3 immediately below the
pixel 121a-1 has a configuration in which the rectangular
20 opening Z161 is arranged so that its left side is at a
distance of the width dx1’ from the left side of the
pixel 121a, and its upper side is at the height dy2’ from
the upper side of the pixel 121a, and the range other
than the rectangular opening Z161 is shielded by the
25 light-shielding film 121b.
[0256]
Similarly, in the pixel 121a adjacent in the
vertical direction, as the arrangement proceeds to the
lower side in the figure, the upper side of the
30 rectangular opening Z161 changes to the heights dy1’, dy2’
··· dym’ from the upper side of the pixel 121a. Here,
97
the interval of the change between the heights dy1’, dy2’
··· dym’ is a value obtained by dividing the height
obtained by subtracting the height of the rectangular
opening Z161 from the vertical height of the range Z152
by the number of pixels m in the vertical 5 direction. In
other words, the interval of the change in the vertical
direction is determined by division by the number of
pixels m in the vertical direction. Thus, the interval
of the change between the heights dy1’, dy2’ ··· dym’ is
10 greater than the interval of the change between the width
heights dy1, dy2 ··· dym.
[0257]
Furthermore, the vertical position of the
rectangular opening Z161 in the pixel 121a in the imaging
15 element 121’ of Fig. 16 is the same in the pixels 121a
having the same vertical position in the imaging element
121’ (pixels 121a in the same row).
[0258]
As described above, by changing the combination of
20 the light-shielding range of the main light-shielding
portion and the opening range of the opening, it becomes
possible to realize the imaging element 121’ including
the pixels 121a having various angles of view (having
various incident angle directivities).
25 [0259]
Moreover, the imaging element 121 may be realized
by combining not only the pixels 121a having the same
angle of view but also the pixels 121a having various
angles of view.
30 [0260]
For example, as illustrated in Fig. 17, four pixels
98
including two pixels × two pixels indicated by a dotted
line are defined as one unit U, in which each unit U
includes a pixel 121a-W having a wide angle of view, a
pixel 121a-M having a medium angle of view, a pixel 121a-
N having a narrow angle of view, and 5 a pixel 121a-AN
having an extremely narrow angle of view.
[0261]
In this case, for example, in a case where the
number of pixels of all the pixels 121a is X, it becomes
10 possible to restore a restored image by using detection
images of X/4 pixels for each of the four types of angles
of view. At this time, four types of different
coefficient sets are used for respective angles of view,
and restored images having different angles of view are
15 restored by four types of different simultaneous
equations.
[0262]
For this reason, by restoring a restored image
having an angle of view to be restored using a detection
20 image obtained from a pixel suitable for imaging the
angle of view to be restored, it becomes possible to
restore an appropriate restored image corresponding to
each of the four types of angles of view.
[0263]
25 Furthermore, an image having an intermediate angle
of view between the four types of angles of view, or an
angle of view around the intermediate angle of view may
be generated by interpolation from images having the four
types of angles of view, and pseudo optical zoom may be
30 realized by seamlessly generating images having various
angles of view.
99
[0264]
Although the light-shielding film has been
described above, the description of this example can also
be applied to a case where incident angle directivity is
given by selectively using a plurality 5 of photodiodes
arranged in a pixel. That is, for example, by
appropriately setting the division position (size and
shape of each partial region), and the position, size,
shape, and the like of each photodiode, or appropriately
10 selecting the photodiode, an incident light directivity
can be realized equivalent to the incident light
directivity by the above-described light-shielding film
121b including the rectangular opening. Of course, also
in this case, the imaging element 121 can be realized by
15 combining the pixels 121a having various angles of view.
Furthermore, an image having an intermediate angle of
view, or an angle of view around the intermediate angle
of view may be generated by interpolation from images
having a plurality of types of angles of view, and pseudo
20 optical zoom may be realized by seamlessly generating
images having various angles of view.
[0265]

By the way, in a case where randomness is given to
25 a range shielded by the light-shielding film 121b of the
pixel 121a in the imaging element 121, as the randomness
of the difference in the range shielded by the lightshielding
film 121b increases, the processing load by the
restoration unit 124 and the like increases. Thus, the
30 processing load may be reduced by reducing the randomness
of the difference by making a part of the difference in
100
the range shielded by the light-shielding film 121b of
the pixel 121a have regularity.
[0266]
For example, an L-shaped type light-shielding film
121b obtained by combining a vertical 5 band type and a
horizontal band type is configured, and the horizontal
band type light-shielding films 121b having the same
width are combined for a predetermined column direction,
and the vertical band type light-shielding films 121b
10 having the same height are combined for a predetermined
row direction. In this way, the light-shielding range of
the light-shielding film 121b of each pixel 121a is set
to a different value randomly in the pixel unit while
having regularity in the column direction and the row
15 direction. As a result, it is possible to reduce the
difference in the light-shielding range of the lightshielding
film 121b of each pixel 121a, in other words,
the randomness of the difference in the incident angle
directivity of each pixel, and it is possible to reduce
20 the processing load outside the imaging element 121 such
as the restoration unit 124.
[0267]
For example, in the case of an imaging element 121’’
of Fig. 18, the horizontal band type light-shielding film
25 121b having the same width X0 is used for the pixels in
the same column indicated by a range Z130, and the
vertical band type light-shielding film 121b having the
same height Y0 is used for the pixels in the same row
indicated by a range Z150, and the L-shaped type light30
shielding film 121b in which these are combined is set
for the pixels 121a specified by each row and each column.
101
[0268]
Similarly, the horizontal band type light-shielding
film 121b having the same width X1 is used for the pixels
in the same column indicated by a range Z131 adjacent to
the range Z130, and the vertical 5 band type lightshielding
film 121b having the same height Y1 is used for
the pixels in the same row indicated by a range Z151
adjacent to the range Z150, and the L-shaped type lightshielding
film 121b in which these are combined is set
10 for the pixels 121a specified by each row and each column.
[0269]
Moreover, the horizontal band type light-shielding
film having the same width X2 is used for the pixels in
the same column indicated by a range Z132 adjacent to the
15 range Z131, and the vertical band type light-shielding
film having the same height Y2 is used for the pixels in
the same row indicated by a range Z152 adjacent to the
range Z151, and the L-shaped type light-shielding film
121b in which these are combined is set for the pixels
20 121a specified by each row and each column.
[0270]
In this way, it is possible to set the range of the
light-shielding film to a different value in the pixel
unit while giving regularity in the horizontal width and
25 position and the vertical height and position of the
light-shielding film 121b, so that it is possible to
suppress the randomness of the difference in the incident
angle directivity. As a result, it becomes possible to
reduce patterns of the coefficient set, and it becomes
30 possible to reduce the processing load of calculation
processing in the subsequent stage (for example, the
102
restoration unit 124 and the like).
[0271]

Variations in the shape of the light-shielding film
121b in pixel units are arbitrary, and 5 are not limited to
the above examples. For example, different incident
angle directivity may be given (set) by setting the
light-shielding film 121b as a triangle and making the
range different, or different incident angle directivity
10 may be given by setting the light-shielding film 121b as
a circle and making the range different. Furthermore,
for example, a light-shielding film or the like having a
linear shape in an oblique direction may be used.
[0272]
15 Furthermore, a variation (pattern) of the lightshielding
film 121b may be set by a plurality of pixel
units constituting a unit including a predetermined
number of multiple pixels. This one unit may include any
pixel. For example, the imaging element 121 may include
20 a color filter, and the unit may include a pixel
constituting a unit of color arrangement of the color
filter. Furthermore, a pixel group in which pixels
having different exposure times are combined may be used
as a unit. Note that, it is desirable that the
25 randomness of the pattern in the range shielded by the
light-shielding film 121b in each pixel constituting the
unit is high, in other words, the pixels constituting the
unit respectively have different incident angle
directivities.
30 [0273]
Furthermore, the arrangement pattern of the light103
shielding film 121b may be set between the units. For
example, the width and position of the light-shielding
film may be changed for each unit. Moreover, a pattern
in a range shielded by the light-shielding film 121b may
be set within a unit including a plurality 5 of pixels
classified in different categories or between units.
[0274]
Although the light-shielding film has been
described above, the description of this example can also
10 be applied to a case where incident angle directivity is
given by selectively using a plurality of photodiodes
arranged in a pixel. That is, for example, by
appropriately setting the division position (size and
shape of each partial region), and the position, size,
15 shape, and the like of each photodiode, or appropriately
selecting the photodiode, an incident light directivity
can be realized equivalent to the incident light
directivity in a case where a part of the change in the
range shielded by the light-shielding film 121b of the
20 pixel 121a described above is made to have regularity.
In this way, it is possible to reduce the randomness of
the difference in the incident angle directivity of each
pixel, and reduce the processing load outside the imaging
element 121 such as the restoration unit 122.
25 [0275]
Although the light-shielding film has been
described above, the description of this example can also
be applied to a case where incident angle directivity is
given by selectively using a plurality of photodiodes
30 arranged in a pixel. That is, by appropriately setting
the division position (size and shape of each partial
104
region), the position, size, shape, and the like of each
photodiode, or appropriately selecting the photodiode, an
incident light directivity can be realized equivalent to
the incident light directivity by a light-shielding film
having an arbitrary shape, for example, 5 a triangle, a
circle, a linear shape in an oblique direction, or the
like.
[0276]
Furthermore, for example, setting of the division
10 position (size and shape of each partial region), setting
of the position, size, shape, and the like of each
photodiode, selection of the photodiode, and the like may
be set for each unit similarly to the case of the lightshielding
film 121b described above.
15 [0277]

In a case where a plurality of photodiodes arranged
in a pixel as described above with reference to Fig. 5 is
selectively used, the incident angle directivity of the
20 output pixel value of the pixel output unit may be made
to be variously changed by switching the presence/absence
and degree of contribution to the output pixel value of
each pixel output unit of the plurality of photodiodes
121f.

CLAIMS
1. An imaging apparatus comprising:
an imaging element including a plurality of pixel
output units that receives incident 5 light entering
without passing through either an imaging lens or a
pinhole, and each outputs one detection signal indicating
an output pixel value modulated by an incident angle of
the incident light; and
10 a read control unit that selectively reads the
output pixel value of each of the pixel output units of
the imaging element.
2. The imaging apparatus according to claim 1, wherein
15 the read control unit selects some pixel unit
outputs among the plurality of pixel output units of the
imaging element, and reads output pixel values of the
pixel output units selected.
20 3. The imaging apparatus according to claim 2, wherein
the read control unit selects some pixel output
units at arbitrary positions among the plurality of pixel
output units of the imaging element.
25 4. The imaging apparatus according to claim 3, wherein
the read control unit selects the pixel output
units such that, regarding an incident angle directivity
of the output pixel value indicating a directivity with
respect to an incident angle of incident light from a
30 subject, a whole of the pixel output units selected has
the incident angle directivity equivalent to an incident
188
angle directivity of all pixel output units of the
imaging element.
5. The imaging apparatus according to claim 2, wherein
the read control unit selects 5 some pixel output
units in a positional relationship having a predetermined
regularity among the plurality of pixel output units of
the imaging element.
10 6. The imaging apparatus according to claim 5, wherein
regarding an incident angle directivity of the
output pixel value indicating a directivity with respect
to an incident angle of incident light from a subject, a
whole of the some pixel output units of the imaging
15 element in the positional relationship having the
regularity selected by the read control unit has the
incident angle directivity equivalent to an incident
angle directivity of all pixel output units of the
imaging element.
20
7. The imaging apparatus according to claim 2, wherein
the read control unit selects a pixel output unit
formed in one partial region of a region in which the
plurality of pixel output units of the imaging element is
25 formed.
8. The imaging apparatus according to claim 7, wherein
regarding an incident angle directivity of the
output pixel value indicating a directivity with respect
30 to an incident angle of incident light from a subject, a
whole of pixel output units of the imaging element formed
189
in the partial region selected by the read control unit
has the incident angle directivity equivalent to an
incident angle directivity of all pixel output units of
the imaging element.
5
9. The imaging apparatus according to claim 1, wherein
the read control unit reads the output pixel values
from all pixel output units of the imaging element, and
selects some of the output pixel values read.
10
10. The imaging apparatus according to claim 1, wherein
the read control unit reads output pixel values of
all pixel output units of the imaging element, and adds
the read output pixel values together for each
15 predetermined number.
11. The imaging apparatus according to claim 10,
wherein
the read control unit adds together output pixel
20 values of pixel output units, the output pixel values
having mutually similar incident angle directivities each
indicating a directivity with respect to an incident
angle of incident light from a subject.
25 12. The imaging apparatus according to claim 10,
wherein
the read control unit adds together output pixel
values of pixel output units close to each other.
30 13. The imaging apparatus according to claim 1, wherein
the plurality of pixel output units has a
190
configuration in which an incident angle directivity of
the output pixel value indicating a directivity with
respect to an incident angle of incident light from a
subject is settable independently for each of the pixel
5 output units.
14. The imaging apparatus according to claim 1, wherein
the plurality of pixel output units has a
configuration in which an incident angle directivity
10 indicating a directivity with respect to an incident
angle of incident light from a subject is settable
independently for each of the pixel output units.
15. The imaging apparatus according to claim 1, wherein
15 the plurality of pixel output units has a
configuration in which an incident angle directivity of
the output pixel value indicating a directivity with
respect to an incident angle of incident light from a
subject is settable independently for each of the pixel
20 output units by making photo diodes (PDs) that contribute
to output different from each other.
16. An imaging method comprising:
imaging a subject by an imaging element including a
25 plurality of pixel output units that receives incident
light entering without passing through either an imaging
lens or a pinhole, and each outputs one detection signal
indicating an output pixel value modulated by an incident
angle of the incident light; and
30 selectively reading the output pixel value of each
of the pixel output units of the imaging element.
191
17. An image processing apparatus comprising:
a resolution setting unit that sets a resolution;
and
a restoration matrix setting 5 unit that sets a
restoration matrix including coefficients used when a
restored image is restored from output pixel values of a
plurality of pixel output units, of an imaging element
including the plurality of pixel output units that
10 receives incident light entering without passing through
either an imaging lens or a pinhole, and each outputs one
detection signal indicating an output pixel value
modulated by an incident angle of the incident light,
depending on the resolution set by the resolution setting
15 unit.
18. The image processing apparatus according to claim
17, wherein
the resolution setting unit sets the resolution by
20 selecting output pixel values of some of the pixel output
units.
19. The image processing apparatus according to claim
17, wherein
25 the resolution setting unit sets the resolution by
adding the output pixel values of the pixel output units
together for each predetermined number.
20. An image processing method comprising:
30 setting a resolution; and
setting a restoration matrix including coefficients
192
used when a restored image is restored from output pixel
values of a plurality of pixel output units, of an
imaging element including the plurality of pixel output
units that receives incident light entering without
passing through either an imaging lens 5 or a pinhole, and
each outputs one detection signal indicating an output
pixel value modulated by an incident angle of the
incident light, depending on the resolution set.