FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION
(See section 10, rule 13)
“IMAGING DEVICE AND METHOD, AND IMAGE PROCESSING
DEVICE AND METHOD”
SONY CORPORATION, of 1-7-1, Konan, Minato-ku,
Tokyo 108-0075, Japan
The following specification particularly describes the invention and the manner in which it is to
be performed.
2
DESCRIPTION
IMAGING DEVICE AND METHOD, AND IMAGE PROCESSING DEVICE
AND METHOD
5 TECHNICAL FIELD
[0001]
The present technology relates to an imaging device
and method, and an image processing device and method,
and in particular, an imaging device and method, and an
10 image processing device and method that can more easily
suppress unauthorized use and tampering of an image.
BACKGROUND ART
[0002]
15 In recent years, size reduction of imaging elements
is in progress by devising an imaging element without an
imaging lens, or the like (see, for example, Patent
Document 1, Patent Document 2, and Non-Patent Document
1), and imaging elements are mounted in a wider variety
20 of devices. Further, a wider variety of devices have
been connected to a network such as the Internet to
perform communication. For this reason, images imaged by
an imaging device are also transmitted via the Internet
or the like and used for various services and the like
25 more frequently.
CITATION LIST
PATENT DOCUMENT
[0003]
30 Patent Document 1: International Publication No.
2016/123529
3
Patent Document 2: Japanese Translation of PCT
International Application Publication No. 2016-510910
NON-PATENT DOCUMENT
5 [0004]
Non-Patent Document 1: M. Salman Asif and four others,
“Flatcam: Replacing lenses with masks and computation”,
“2015 IEEE International Conference on Computer Vision
Workshop (ICCVW)”, 2015, pp. 663 - 666
10
SUMMARY OF THE INVENTION
PROBLEMS TO BE SOLVED BY THE INVENTION
[0005]
However, in such transmission of an image, there is
15 a risk of leak to an unauthorized user, and there has
been a risk of unauthorized use and tampering of the
image. Accordingly, for example, it is conceivable to
encrypt and then transmit image data in transmission, but
there has been a risk that a load increases due to
20 processes of encryption and the like, thereby increasing
power consumption, processing time, cost, and the like.
[0006]
The present disclosure has been made in view of
such a situation, and makes it possible to suppress
25 unauthorized use and tampering of an image more easily.
SOLUTIONS TO PROBLEMS
[0007]
An imaging device according to one aspect of the
30 present technology is an imaging device including an
imaging element that includes a plurality of pixel output
4
units that receives incident light entering without
passing through either an imaging lens or a pinhole, and
that each outputs one detection signal indicating an
output pixel value modulated by an incident angle of the
incident light, and an output processing 5 unit that
outputs a detection image formed by a detection signal
obtained in the pixel output units of the imaging
element, without associating with a restoration matrix
including coefficients used when a restored image is
10 restored from the detection image.
[0008]
The plurality of pixel output units can have a
configuration in which an incident angle directivity
indicating a directivity of the output pixel value with
15 respect to an incident angle of incident light from an
object is independently settable in each of the pixel
output units.
[0009]
The plurality of pixel output units can have a
20 configuration in which an incident angle directivity
indicating a directivity with respect to an incident
angle of incident light from an object is independently
settable in each of the pixel output units.
[0010]
25 The plurality of pixel output units can have a
configuration in which, by having different photodiodes
(PD) from each other that contribute to output, an
incident angle directivity indicating a directivity of
the output pixel value with respect to an incident angle
30 of incident light from an object is independently
settable in each of the pixel output units.
5
[0011]
A readout control unit that controls reading out of
the detection signal from each of the pixel output units
of the imaging element can be further included, in which
the readout control unit regularly 5 or irregularly
switches the pixel output unit from which the detection
signal is read out.
[0012]
A restoration matrix setting unit that sets the
10 restoration matrix in a case where a predetermined
condition is satisfied can be further included, in which
the output processing unit is configured to output the
restoration matrix set by the restoration matrix setting
unit.
15 [0013]
An encryption unit that encrypts the restoration
matrix set by the restoration matrix setting unit can be
further included, in which the output processing unit is
configured to output the restoration matrix encrypted by
20 the encryption unit.
[0014]
The detection image can be an image in which an
object can be visually unrecognizable, and the restored
image can be an image in which the object is visually
25 recognizable.
[0015]
A restoration unit that restores the restored image
from the detection image using the restoration matrix can
be further included, in which the output processing unit
30 is configured to display the restored image restored by
the restoration unit.
6
[0016]
An imaging method according to one aspect of the
present technology includes imaging an object by an
imaging element including a plurality of pixel output
units that receives incident light 5 entering without
passing through either an imaging lens or a pinhole, and
that each outputs one detection signal indicating an
output pixel value modulated by an incident angle of the
incident light, and outputting a detection image obtained
10 by the imaging and formed by a detection signal obtained
in the pixel output units of the imaging element, without
associating with a restoration matrix including
coefficients used when a restored image is restored from
the detection image.
15 [0017]
An image processing device according to another
aspect of the present technology is an image processing
device including a restoration unit that restores a
restored image from a detection image obtained by an
20 external device using a restoration matrix including
coefficients used when a restored image is restored from
the detection image that is obtained by imaging an object
by an imaging element including a plurality of pixel
output units and is formed by a detection signal obtained
25 in the pixel output units, the plurality of pixel output
units receiving incident light entering without passing
through either an imaging lens or a pinhole, and each
outputting one detection signal indicating an output
pixel value modulated by an incident angle of the
30 incident light.
[0018]
7
A restoration matrix setting unit that sets a
restoration matrix used to restore the restored image by
the restoration unit can be further included, in which
the restoration unit is configured to restore the
restored image from the detection 5 image using the
restoration matrix set by the restoration matrix setting
unit.
[0019]
The restoration matrix setting unit can regularly
10 or irregularly switch the restoration matrix used to
restore the restored image.
[0020]
The restoration matrix setting unit can generate
the restoration matrix used to restore the restored
15 image.
[0021]
A restoration matrix storage unit that stores
candidate restoration matrices can be further included,
in which the restoration matrix setting unit is
20 configured to set the restoration matrix used to restore
the restored image from the restoration matrices stored
in the restoration matrix storage unit.
[0022]
A restoration matrix communication unit that
25 obtains a restoration matrix from the external device by
communication can be further included, in which the
restoration matrix storage unit is configured to store
the restoration matrix obtained by the restoration matrix
communication unit.
30 [0023]
A decryption unit that decrypts an encrypted
8
restoration matrix obtained by the restoration matrix
communication unit can be further included, in which the
restoration matrix storage unit is configured to store
the restoration matrix decrypted by the decryption unit.
5 [0024]
The restoration matrix communication unit can
obtain an encrypted restoration matrix associated with
the detection image.
[0025]
10 An authentication unit that performs an
authentication process of itself for the external device
can be further included, in which the restoration matrix
communication unit is configured to obtain a restoration
matrix from the external device in a case where
15 authentication by the authentication unit is successful.
[0026]
An image processing method according to another
aspect of the present technology includes restoring a
restored image from a detection image obtained by an
20 external device using a restoration matrix including
coefficients used when a restored image is restored from
the detection image that is obtained by imaging an object
by an imaging element including a plurality of pixel
output units and is formed by a detection signal obtained
25 in the pixel output units, the plurality of pixel output
units receiving incident light entering without passing
through either an imaging lens or a pinhole, and each
outputting one detection signal indicating an output
pixel value modulated by an incident angle of the
30 incident light.
[0027]
9
In an imaging device and method according to one
aspect of the present technology, an object is imaged by
an imaging element including a plurality of pixel output
units that receives incident light entering without
passing through either an imaging lens 5 or a pinhole, and
that each outputs one detection signal indicating an
output pixel value modulated by an incident angle of the
incident light, and a detection image obtained by the
imaging and formed by a detection signal obtained in the
10 pixel output units of the imaging element is output
without associating with a restoration matrix including
coefficients used when a restored image is restored from
the detection image.
[0028]
15 In an image processing device and method according
to another aspect of the present technology, a restored
image is restored from a detection image and the restored
image that has been restored is displayed, the detection
image being obtained by an external device using a
20 restoration matrix including coefficients used when a
restored image is restored from the detection image that
is obtained by imaging an object by an imaging element
including a plurality of pixel output units and is formed
by a detection signal obtained in the pixel output units,
25 the plurality of pixel output units receiving incident
light entering without passing through either an imaging
lens or a pinhole, and each outputting one detection
signal indicating an output pixel value modulated by an
incident angle of the incident light.
30
EFFECTS OF THE INVENTION
10
[0029]
By the present technology, an object can be imaged
or an image can be processed. Further, by the present
technology, unauthorized use and tampering of an image
can be suppressed 5 more easily.
BRIEF DESCRIPTION OF DRAWINGS
[0030]
Fig. 1 is a diagram for describing an example of a
10 state of detection image transmission.
Fig. 2 is a diagram for describing an example of a
transmission medium.
Fig. 3 is a diagram for describing an example of a
transmission medium.
15 Fig. 4 is a block diagram illustrating a main
configuration example of an imaging device.
Fig. 5 is a diagram describing principles of
imaging in the imaging device to which a technology of
the present disclosure is applied.
20 Fig. 6 is a diagram describing a difference in
configuration between a conventional imaging element and
an imaging element of the present disclosure.
Fig. 7 is a view describing a first configuration
example of the imaging element.
25 Fig. 8 is a view describing a first configuration
example of the imaging element.
Fig. 9 is a diagram describing principles of
generation of an incident angle directivity.
Fig. 10 is a diagram describing changes in the
30 incident angle directivity using an on-chip lens.
Fig. 11 is a diagram describing a design of the
11
incident angle directivity.
Fig. 12 is a diagram describing a relationship
between an object distance and a coefficient expressing
the incident angle directivity.
Fig. 13 is a diagram describing 5 a relationship
between a narrow angle-of-view pixel and a wide angle-ofview
pixel.
Fig. 14 is a diagram describing the relationship
between the narrow angle-of-view pixel and the wide
10 angle-of-view pixel.
Fig. 15 is a diagram describing the relationship
between the narrow angle-of-view pixel and the wide
angle-of-view pixel.
Fig. 16 is a diagram describing a modification
15 example.
Fig. 17 is a diagram describing a modification
example.
Fig. 18 is a diagram describing the modification
example.
20 Fig. 19 is a diagram describing an example in which
an angle of view is changed by applying the modification
example.
Fig. 20 is a diagram describing an example of
combining pixels having a plurality of angles of view
25 when an angle of view is changed by applying a
modification example.
Fig. 21 is a diagram describing a modification
example.
Fig. 22 is a diagram describing a reason why a
30 calculation amount and a memory capacity are reduced by
providing rules for a light-shielding range in each of a
12
horizontal direction and a vertical direction.
Fig. 23 is a diagram describing a reason why the
calculation amount and the memory capacity are reduced by
providing rules for the light-shielding range in each of
the horizontal direction and the vertical 5 direction.
Fig. 24 is a diagram describing a reason why the
calculation amount and the memory capacity are reduced by
providing rules for the light-shielding range in each of
the horizontal direction and the vertical direction.
10 Fig. 25 is a diagram describing a reason why the
calculation amount and the memory capacity are reduced by
providing rules for the light-shielding range in each of
the horizontal direction and the vertical direction.
Fig. 26 is a diagram describing a modification
15 example.
Fig. 27 is a diagram describing a modification
example.
Fig. 28 is a flowchart describing an example of
flow of an imaging process.
20 Fig. 29 is a block diagram illustrating a main
configuration example of an image output device.
Fig. 30 is a flowchart describing an example of
flow of an image output process.
Fig. 31 is a diagram for describing an example of a
25 state of transmission of a detection image.
Fig. 32 is a diagram for describing an example of a
state of transmission of a detection image.
Fig. 33 is a block diagram illustrating a main
configuration example of the imaging device.
30 Fig. 34 is a flowchart describing an example of
flow of an imaging process.
13
Fig. 35 is a block diagram illustrating a main
configuration example of the image output device.
Fig. 36 is a flowchart describing an example of
flow of an image output process.
Fig. 37 is a block diagram illustrating 5 a main
configuration example of the imaging device.
Fig. 38 is a flowchart describing an example of
flow of a restoration matrix providing process.
Fig. 39 is a block diagram illustrating a main
10 configuration example of the image output device.
Fig. 40 is a flowchart describing an example of
flow of a restoration matrix obtaining process.
Fig. 41 is a block diagram illustrating a main
configuration example of the imaging device.
15 Fig. 42 is a flowchart describing an example of
flow of an imaging process.
Fig. 43 is a block diagram illustrating a main
configuration example of the image output device.
Fig. 44 is a flowchart describing an example of
20 flow of an image output process.
Fig. 45 is a diagram for describing an example of a
state of transmission of a detection image and a
restoration matrix via a server.
Fig. 46 is a block diagram illustrating a main
25 configuration example of the server.
Fig. 47 is a flowchart describing an example of
flow of a detection image transmission process.
Fig. 48 is a flowchart describing an example of
flow of a detection image transmission process.
30 Fig. 49 is a flowchart describing an example of
flow of a restoration matrix transmission process.
14
Fig. 50 is a flowchart describing an example of
flow of a restoration matrix transmission process.
Fig. 51 is a block diagram illustrating a main
configuration example of the imaging device.
Fig. 52 is a flowchart describing 5 an example of
flow of an imaging process.
Fig. 53 is a flowchart describing an example of
flow of an image display process.
Fig. 54 is a diagram for describing an example of a
10 state of transmission of a detection image in a case of a
color image.
Fig. 55 is a diagram for describing a use case of a
system.
Fig. 56 is a diagram for describing a use case of
15 the system.
Fig. 57 is a diagram for describing a use case of
the system.
Fig. 58 is a diagram for describing a use case of
the system.
20 Fig. 59 is a view illustrating a main configuration
example of the imaging element.
Fig. 60 is a diagram illustrating a case where a
black-and-white pattern mask is used.
Fig. 61 is a diagram illustrating a case where an
25 optical interference mask is used.
Fig. 62 is a diagram illustrating a modification
example of the imaging element.
MODE FOR CARRYING OUT THE INVENTION
30 [0031]
Hereinafter, modes for carrying out the present
15
disclosure (hereinafter referred to as embodiments) will
be described. Note that the description will be made in
the following order.
1. First embodiment (imaging display system: unique
5 pattern)
2. Second embodiment (imaging display system:
variable pattern)
3. Third embodiment (imaging display system:
restoration matrix provision)
10 4. Fourth embodiment (imaging display system:
association)
5. Fifth embodiment (imaging display system:
transmission via server)
6. Sixth embodiment (imaging device: self15
restoration)
7. Seventh embodiment (imaging display system:
color image)
8. Eighth embodiment (imaging display system: use
case)
20 9. Ninth embodiment (another configuration example
of imaging element, imaging device, and image output
device)
10. Others
[0032]
25 <1. First Embodiment>

In recent years, for example, an imaging element
that does not use an imaging lens as described in, for
example, Patent Document 1 has been considered. Use of
30 such an imaging element eliminates the need for an
imaging lens (makes a device imaging lens-free), and thus
16
an imaging device can be reduced in size and can be
mounted on a wider variety of devices.
[0033]
Further, in recent years, more various devices have
been connected to a network such as 5 the Internet for
communication. For this reason, images imaged by an
imaging device are also transmitted via the Internet or
the like and used for various services and the like more
frequently.
10 [0034]
However, in such transmission of an image, there is
a risk of leak to an unauthorized user, and there has
been a risk of unauthorized use and tampering of the
image. Accordingly, for example, it is conceivable to
15 encrypt and then transmit image data in transmission, but
there has been a risk that a load increases due to
processes of encryption and the like, thereby increasing
power consumption, processing time, cost, and the like.
For example, in a case of a device driven by a battery,
20 there has been a possibility that a continuous driving
time is reduced due to an increase in power consumption.
Further, for example, in a low-performance device or the
like, there has been a concern that the load exceeds
processing capacity of the device, and encryption cannot
25 be implemented.
[0035]
Accordingly, an object is imaged by an imaging
element including a plurality of pixel output units that
receives incident light entering without passing through
30 either an imaging lens or a pinhole, and that each
outputs one detection signal indicating an output pixel
17
value modulated by an incident angle of the incident
light, and a detection image obtained by the imaging and
formed by a detection signal obtained in the pixel output
units of the imaging element is transmitted. In
particular, the detection image is transmitted 5 without
being associated with a restoration matrix including
coefficients used when a restored image is restored from
the detection image.
[0036]
10 Fig. 1 illustrates an overview thereof. Although
details will be described later, a light-shielding film
of a mask pattern A is formed on a pixel array of an
imaging element 11-1 on an imaging side, and thus each
pixel output unit has “a characteristic with respect to
15 an incident angle of incident light from an object (also
referred to as incident angle directivity)”. Thus, the
imaging element 11-1 can image an object without an
imaging lens. However, a detection image obtained by the
imaging is an image in which an object is visually
20 unrecognizable. Then, this detection image in which an
object is visually unrecognizable is transmitted.
[0037]
This detection image can be converted into an
imaged image (restored image) in which the object is
25 visually recognizable by performing a predetermined
calculation. More specifically, the restored image can
be restored by multiplying the detection image by the
restored image.
[0038]
30 That is, in a restoration unit 12-1 on a receiving
side, a transmitted detection image A is obtained and the
18
detection image A is multiplied by a restoration matrix
A, thereby obtaining a restored image A. The restored
image is an image in which the object is visually
recognizable, and thus the restored image A is displayed
on a display unit 13. Thus, the imaged 5 image imaged by
the imaging element 11-1 can be transmitted from the
imaging side to a display side and displayed on the
display unit 13.
[0039]
10 In the transmission, the detection image A in which
the object is visually unrecognizable is transmitted, it
is possible to suppress leak of the imaged image to an
unauthorized user or the like during the transmission.
That is, unauthorized use and tampering of an image can
15 be suppressed.
[0040]
In addition, since only the detection image
obtained by imaging is transmitted as it is on the
imaging side, processing such as encryption is
20 unnecessary, and an increase in load can be suppressed.
That is, unauthorized use and tampering of an image can
be suppressed more easily.
[0041]
Further, in addition, since the detection image A
25 is an image generated under an influence of the mask
pattern A, the detection image A cannot be converted into
a restored image A in which the object is visually
recognizable unless the restoration matrix A that is a
coefficient matrix (corresponding to the mask pattern A)
30 reflecting the influence of this mask pattern A is used.
That is, unless a correct restoration matrix is used, the
19
detection image cannot be converted into an image in
which the object is visually recognizable. Therefore,
even if an unauthorized user has a restoration matrix,
the restored image cannot be restored correctly unless
the restoration matrix correctly 5 corresponds to the
detection image (the mask pattern of the imaging
element).
[0042]
For example, it is assumed that a light-shielding
10 film of a mask pattern B, which is a pattern different
from the mask pattern A, is formed on an imaging element
11-2. In the imaging element 11-2, a detection image B is
formed by imaging under an influence of the mask pattern
B.
15 [0043]
A restoration unit 12-2 on the receiving side has a
restoration matrix B corresponding to the mask pattern B,
and can obtain a restored image B by obtaining a
transmitted detection image B and multiplying the
20 detection image B by the restoration matrix B.
Therefore, this restored image B is displayed on the
display unit 13. Thus, similarly to the case of the
restoration unit 12-1, the imaged image imaged by the
imaging element 11-2 can be transmitted from the imaging
25 side to the display side and displayed on the display
unit 13.
[0044]
However, even if a restoration unit 12-3 having the
restoration matrix B similarly to the restoration unit
30 12-2 obtains the detection image A, the restoration unit
12-3 cannot perform correct restoration with this
20
restoration matrix B, and an image in which the object is
visually recognizable cannot be displayed on the display
unit 13.
[0045]
In this manner, unauthorized use 5 and tampering of
an image can be suppressed.
[0046]
Note that the mask pattern of the imaging element
may be different for each individual. That is, in
10 manufacturing, the imaging element may be manufactured by
changing the mask pattern for each individual. In this
manner, possibility that a restoration matrix will leak
can be reduced.
[0047]
15
Note that a transmission path for a detection image
is arbitrary. For example, as illustrated in A of Fig.
2, the detection image may be transmitted via a network.
Assuming that an imaging side device is an imaging device
20 101 and a display side device is an image output device
102, the imaging device 101 and the image output device
102 are each connected to the network 103 in a
communicable manner, as illustrated in A of Fig. 2.
[0048]
25 The network 103 is an arbitrary communication
network and may be a wired communication network or a
wireless communication network, or may be constituted of
both of them. Further, the network 103 may be
constituted of one communication network, or may be
30 constituted of a plurality of communication networks.
For example, the network 103 may include a communication
21
network and a communication path of arbitrary
communication standards such as the Internet, a public
telephone line network, a wide area communication network
for wireless mobile terminals such as what is called 3G
network and 4G network, a wide area 5 network (WAN), a
local area network (LAN), a wireless communication
network for performing communication conforming to
Bluetooth (registered trademark) standards, a
communication path of a short-range wireless
10 communication such as a near field communication (NFC), a
communication path of an infrared communication, a
communication network of wired communication conforming
to standards such as High-Definition Multimedia Interface
(HDMI) (registered trademark) and universal serial bus
15 (USB).
[0049]
The imaging device 101 and the image output device
102 may communicate via such a network 103, and the
detection image may be transmitted by the communication.
20 [0050]
Further, for example, as illustrated in B of Fig.
2, the detection image may be transmitted via a
predetermined cable. In a case of B of Fig. 2, the
imaging device 101 and the image output device 102 are
25 connected by, for example, a cable 104 with a
predetermined standard such as HDMI (registered
trademark) or USB. The imaging device 101 and the image
output device 102 may transmit the detection image via
such a cable 104.
30 [0051]
Further, for example, as illustrated in C of Fig.
22
2, the detection image may be transmitted via a recording
medium. That is, the imaging device 101 may record a
detection image on a recording medium 105 attached to
itself, the recording medium 105 may be attached to the
image output device 102, and the image 5 output device 102
may read out the detection image from the recording
medium 105 attached to itself.
[0052]
Further, for example, as illustrated in Fig. 3, a
10 detection image may be transmitted via a server. In a
case of an example in Fig. 3, a server 106 is connected
to the network 103, and the imaging device 101 and the
image output device 102 exchange detection images via the
server 106. That is, the imaging device 101 uploads a
15 detection image obtained by imaging to the server 106.
The image output device 102 downloads the detection image
from the server 106.
[0053]
Of course, the transmission method (transmission
20 path) for detection images is arbitrary and is not
limited to the above example.
[0054]

Next, an imaging device that generates a detection
25 image will be described. Fig. 4 is a diagram
illustrating a main configuration example of an imaging
device that is an embodiment of an imaging device to
which the present technology is applied. An imaging
device 101 illustrated in Fig. 4 is a device that images
30 an object and obtains electronic data related to the
imaged image.
23
[0055]
As illustrated in Fig. 4, the imaging device 101
has a control unit 110, an input unit 111, an output unit
112, a storage unit 113, a communication unit 114, a
recording-reproduction unit 115, an imaging 5 element 121,
a sensor unit 122, and the like. Each processing unit
and the like are connected via a bus 120 and can exchange
information, instructions, and the like with each other.
[0056]
10 The control unit 110 is configured to perform
processing related to control of each processing unit and
the like in the imaging device 101. For example, the
control unit 110 has a central processing unit (CPU), a
read only memory (ROM), a random access memory (RAM), and
15 the like, and performs the above-described processing by
executing a program using the CPU and the like.
[0057]
The input unit 111 is configured to perform
processing related to input of information. For example,
20 the input unit 111 has input devices such as an operating
button, a dial, a switch, a touch panel, a remote
controller, and a sensor, and an external input terminal.
For example, the input unit 111 receives an instruction
(information corresponding to an input operation) from
25 the outside by a user or the like with these input
devices. Further, for example, the input unit 111
obtains arbitrary information (program, command, data,
and the like) supplied from an external device via the
external input terminal. Further, for example, the input
30 unit 111 supplies the received information (obtained
information) to another processing unit or the like via
24
the bus 120.
[0058]
Note that the sensor included in the input unit 111
may be, for example, any sensor such as an acceleration
sensor as long as it is capable 5 of receiving an
instruction from the outside by the user or the like.
Further, the input device which the input unit 111 has is
arbitrary, and the number thereof is also arbitrary. The
input unit 111 may have a plurality of types of input
10 devices. For example, the input unit 111 may have a part
of the above-described examples, or may have all of them.
Further, the input unit 111 may have an input device
other than the examples described above. Moreover, for
example, the input unit 111 may obtain control
15 information of itself (input device or the like) supplied
via the bus 120 and drive on the basis of the control
information.
[0059]
The output unit 112 is configured to perform
20 processing related to output of information. For
example, the output unit 112 has an image display device
such as a monitor, an image projection device such as a
projector, an audio output device such as a speaker, an
external output terminal, and the like. For example, the
25 output unit 112 outputs information supplied from another
processing unit or the like via the bus 120 using these
output devices or the like. For example, the output unit
112 displays an arbitrary image of a graphical user
interface (GUI) or the like on a monitor for example,
30 projects the arbitrary image of the GUI or the like from
a projector for example, outputs audio (for example,
25
audio corresponding to an input operation, a processing
result, or the like), or outputs arbitrary information
(program, command, data, or the like) to the outside
(another device).
5 [0060]
Note that the output device and the like which the
output unit 112 has are arbitrary, and the number thereof
is also arbitrary. The output unit 112 may have a
plurality of types of output devices and the like. For
10 example, the output unit 112 may have a part of the
above-described examples, or may have all of them.
Further, the output unit 112 may have an output device
and the like other than the examples described above.
Furthermore, for example, the output unit 112 may obtain
15 control information of itself (output device or the like)
supplied via the bus 120 and drive on the basis of the
control information.
[0061]
The storage unit 113 is configured to perform
20 processing related to storage of information. For
example, the storage unit 113 has an arbitrary storage
medium such as a hard disk or a semiconductor memory.
For example, the storage unit 113 stores information
(program, command, data, or the like) supplied from other
25 processing units and the like via the bus 120 in the
storage medium. Further, the storage unit 113 may store
arbitrary information (program, command, data, or the
like) at a time of shipment. Furthermore, the storage
unit 113 reads out information stored in the storage
30 medium at an arbitrary timing or in response to a request
from another processing unit or the like, and supplies
26
the read-out information to another processing unit or
the like via the bus 120.
[0062]
Note that the storage medium included in the
storage unit 113 is arbitrary, and the 5 number thereof is
also arbitrary. The storage unit 113 may have a
plurality of types of storage media. For example, the
storage unit 113 may have a part of the examples of the
storage medium described above, or may have all of them.
10 Further, the storage unit 113 may have a storage medium
and the like other than the examples described above.
Further, for example, the storage unit 113 may obtain
control information of itself supplied via the bus 120
and drive on the basis of the control information.
15 [0063]
The communication unit 114 is configured to perform
processing related to communication with other devices.
For example, the communication unit 114 has a
communication device that performs communication for
20 exchanging information such as programs and data with an
external device via a predetermined communication medium
(for example, an arbitrary network such as the Internet).
For example, the communication unit 114 communicates with
other devices, and supplies information (program,
25 command, data, or the like) supplied from another
processing unit or the like via the bus 120 to another
device that is a communication partner thereof. Further,
for example, the communication unit 114 communicates with
other devices, obtains information supplied from another
30 device that is a communication partner thereof, and
supplies the information to another processing unit or
27
the like via the bus 120.
[0064]
The communication device which the communication
unit 114 has may be any device. For example, the
communication device may be a network 5 interface. A
communication method and a communication standard are
arbitrary. For example, the communication unit 114 may
be capable of performing wired communication, wireless
communication, or both of them. Further, for example,
10 the communication unit 114 may obtain control information
of itself (communication device or the like) supplied via
the bus 120 and drive on the basis of the control
information.
[0065]
15 The recording-reproduction unit 115 is configured
to perform processing related to recording and
reproduction of information using the recording medium
116 attached to itself. For example, the recordingreproduction
unit 115 reads out information (program,
20 command, data, or the like) recorded on the recording
medium 116 attached to itself, and supplies the
information to another processing unit or the like via
the bus 120. Further, for example, the recordingreproduction
unit 115 obtains information supplied from
25 another processing unit or the like via the bus 120, and
writes (records) the information in (on) the recording
medium 116 attached to itself. Note that, for example,
the recording-reproduction unit 115 may obtain control
information of itself supplied via the bus 120 and drive
30 on the basis of the control information.
[0066]
28
Note that the recording medium 116 may be any type.
For example, the recording medium 116 may be a magnetic
disk, an optical disk, a magneto-optical disk, a
semiconductor memory, or the like.
5 [0067]
The imaging element 121 is configured to perform
processing related to imaging of an object. For example,
the imaging element 121 images an object and obtains data
(electronic data) related to the imaged image. At that
10 time, the imaging element 121 can image an object without
using an imaging lens, an optical filter or the like such
as a diffraction grating, a pinhole, or the like, and
obtain data related to the imaged image. For example,
the imaging element 121 images an object and obtains data
15 (detection signal or the like) that allows obtaining of
data of the imaged image by a predetermined calculation.
[0068]
Note that the imaged image is an image that is
formed by pixel values by which an image of an object is
20 formed and is visually recognizable as an image by the
user. On the other hand, an image (referred to as a
detection image) formed by a detection signal that is a
detection result of incident light in a pixel unit output
of the imaging element 121 is an image that cannot be
25 visually recognized as an image by the user because an
image of an object is not formed (that is, the object is
visually unrecognizable). That is, the detection image
is an image different from the imaged image. However, as
described above, by performing a predetermined
30 calculation on data of the detection image, the imaged
image, that is, an image in which the image of the object
29
is formed and which is visually recognizable as an image
by the user (that is, the object is visually
recognizable) can be restored. This restored imaged
image is referred to as a restored image. That is, the
detection image is an image different 5 from the restored
image.
[0069]
Note that an image that constitutes the restored
image and that has not yet been subjected to a
10 synchronization process, a color separation process, or
the like (for example, a demosaic process or the like) is
referred to as a raw image. Similarly to the imaged
image, this raw image is also an image that is visually
recognizable as an image by the user (that is, the object
15 is visually recognizable). In other words, the detection
image is an image according to an array of a color
filter, but is an image different from the raw image.
[0070]
However, in a case where the imaging element 121
20 has sensitivity only to, for example, invisible light
such as infrared light and ultraviolet light, the
restored image (raw image or imaged image) also becomes
an image that is not visually recognizable as an image
(the object is not visually recognizable) by the user.
25 However, since this is due to a wavelength range of
detected light, the restored image can be an image in
which the object is visually recognizable by converting
the wavelength range to a visible light range. On the
other hand, since the image of the object is not formed,
30 the detection image cannot become an image in which the
object is visually recognizable by just converting the
30
wavelength range. Therefore, even in a case where the
imaging element 121 has sensitivity only to invisible
light, an image obtained by performing a predetermined
calculation on the detection image as described above is
referred to as a restored image. 5 Note that in the
following, the present technology will be described
basically using a case where the imaging element 121
receives visible light as an example unless otherwise
specified.
10 [0071]
That is, the imaging element 121 can image an
object and obtain data related to the detection image.
The imaging element 121 supplies data related to the
detection image to other processing units (for example,
15 the output unit 112, the storage unit 113, the
communication unit 114, the recording-reproduction unit
115, and the like) or the like via the bus 120. Further,
for example, the imaging element 121 may obtain control
information of itself supplied via the bus 120 and drive
20 on the basis of the control information.
[0072]
The sensor unit 122 is configured to perform
processing related to detection. For example, the sensor
unit 122 has an arbitrary sensor and performs detection
25 for a predetermined parameter. For example, the sensor
unit 122 detects a parameter related to a state around
the imaging device 101, a parameter related to a state of
the imaging device 101, and the like. For example, the
sensor unit 122 performs detection for a parameter
30 related to a state of the imaging element 121. Further,
for example, the sensor unit 122 supplies detected
31
information to another processing unit or the like via
the bus 120. Note that, for example, the sensor unit 122
may obtain control information of itself supplied via the
bus 120 and drive on the basis of the control
5 information.
[0073]
Note that a part or all of the output unit 112, the
storage unit 113, the communication unit 114, and the
recording-reproduction unit 115 may be integrated into an
10 output processing unit 123. The output processing unit
123 is configured to perform processing related to output
of the detection image. The output processing unit 123
may be implemented by any physical component. For
example, the output processing unit 123 may be
15 implemented as a processor such as a system large scale
integration (LSI). Further, the output processing unit
123 may be implemented as, for example, a module using a
plurality of processors or the like, a unit using a
plurality of modules or the like, or a set obtained by
20 further adding other functions to a unit, or the like
(that is, a part of the configuration of the device).
Further, the output processing unit 123 may be
implemented as a device.
[0074]
25
Next, the imaging element 121 will be described
with reference to Figs. 5 to 27.
[0075]

30 In the present description, the term “pixel” (or
“pixel output unit”) is used to describe the present
32
technology. In the present description, a “pixel” (or
“pixel output unit”) refers to a division unit, including
at least one physical component that can receive light
independently from other pixels, of an area (also
referred to as a pixel area) in 5 which a physical
component for receiving incident light of the imaging
element 121 is formed. The physical component capable of
receiving light is, for example, a photoelectric
conversion element, or for example, a photodiode (PD).
10 The number of physical components (for example,
photodiodes) formed in one pixel is arbitrary, and may be
singular or plural. The type, size, shape, and the like
thereof are also arbitrary.
[0076]
15 Further, the physical component of this “pixel”
unit includes not only the above-described “physical
component capable of receiving light”, but also includes,
for example, all physical components related to reception
of incident light, such as an on-chip lens, a light20
shielding film, a color filter, a planarization film, and
an antireflection film. Furthermore, a component such as
a readout circuit may also be included. That is, the
physical component of this pixel unit may be any
component.
25 [0077]
Further, a detection signal read out from a “pixel”
(that is, a physical component of a pixel unit) may also
be referred to as a “detection signal of a pixel unit (or
pixel output unit)” or the like. Moreover, the detection
30 signal of this pixel unit (or pixel output unit) is also
referred to as a “pixel unit detection signal (or pixel
33
output unit detection signal)”. Further, this pixel unit
detection signal is also referred to as a “pixel output”.
Moreover, the value is also referred to as an “output
pixel value”.
5 [0078]
The value of the detection signal (output pixel
value) in pixel units of the imaging element 121 can have
an incident angle directivity indicating a directivity
with respect to the incident angle of incident light from
10 the object independently from the others. That is, each
pixel unit (pixel output unit) of the imaging element 121
has a configuration in which the incident angle
directivity indicating a directivity of the output pixel
value with respect to the incident angle of incident
15 light from the object is independently settable. For
example, in the imaging element 121, output pixel values
of at least two pixel units can have different incident
angle directivities indicating directivities with respect
to the incident angle of incident light from the object.
20 [0079]
Note that since the number of “physical components
capable of receiving light” included in a “pixel (or
pixel output unit)” is arbitrary as described above, the
pixel unit detection signal may be a detection signal
25 obtained by a single “physical component capable of
receiving light” or may be detection signals obtained by
a plurality of “physical components capable of receiving
light”.
[0080]
30 Further, a plurality of pixel unit detection
signals (output pixel values) can be combined into one at
34
an arbitrary stage. For example, output pixel values of
a plurality of pixels may be added in the state of an
analog signal, or may be added after being converted into
a digital signal.
5 [0081]
Further, after this detection signal is read out
from the imaging element 121, that is, in the detection
image, a plurality of detection signals can be combined
into a single detection signal or a single detection
10 signal can be converted into a plurality of detection
signals. That is, resolution (the number of pieces of
data) of the detection image is variable.
[0082]
Incidentally, for convenience of description below,
15 unless otherwise specified, the imaging element 121 will
be described as having a pixel area in which a plurality
of pixels is disposed in a matrix (a pixel array is
formed). Note that an arrangement pattern of pixels (or
pixel output units) of the imaging element 121 is
20 arbitrary, and is not limited to this example. For
example, the pixels (or pixel output units) may be
disposed in a honeycomb structure. Further, for example,
the pixels (or pixel output units) may be disposed in one
row (or one column). That is, the imaging element 121
25 may be a line sensor.
[0083]
Note that a wavelength range in which the imaging
element 121 (pixels thereof) has sensitivity is
arbitrary. For example, the imaging element 121 (pixels
30 thereof) may be sensitive to visible light, may be
sensitive to invisible light such as infrared light and
35
ultraviolet light, or may be sensitive to both visible
light and invisible light. For example, in a case where
the imaging element detects far-infrared light that is
invisible light, a thermograph (an image representing a
heat distribution) can be generated using 5 an imaged image
obtained in the imaging element. However, in a case of
an imaging element with an imaging lens, since it is
difficult for glass to transmit far-infrared light, an
imaging lens including an expensive special material is
10 necessary, which may increase manufacturing cost. Since
the imaging element 121 can image an object without using
an imaging lens or the like and obtain data related to
the imaged image, by making the pixels thereof capable of
detecting far-infrared light, an increase in
15 manufacturing cost can be suppressed. That is, farinfrared
light can be imaged at lower cost (a thermograph
can be obtained at lower cost). Note that in a case
where the imaging element 121 (pixels thereof) has
sensitivity to invisible light, the restored image is not
20 an image in which the user can visually recognize the
object but is an image in which the user cannot visually
recognize the object. In other words, the restored image
may be an image of visible light or an image of invisible
light (for example, (far) infrared light, ultraviolet
25 light, and the like).
[0084]

The imaging element 121 has a plurality of pixel
output units that receives incident light entering
30 without passing through either an imaging lens or a
pinhole, and that each outputs one detection signal
36
indicating an output pixel value modulated by an incident
angle of the incident light. For example, the imaging
element 121 has a configuration such that incident angle
directivities, indicating directivities with respect to
the incident angle of incident light 5 from an object, of
output pixel values of at least two pixel output units
out of the plurality of pixel output units are different
characteristics from each other. That is, in this case,
the imaging element 121 can obtain detection signals for
10 a plurality of pixel output units (a plurality of pixel
output unit detection signals), and the incident angle
directivities, indicating directivities with respect to
the incident angle of incident light from an object, of
at least two pixel output unit detection signals thereof
15 are different from each other.
[0085]
Here, the “incident angle directivity” refers to a
light receiving sensitivity characteristic corresponding
to an incident angle of incident light, that is,
20 detection sensitivity with respect to the incident angle
of the incident light. For example, even if incident
light has the same light intensity, the detection
sensitivity may change depending on the incident angle
thereof. Such a deviation in detection sensitivity
25 (including a case where there is no deviation) will be
referred to as an “incident angle directivity”.
[0086]
For example, in a case where incident light having
the same light intensities as each other are incident on
30 physical components of two pixel output units thereof at
the same incident angles as each other, signal levels
37
(detection signal levels) of detection signals of the
pixel output units can have different values from each
other. The imaging element 121 (each pixel output unit)
has physical components having such characteristics.
5 [0087]
This incident angle directivity may be achieved by
any method. For example, the incident angle directivity
may be achieved by providing a light-shielding film, or
the like in front (light incident side) of a
10 photoelectric conversion element (photodiode or the like)
of an imaging element having a basic structure similar to
that including a common imaging element such as a
complementary metal oxide semiconductor (CMOS) image
sensor for example.
15 [0088]
If imaging is performed only with a general imaging
element including pixels having the same incident angle
directivities, light with substantially the same light
intensity is incident on all pixels of the imaging
20 element, and an image of an image-formed object cannot be
obtained. Accordingly, in general, an imaging lens or a
pinhole is provided in front of the imaging element (on
the light incident side). For example, by providing an
imaging lens, light from an object plane can be image25
formed on the imaging surface of the imaging element.
Therefore, the imaging element can obtain a detection
signal with a level corresponding to the image of the
image-formed object at each pixel (that is, an imaged
image of the imaged object can be obtained). However, in
30 this case, the size is physically increased, and it may
be difficult to reduce the size of the device. Further,
38
in a case where a pinhole is provided, although the size
can be reduced as compared with a case where an imaging
lens is provided, measures such as increasing an exposure
time or increasing gain are necessary because the amount
of light incident on the imaging element 5 is reduced, and
there has been a possibility that blurring is liable to
occur in high-speed imaging of an object, or color
expression is not natural.
[0089]
10 On the other hand, the imaging element 121 has
incident angle directivities in which the detection
sensitivities of the pixels are different from each
other, as illustrated in an upper left part of Fig. 5.
That is, the light receiving sensitivity characteristic
15 corresponding to an incident angle of incident light is
different for each pixel. However, it is not necessary
that the light receiving sensitivity characteristics of
all the pixels are completely different, and part of
pixels may include pixels having the same light receiving
20 sensitivity characteristics and part of pixels may have
different light receiving sensitivity characteristics.
[0090]
In a case where it is assumed that a light source
constituting an object plane 131 is a point light source,
25 in the imaging element 121, a light beam with the same
light intensity emitted from the same point light source
is incident on all the pixels, but is incident at a
different incident angle on every pixel. Then, since
respective pixels of the imaging element 121 have
30 different incident angle directivities from each other,
the light beam with the same light intensity is detected
39
by different sensitivity from each other. That is, a
detection signal with a different signal level is
detected for each pixel.
[0091]
More specifically, the sensitivity 5 characteristic
according to the incident angle of the incident light
received at each pixel of the imaging element 121, that
is, the incident angle directivity according to the
incident angle at each pixel is expressed by a
10 coefficient representing light receiving sensitivity
according to the incident angle, and the signal level of
the detection signal according to the incident light in
each pixel (also referred to as a detection signal level)
can be obtained by multiplication by a coefficient that
15 is set corresponding to the light receiving sensitivity
according to the incident angle of the incident light.
[0092]
More specifically, as illustrated in the upper left
part of Fig. 5, detection signal levels DA, DB, DC at
20 positions Pa, Pb, Pc are expressed by the following
equations (1) to (3), respectively.
[0093]
DA = α1 × a + β1 × b + γ1 × c
... (1)
25 DB = α2 × a + β2 × b + γ2 × c
... (2)
DC = α3 × a + β3 × b + γ3 × c
... (3)
[0094]
30 Here, α1 is a coefficient that is set according to
an incident angle of a light beam from a point light
40
source PA on the object plane 131 to be restored at the
position Pa on the imaging element 121. β1 is a
coefficient that is set according to an incident angle of
a light beam from a point light source PB on the object
plane 131 to be restored at the 5 position Pa on the
imaging element 121. γ1 is a coefficient that is set
according to an incident angle of a light beam from a
point light source PC on the object plane 131 to be
restored at the position Pa on the imaging element 121.
10 [0095]
As described in equation (1), the detection signal
level DA at the position Pa is expressed by a sum
(composite value) of a product of a light intensity “a”
of the light beam from the point light source PA at the
15 position Pa and the coefficient α1, a product of a light
intensity “b” of the light beam from the point light
source PB at the position Pa and the coefficient β1, and
a product of a light intensity “c” of the light beam from
the point light source PC at the position Pa and the
20 coefficient γ1. Hereinafter, coefficients αx, βx, γx (x
is a natural number) are collectively referred to as a
coefficient set.
[0096]
Similarly, a coefficient set α2, β2, γ2 of equation
25 (2) is a coefficient set that is set according to the
incident angles of the light beams from the point light
sources PA, PB, PC on the object plane 131 to be restored
at the position Pb on the imaging element 121. That is,
as in the above equation (2), the detection signal level
30 DB at the position Pb is expressed by a sum (composite
value) of a product of a light intensity “a” of the light
41
beam from the point light source PA at the position Pb
and the coefficient α2, a product of a light intensity
“b” of the light beam from the point light source PB at
the position Pb and the coefficient β2, and a product of
a light intensity “c” of the light beam 5 from the point
light source PC at the position Pb and the coefficient
γ2. Further, a coefficient set α3, β3, γ3 of equation
(3) is a coefficient set that is set according to the
incident angles of the light beams from the point light
10 sources PA, PB, PC on the object plane 131 to be restored
at the position Pc on the imaging element 121. That is,
as in the above equation (3), the detection signal level
DC at the position Pc is expressed by a sum (composite
value) of a product of a light intensity “a” of the light
15 beam from the point light source PA at the position Pc
and the coefficient α3, a product of a light intensity
“b” of the light beam from the point light source PB at
the position Pc and the coefficient β3, and a product of
a light intensity “c” of the light beam from the point
20 light source PC at the position Pc and the coefficient
γ3.
[0097]
As described above, these detection signal levels
are different from those in which an image of an object
25 is formed because light intensities of light beams
emitted from each of the point light sources PA, PB, PC
are mixed. That is, the detection signal level
illustrated in an upper right part of Fig. 5 is not a
detection signal level corresponding to an image (imaged
30 image) on which an image of an object is formed, and
hence is different from a pixel value illustrated in a
42
lower right part of Fig. 5 (generally the both do not
match).
[0098]
However, by composing simultaneous equations using
these coefficient set α1, β1, γ1, coefficient 5 set α2, β2,
γ2, and coefficient set α3, β3, γ3 and the detection
signal levels DA, DB, DC, and by solving the simultaneous
equations of the above-described equations (1) to (3)
with a, b, c being variables, pixel values at the
10 respective positions Pa, Pb, Pc as illustrated in the
lower right part of Fig. 5 can be obtained. Thus, a
restored image that is a set of pixel values (an image in
which an image of an object is formed) is restored.
[0099]
15 With such a configuration, the imaging element 121
has incident angle directivities different from each
other in at least two pixel output units without a need
for an imaging lens, an optical filter including a
diffraction grating or the like, a pinhole, or the like.
20 Consequently, an imaging lens, an optical filter
including a diffraction grating or the like, a pinhole,
or the like is not a necessary component, and thus it
becomes possible to reduce the height of the imaging
device, that is, the thickness in an incident direction
25 of light in a configuration that achieves an imaging
function.
[0100]

A left part of Fig. 6 illustrates a front view of a
30 part of a pixel array unit of a general imaging element,
and a right part of Fig. 6 illustrates a front view of a
43
part of the pixel array unit of the imaging element 121.
Note that Fig. 6 illustrates an example of a case where
setting of the numbers of pixels in horizontal direction
× vertical direction of the pixel array unit is 6 pixels
× 6 pixels, respectively, but the setting 5 of the numbers
of pixels is not limited to this.
[0101]
The incident angle directivity can be formed by a
light-shielding film, for example. It is illustrated
10 that in a general imaging element 151, as in an example
of the left part of Fig. 6, pixels 151a having the same
incident angle directivity are arranged in an array. On
the other hand, the imaging element 121 in an example of
the right part of Fig. 6 is provided with a light15
shielding film 121b, which is one of modulation elements,
so as to cover a part of a light receiving area of a
photodiode thereof in each pixel 121a, and incident light
incident on each pixel 121a is optically modulated
according to an incident angle. Then, for example, by
20 providing the light-shielding film 121b in a different
range for each pixel 121a, light receiving sensitivity
with respect to the incident angle of incident light
differs for each pixel 121a, and each pixel 121a has a
different incident angle directivity.
25 [0102]
For example, in a pixel 121a-1 and a pixel 121a-2,
ranges of shielding pixels from light by a lightshielding
film 121b-1 and a light-shielding film 121b-2
provided are different (at least one of light-shielding
30 region (position) and light-shielding area is different).
That is, in the pixel 121a-1, the light-shielding film
44
121b-1 is provided so as to shield a left part in the
light receiving area of the photodiode from light by a
predetermined width, and in the pixel 121a-2, the lightshielding
film 121b-2 is provided so as to shield a right
part in the light receiving area from 5 light by a width
wider in a horizontal direction than the light-shielding
film 121b-1. In the other pixel 121a, similarly, the
light-shielding film 121b is provided so that a different
range in the light receiving area is shielded from light
10 in each pixel, and is randomly disposed in the pixel
array.
[0103]
Note that the range of the light-shielding film
121b is desirably set to an area that can secure a
15 desired amount of light because the larger the ratio of
covering the light receiving area of each pixel, the
smaller the quantity of light that can be received. For
example, a limitation such that the area of the lightshielding
film 121b is up to approximately 3/4 of the
20 entire range capable of receiving light may be added.
With such a configuration, it becomes possible to ensure
an amount of light equal to or more than a desired
amount. However, if each pixel is provided with a nonshielded
range having a width corresponding to the
25 wavelength of light to be received, it is possible to
receive a minimum amount of light. That is, for example,
in a case of a blue pixel (B pixel), the wavelength is
about 500 nm, and it is possible to receive a minimum
amount of light as long as light shielding thereof is not
30 equal to or larger than a width corresponding to this
wavelength.
45
[0104]

A configuration example of the imaging element 121
in this case will be described with reference to Fig. 7.
An upper part of Fig. 7 is a cross-sectional 5 side view of
the imaging element 121, and a middle part of Fig. 7 is a
top view of the imaging element 121. Further, the crosssectional
side view of the upper part of Fig. 7 is an AB
cross section in a middle part of Fig. 7. Furthermore, a
10 lower part of Fig. 7 is a circuit configuration example
of the imaging element 121.
[0105]
The imaging element 121 having the configuration
illustrated in Fig. 7 includes a plurality of pixel
15 output units that receives incident light entering
without passing through either an imaging lens or a
pinhole, and each outputs one detection signal indicating
an output pixel value modulated by the incident angle of
the incident light. For example, this imaging element
20 121 has a configuration such that incident angle
directivities, indicating directivities with respect to
the incident angle of incident light from an object, of
output pixel values of at least two pixel output units
out of the plurality of pixel output units are different
25 characteristics from each other. Further, in the imaging
element 121 in this case, a plurality of pixel output
units thereof has a configuration in which an incident
angle directivity indicating a directivity with respect
to the incident angle of incident light from an object is
30 independently settable in each of the pixel output units.
[0106]
46
In the imaging element 121 in the upper part of
Fig. 7, incident light enters from an upper side to a
lower side in the drawing. Adjacent pixels 121a-15,
121a-16 are of what is called a back-illuminated type in
which a wiring layer Z12 is provided in 5 a lowermost layer
in the drawing and a photoelectric conversion layer Z11
is provided thereon.
[0107]
Note that in a case where it is not necessary to
10 distinguish the pixels 121a-15, 121a-16, the pixels 121a-
15, 121a-16 are simply referred to as a pixel 121a, and
other components are also referred to similarly.
Further, in Fig. 7, there are a side view and a top view
for two pixels constituting a pixel array of the imaging
15 element 121, but it goes without saying that a larger
number of pixels 121a are arranged but omitted from
illustration.
[0108]
Moreover, the pixels 121a-15, 121a-16 include
20 photodiodes 121e-15, 121e-16, respectively, in the
photoelectric conversion layer Z11. Further, on the
photodiodes 121e-15, 121e-16, on-chip lenses 121c-15,
121c-16 and color filters 121d-15, 121d-16 are
respectively formed from above.
25 [0109]
The on-chip lenses 121c-15, 121c-16 collect
incident light on the photodiodes 121e-15, 121e-16.
[0110]
The color filters 121d-15, 121d-16 are, for
30 example, optical filters that transmit light with
specific wavelengths such as red, green, blue, infrared,
47
and white. Note that in a case of white, the color
filters 121d-15, 121d-16 may or may not be transparent
filters.
[0111]
In the photoelectric conversion 5 layer Z11 of the
pixels 121a-15, 121a-16, light-shielding films 121p-15 to
121p-17 are respectively formed at a boundary between the
pixels, so as to suppress crosstalk between the adjacent
pixels.
10 [0112]
Further, light-shielding films 121b-15, 121b-16,
which are one of modulation elements, shield a part of a
light receiving surface S from light as illustrated in
the upper and middle parts of Fig. 7. Since the part of
15 the light receiving surface S is shielded from light by
the light-shielding film 121b, incident light incident on
the pixel 121a is optically modulated according to an
incident angle. Since the pixel 121a detects the
optically modulated incident light, the pixel 121a has an
20 incident angle directivity. On the light receiving
surfaces S of the photodiodes 121e-15, 121e-16 in the
pixels 121a-15, 121a-16, different ranges are shielded
from light by the light-shielding films 121b-15, 121b-16,
respectively, and thus a different incident angle
25 directivity is set in every pixel. However, the lightshielded
range is not limited to a case of being
different in each of all the pixels 121a of the imaging
element 121, and there may be pixels 121a in which the
same range is shielded from light in part.
30 [0113]
With the configuration illustrated in the upper
48
part of Fig. 7, a right end portion of the lightshielding
film 121p-15 and an upper end portion of the
light-shielding film 121b-15 are connected, and a left
end portion of the light-shielding film 121b-16 and an
upper end portion of the light-shielding 5 film 121p-16 are
connected, forming an L-shape when viewed from the side.
[0114]
Moreover, the light-shielding films 121b-15 to
121b-17 and the light-shielding films 121p-15 to 121p-17
10 include metal, and include, for example, tungsten (W),
aluminum (Al), or an alloy of Al and copper (Cu).
Further, the light-shielding films 121b-15 to 121b-17 and
the light-shielding films 121p-15 to 121p-17 may be
formed using the same metal as that of wiring by the same
15 process of forming the wiring in a semiconductor process.
Note that thicknesses of the light-shielding films 121b-
15 to 121b-17 and the light-shielding films 121p-15 to
121p-17 do not have to be the same depending on the
position.
20 [0115]
Further, as illustrated in the lower part of Fig.
7, the pixel 121a includes a photodiode 161
(corresponding to the photodiode 121e), a transfer
transistor 162, a floating diffusion (FD) unit 163, a
25 selection transistor 164, an amplification transistor
165, and a reset transistor 166, and is connected to a
current source 168 via a vertical signal line 167.
[0116]
In the photodiodes 161, anode electrodes are
30 respectively grounded, and cathode electrodes are
respectively connected to gate electrodes of the
49
amplification transistors 165 via the transfer
transistors 162.
[0117]
The transfer transistors 162 are each driven
according to a transfer signal TG. For 5 example, when the
transfer signal TG supplied to a gate electrode of the
transfer transistor 162 becomes a high level, the
transfer transistor 162 is turned on. Thus, charges
accumulated in the photodiode 161 are transferred to the
10 FD unit 163 via the transfer transistor 162.
[0118]
The amplification transistor 165 is as an input
unit of a source follower that is a readout circuit that
reads out a signal obtained by photoelectric conversion
15 in the photodiode 161, and outputs a pixel signal of a
level corresponding to charges accumulated in the FD unit
163 to the vertical signal line 23. In other words, the
amplification transistor 165 has a drain terminal
connected to a power supply voltage VDD and a source
20 terminal connected to the vertical signal line 167 with
the selection transistor 164 interposed therebetween,
thereby forming a source follower with the current source
168 connected to one end of the vertical signal line 167.
[0119]
25 The floating diffusion (FD) unit 163 is a floating
diffusion region having a charge capacitance C1 provided
between the transfer transistor 162 and the amplification
transistor 165, and temporarily accumulate charges
transferred from the photodiode 161 via the transfer
30 transistor 162. The FD unit 163 is a charge detection
unit that converts charge into voltage, and the charges
50
accumulated in the FD unit 163 are converted into voltage
in the amplification transistor 165.
[0120]
The selection transistor 164 is driven according to
a selection signal SEL, and is 5 turned on when the
selection signal SEL supplied to a gate electrode becomes
a high level, and connects the amplification transistor
165 and the vertical signal line 167.
[0121]
10 The reset transistor 166 is driven according to a
reset signal RST. For example, the reset transistor 166
is turned on when the reset signal RST supplied to a gate
electrode becomes a high level, discharges the charges
accumulated in the FD unit 163 to the power supply
15 voltage VDD, and resets the FD unit 163.
[0122]
With the circuit configuration as described above,
the pixel circuit illustrated in the lower part of Fig. 7
operates as follows.
20 [0123]
That is, as a first operation, the reset transistor
166 and the transfer transistor 162 are turned on, the
charges accumulated in the FD unit 163 are discharged to
the power supply voltage VDD, and the FD unit 163 is
25 reset.
[0124]
As a second operation, the reset transistor 166 and
the transfer transistor 162 are turned off, an exposure
period is started, and charges corresponding to the
30 amount of incident light are accumulated by the
photodiode 161.
51
[0125]
As a third operation, after the reset transistor
166 is turned on and the FD unit 163 is reset, the reset
transistor 166 is turned off. By this operation, the FD
unit 163 is reset and set to a reference 5 potential.
[0126]
As a fourth operation, a potential of the FD unit
163 in a reset state is output from the amplification
transistor 165 as the reference potential.
10 [0127]
As a fifth operation, the transfer transistor 162
is turned on, and the charges accumulated in the
photodiode 161 are transferred to the FD unit 163.
[0128]
15 As a sixth operation, the potential of the FD unit
163 to which the charges of the photodiode are
transferred is output from the amplification transistor
165 as a signal potential.
[0129]
20 Through the above processing, the reference
potential is subtracted from the signal potential and is
output as a detection signal by correlated double
sampling (CDS). The value of this detection signal
(output pixel value) is modulated according to an
25 incident angle of incident light from an object, and
differs in a characteristic (directivity) depending on
the incident angle (has an incident angle directivity).
[0130]
As described above, the pixels 121a in a case of
30 Fig. 7 are each provided with one photodiode 121e, a
different range is shielded from light by the light52
shielding film 121b in each of the pixels 121a, and a
detection signal for one pixel of a detection image
having an incident angle directivity can be expressed by
one pixel 121a by an optical modulation using the lightshielding
5 film 121b.
[0131]

Further, the incident angle directivity can be
formed by, for example, a position, a size, a shape, and
10 the like in a pixel of a light receiving element (for
example, a photodiode). Pixels having different
parameters from each other have different sensitivities
to incident light having a same light intensity from a
same direction. That is, by setting these parameters for
15 each pixel, the incident angle directivity can be set for
each pixel.
[0132]
For example, a plurality of light receiving
elements (for example, photodiodes) may be provided in a
20 pixel and used selectively. In this manner, the incident
angle directivity can be set for each pixel by selecting
the light receiving elements.
[0133]
Fig. 8 is a diagram illustrating another
25 configuration example of the imaging element 121. An
upper part of Fig. 8 illustrates a cross-sectional side
view of a pixel 121a of the imaging element 121, and a
middle part of Fig. 8 illustrates a top view of the
imaging element 121. Further, the cross-sectional side
30 view of the upper part of Fig. 8 is an AB cross section
in the middle part of Fig. 8. Furthermore, a lower part
53
of Fig. 8 is a circuit configuration example of the
imaging element 121.
[0134]
The imaging element 121 having the configuration
illustrated in Fig. 8 includes a 5 plurality of pixel
output units that receives incident light entering
without passing through either an imaging lens or a
pinhole, and each outputs one detection signal indicating
an output pixel value modulated by the incident angle of
10 the incident light. For example, this imaging element
121 has a configuration such that incident angle
directivities, indicating directivities with respect to
the incident angle of incident light from an object, of
output pixel values of at least two pixel output units
15 out of the plurality of pixel output units are different
characteristics from each other. Further, in the imaging
element 121 in this case, a plurality of pixel output
units thereof has different photodiodes (PD) from each
other that contribute to output, and thus the incident
20 angle directivity indicating a directivity of the output
pixel value with respect to the incident angle of
incident light from an object is independently settable
in each of the pixel output units.
[0135]
25 As illustrated in Fig. 8, the imaging element 121
has a different configuration from the imaging element
121 of Fig. 8 in that four photodiodes 121f-1 to 121f-4
are formed in the pixel 121a, and a light-shielding film
121p is formed in a region separating the photodiodes
30 121f-1 to 121f-4 from each other. Specifically, in the
imaging element 121 of Fig. 8, the light-shielding film
54
121p is formed in a “+” shape when viewed from above.
Note that common components thereof are given the same
reference signs and detailed description will be omitted.
[0136]
In the imaging element 5 121 configured as
illustrated in Fig. 8, electrical and optical crosstalk
among the photodiodes 121f-1 to 121f-4 can be prevented
by separation into the photodiodes 121f-1 to 121f-4 by
the light-shielding film 121p. That is, the light10
shielding film 121p in Fig. 5 is for preventing crosstalk
and is not for providing an incident angle directivity,
similarly to the light-shielding film 121p of the imaging
element 121 in Fig. 4.
[0137]
15 Although details will be described later, the
photodiodes 121f-1 to 121f-4 have different incident
angles from each other at which the light receiving
sensitivity characteristics are high. That is, a desired
incident angle directivity can be given to an output
20 pixel value of the pixel 121a depending on from which of
the photodiodes 121f-1 to 121f-4 charges are read out.
That is, the incident angle directivity of the output
pixel value of the pixel 121a can be controlled.
[0138]
25 In the configuration example of the imaging element
121 in Fig. 8, one FD unit 163 is shared by four
photodiodes 121f-1 to 121f-4. The lower part of Fig. 8
illustrates a circuit configuration example in which one
FD unit 163 is shared by four photodiodes 121f-1 to 121f-
30 4. Note that in the lower part of Fig. 8, description of
the same configuration as the lower part of Fig. 7 will
55
be omitted.
[0139]
In the lower part of Fig. 8, differences from the
circuit configuration of the lower part of Fig. 7 are
that photodiodes 161-1 to 161-4 (corresponding 5 to the
photodiodes 121f-1 to 121f-4 in the upper part of Fig. 8)
and transfer transistors 162-1 to 162-4 are provided
instead of the photodiode 161 and the transfer transistor
162, and the FD unit 163 is shared.
10 [0140]
In the circuit illustrated in the lower part of
Fig. 8, in a case where it is not necessary to
distinguish the photodiodes 161-1 to 161-4 from each
other to describe them, they are referred to as
15 photodiodes 161. Further, in a case where it is not
necessary to distinguish the transfer transistors 162-1
to 162-4 from each other to describe them, they are
referred to as transfer transistors 162.
[0141]
20 In the circuit illustrated in the lower part of
Fig. 8, if any one of the transfer transistors 162 is
turned on, charges of the photodiode 161 corresponding to
the transfer transistor 162 are read out and transferred
to the common FD unit 163. Then, a signal corresponding
25 to a level of charges held in the FD unit 163 is read out
as a detection signal in a pixel output unit. That is,
charges of each photodiode 161 can be read out
independently of each other, and the photodiode 161 from
which charges are read out can be controlled by which
30 transfer transistor 162 is turned on. In other words, a
degree of contribution of each photodiode 161 to the
56
output pixel value can be controlled by which transfer
transistor 162 is turned on. For example, the
photodiodes 161 that contribute to the output pixel value
can be made different from each other by making the
photodiodes 161 from which charges are 5 read out different
from each other between at least two pixels. That is, by
selecting the photodiode 161 that reads out charges, a
desired incident angle directivity can be given to the
output pixel value of the pixel 121a. That is, the
10 detection signal output from each pixel 121a can be a
value (output pixel value) modulated according to the
incident angle of the incident light from the object.
[0142]
For example, in Fig. 8, by transferring charges of
15 the photodiode 121f-1 and the photodiode 121f-3 to the FD
unit 163 and adding signals obtained respectively by
reading out them, an incident angle directivity in a
left-right direction in the diagram can be given to the
output pixel value of the pixel 121a. Similarly, by
20 transferring charges of the photodiode 121f-1 and the
photodiode 121f-2 to the FD unit 163 and adding signals
obtained respectively by reading out them, an incident
angle directivity in an up-down direction in the diagram
can be given to the output pixel value of the pixel 121a.
25 [0143]
Note that a signal obtained on the basis of charges
of each photodiode 121f of the pixel 121a in Fig. 8 may
be added after being read out from the pixel, or may be
added within the pixel (for example, the FD unit 163).
30 [0144]
Further, the combination of the photodiodes 121f
57
for adding charges (or signals corresponding to the
charges) is arbitrary, and is not limited to the above
example. For example, charges (or signals corresponding
to the charges) of three or more photodiodes 121f may be
added. Further, for example, charges 5 of one photodiode
121f may be read out without performing addition.
[0145]
Note that a desired incident angle directivity may
be given to (a detection sensitivity of) the pixel 121a
10 by resetting a detection value (charges) accumulated in
the photodiode 161 (photodiode 121f) before reading out
charges to the FD unit 163 by using an electronic shutter
function, or the like.
[0146]
15 For example, in a case where the electronic shutter
function is used, if resetting is performed immediately
before reading out charges of the photodiode 121f to the
FD unit 163, the photodiode 121f can be in a state of
making no contribution to the detection signal level of
20 the pixel 121a, and if a time is given between the
resetting and the reading out to the FD unit 163, a
partial contribution can be made.
[0147]
As described above, the pixel 121a in Fig. 8 is
25 provided with four photodiodes 121f per pixel, and the
light-shielding film 121b is not formed on the lightreceiving
surface. However, the light-shielding film
121p divides the pixel into a plurality of regions to
form the four photodiodes 121f-1 to 121f-4, so as to
30 express a detection signal for one pixel of a detection
image having an incident angle directivity. In other
58
words, for example, a range of the photodiodes 121f-1 to
121f-4 that does not contribute to output functions
similarly to a light-shielded range, and expresses a
detection signal of one pixel of a detection image having
an incident angle directivity. Note that 5 in a case where
a detection signal for one pixel is expressed using the
photodiodes 121f-1 to 121f-4, since the light-shielding
film 121b is not used, the detection signal is not a
signal obtained by optical modulation.
10 [0148]
The example in which four photodiodes are arranged
in a pixel has been described above, but the number of
photodiodes arranged in the pixel is arbitrary and is not
limited to the above-described example. That is, the
15 number of partial areas in which photodiodes are arranged
in a pixel is also arbitrary.
[0149]
Further, although the photodiodes are described as
being arranged in four partial areas obtained by equally
20 dividing the inside of the pixel into four parts in the
above description, it is not necessary that the partial
areas are equally divided. That is, it is not necessary
that sizes and shapes of respective partial areas are all
unified (a partial area that is different in size and
25 shape from others may be included). Alternatively, a
position (position in the partial area), a size, a shape,
and the like of the photodiode arranged in each partial
area may be different for each photodiode (each partial
area). At this time, the sizes and shapes of the
30 respective partial areas may be all unified or may not be
unified.
59
[0150]
Moreover, these parameters do not have to be
unified for all pixels of the imaging element 121. That
is, in one or more pixels of the imaging element 121, one
or more of these parameters may be different 5 from those
of other pixels.
[0151]
For example, a pixel in which a dividing position
for forming a partial area in which a photodiode is
10 arranged in the pixel is different from that of other
pixels may be included in the pixel group of the imaging
element 121. That is, the imaging element 121 may
include one or more pixels in which partial areas have
different sizes and shapes from those of other pixels.
15 For example, even if only an upper left photodiode is
used in a plurality of pixels by making the dividing
position different for each pixel, incident angle
directivities of detection signals respectively detected
in the plurality of pixels can be made different from
20 each other.
[0152]
Further, for example, the pixel group of the
imaging element 121 may include a pixel in which
positions, sizes, shapes, and the like of a plurality of
25 photodiodes arranged in the pixel are different from
those of other pixels. That is, the imaging element 121
may include one or more pixels in which at least one of
positions, sizes, and shapes of a plurality of
photodiodes arranged is different from those of other
30 pixels. For example, even if only an upper left
photodiode is used in a plurality of pixels by making the
60
positions, sizes, shapes, and the like of the photodiode
different for each pixel, incident angle directivities of
detection signals respectively detected in the plurality
of pixels can be made different from each other.
5 [0153]
Furthermore, for example, one or more pixels in
which both parameters (size, shape) of partial areas and
parameters (position, size, shape) of photodiodes are
different from those of other pixels may be provided.
10 [0154]
Further, for example, a pixel in which the number
of divisions for forming partial areas in which
photodiodes are arranged in the pixel is different from
that of other pixels may be included in the pixel group
15 of the imaging element 121. That is, the imaging element
121 may include one or more pixels in which the number of
photodiodes arranged is different from that of other
pixels. For example, by making the number of divisions
(the number of photodiodes) different for each pixel, the
20 incident angle directivity can be set more freely.
[0155]

The incident angle directivity of each pixel in the
25 imaging element 121 is generated according to principles
illustrated in Fig. 9, for example. Note that an upper
left part and an upper right part of Fig. 9 are diagrams
describing principles of generation of the incident angle
directivity in the imaging element 121 of Fig. 7, and a
30 lower left part and a lower right part of Fig. 9 are
diagrams describing principles of generation of the
61
incident angle directivity in the imaging element 121 of
Fig. 8.
[0156]
Further, one pixel in both of the upper left part
and the upper right part in Fig. 5 9 includes one
photodiode 121e. On the other hand, one pixel in both of
the lower left part and the lower right part in Fig. 9
includes two photodiodes 121f. Note that, here, an
example in which one pixel includes two photodiodes 121f
10 is described, but this is for convenience of description,
and the number of photodiodes 121f constituting one pixel
may be another number.
[0157]
In the upper left part of Fig. 9, a light-shielding
15 film 121b-11 is formed so as to shield a right half of a
light receiving surface of the photodiode 121e-11 from
light when incident light enters from an upper side to a
lower side in the diagram. Further, in the upper right
part of Fig. 9, a light-shielding film 121b-12 is formed
20 so as to shield a left half of the light receiving
surface of the photodiode 121e-12 from light. Note that
a dotted and dashed line in the diagram indicates a
center position, in the horizontal direction in the
diagram, of the light receiving surface of the photodiode
25 121e and a direction perpendicular to the light receiving
surface.
[0158]
For example, in a case of a configuration
illustrated in the upper left part of Fig. 9, incident
30 light, from an upper right direction in the diagram,
indicated by an arrow forming an incident angle θ1 with
62
respect to a dotted and dashed line in the diagram is
easily received in the left half range that is not
shielded from light by the light-shielding film 121b-11
of the photodiode 121e-11, but incident light, from an
upper left direction in the diagram, 5 indicated by an
arrow forming an incident angle θ2 with respect to the
dotted and dashed line in the diagram is difficult to
receive in the left half range that is not shielded from
light by the light-shielding film 121b-11 of the
10 photodiode 121e-11. Therefore, in the case of the
configuration illustrated in the upper left part of Fig.
9, there is provided an incident angle directivity that a
light receiving sensitivity characteristic is high for
the incident light from the upper right direction in the
15 diagram and the light receiving sensitivity
characteristic is low for the incident light from the
upper left direction.
[0159]
On the other hand, for example, in a case of a
20 configuration illustrated in the upper right part of Fig.
9, incident light from an upper right direction in the
diagram indicated by an arrow forming an incident angle
θ11 with respect to a dotted and dashed line in the
diagram is difficult to receive in the left half range
25 that is shielded from light by the light-shielding film
121b-12 of the photodiode 121e-12, but incident light
from an upper left direction in the diagram indicated by
an arrow forming an incident angle θ12 with respect to
the dotted and dashed line in the diagram is easily
30 received in the right half range that is not shielded
from light by the light-shielding film 121b-12 of the
63
photodiode 121e-12. Therefore, in the case of the
configuration illustrated in the upper right part of Fig.
9, there is provided an incident angle directivity that a
light receiving sensitivity characteristic is low for the
incident light from the upper right 5 direction in the
diagram and the light receiving sensitivity
characteristic is high for the incident light from the
upper left direction.
[0160]
10 Further, in the case of the lower left part of Fig.
9, photodiodes 121f-1, 121f-2 are provided on the left
and right in the diagram, and the incident angle
directivity is provided without providing the lightshielding
film 121b by reading out detection signals of
15 either of the photodiodes.
[0161]
Specifically, as illustrated in the lower left part
of Fig. 9, in a case where two photodiodes 121f-1, 121f-2
are formed in the pixel 121a, by making a detection
20 signal of the photodiode 121f-1 provided on the left side
in the diagram contribute to the detection signal level
of this pixel 121a, it is possible to provide a similar
incident angle directivity to that of the configuration
in the upper left part of Fig. 9. That is, incident
25 light from an upper right direction in the diagram
indicated by an arrow forming an incident angle θ21 with
respect to a dotted and dashed line in the diagram is
incident on the photodiode 121f-1 and received, and a
detection signal thereof is read out and contributes to
30 the detection signal level of this pixel 121a. On the
other hand, incident light from an upper left direction
64
in the diagram indicated by an arrow forming an incident
angle θ22 with respect to the dotted and dashed line in
the diagram is incident on the photodiode 121f-2, but a
detection signal thereof is not read out and does not
contribute to the detection signal level 5 of this pixel
121a.
[0162]
Similarly, as illustrated in the lower right part
of Fig. 9, in a case where two photodiodes 121f-11, 121f-
10 12 are formed in the pixel 121a, by making a detection
signal of the photodiode 121f-12 provided on the left
side in the diagram contribute to the detection signal
level of this pixel 121a, it is possible to provide a
similar incident angle directivity to that of the
15 configuration in the upper right part of Fig. 9. That
is, incident light from an upper right direction in the
diagram indicated by an arrow forming an incident angle
θ31 with respect to a dotted and dashed line in the
diagram is incident on the photodiode 121f-11, but a
20 detection signal thereof is not read out and does not
contribute to the detection signal level of this pixel
121a. On the other hand, incident light from an upper
left direction in the diagram indicated by an arrow
forming an incident angle θ32 with respect to the dotted
25 and dashed line in the diagram is incident on the
photodiode 121f-12 and received, and a detection signal
thereof is read out and contributes to the detection
signal level of the pixel 121a.
[0163]
30 Note that in Fig. 9, the example in which a dotted
and dashed line in the vertical direction is a center
65
position in the horizontal direction in the diagram of a
light receiving surface of the photodiode 121e has been
described, but this is for convenience of description and
another position may be employed. Different incident
angle directivities can be generated 5 by having a
different position in the horizontal direction of the
light-shielding film 121b indicated by the dotted and
dashed line in the vertical direction.
[0164]
10
The principles of generation of the incident angle
directivity have been described above. Here, an incident
angle directivity in a configuration including an on-chip
15 lens 121c will be described.
[0165]
That is, an incident angle directivity of each
pixel in the imaging element 121 is set as illustrated in
Fig. 10, for example, by using an on-chip lens 121c in
20 addition to the light-shielding film 121b described
above. That is, in a middle left part of Fig. 10, an onchip
lens 121c-11 that collects incident light from an
incident direction in the upper part of diagram, a color
filter 121d-11 that transmits light with a predetermined
25 wavelength, and a photodiode 121e-11 that generates a
pixel signal by photoelectric conversion are stacked in
this order, and in a middle right part of Fig. 10, an onchip
lens 121c-12, a color filter 121d-12, and a
photodiode 121e-12 are provided in this order from the
30 incident direction in the upper part of the diagram.
[0166]
66
Note that in a case where it is not necessary to
distinguish between the on-chip lenses 121c-11, 121c-12,
the color filters 121d-11, 121d-12, and the photodiodes
121e-11, 121e-12, they will be simply referred to as onchip
lenses 121c, color filters 121d, 5 and photodiodes
121e.
[0167]
The imaging element 121 is further provided with
light-shielding films 121b-11, 121b-12 that shields a
10 part of an area that receives incident light from light,
as illustrated in each of the middle left part and the
middle right part of Fig. 10.
[0168]
As illustrated in the middle left part of Fig. 10,
15 in a case where the light-shielding film 121b-11 that
shields a right half of the photodiode 121e-11 in the
diagram from light is provided, a detection signal level
of the photodiode 121e-11 changes according to an
incident angle θ of incident light as indicated by a
20 solid-line waveform in an upper part of Fig. 10.
[0169]
Specifically, if the incident angle θ, which is an
angle formed by the incident light, increases with
respect to the dotted and dashed line that is at center
25 positions of the photodiode 121e and the on-chip lens
121c and is perpendicular to each of them (if the
incident angle θ increases in a positive direction (if it
tilts rightward in the diagram)), the light is collected
in a range where the light-shielding film 121b-11 is not
30 provided, and thus the detection signal level of the
photodiode 121e-11 increases. Conversely, the smaller
67
the incident angle θ (the larger the incident angle θ is
in the negative direction (if it tilts leftward in the
diagram)), the more the light is collected in a range
where the light-shielding film 121b-11 is provided, and
thus the detection signal level of the 5 photodiode 121e-11
decreases.
[0170]
Note that the incident angle θ mentioned here is
assumed as 0 degree in a case where the direction of the
10 incident light matches the dotted and dashed line, the
incident angle θ on the incident angle θ21 side on the
middle left side of Fig. 10 where the incident light from
the upper right in the diagram is incident is assumed as
a positive value, and the incident angle θ on the
15 incident angle θ22 side on the middle right side of Fig.
10 is assumed as a negative value. Therefore, in Fig.
10, the incident light incident on the on-chip lens 121c
from the upper right has a larger incident angle than the
incident light incident from the upper left. That is, in
20 Fig. 10, the incident angle θ increases as a traveling
direction of incident light tilts rightward (increases in
the positive direction) and decreases as it tilts
leftward (increases in the negative direction).
[0171]
25 Further, as illustrated in the middle right part of
Fig. 10, in a case where the light-shielding film 121b-12
that shields a left half of the photodiode 121e-12 in the
diagram from light is provided, a detection signal level
of the photodiode 121e-12 changes according to an
30 incident angle θ of incident light as indicated by a
dotted-line waveform in the upper part of Fig. 10.
68
[0172]
Specifically, as illustrated by the dotted-line
waveform in the upper part of Fig. 10, the larger the
incident angle θ, which is an angle formed by the
incident light with respect to the dotted 5 and dashed line
that is at the center positions of the photodiode 121e
and the on-chip lens 121c and is perpendicular to each of
them (the larger the incident angle θ in the positive
direction), the more collected the light is in a range
10 where the light-shielding film 121b-12 is provided, and
thus the detection signal level of the photodiode 121e-12
decreases. Conversely, the smaller the incident angle θ
(the larger the incident angle θ is in the negative
direction), the more the light enters a range where the
15 light-shielding film 121b-12 is not provided, and thus
the detection signal level of the photodiode 121e-12
increases.
[0173]
Note that in the upper part of Fig. 10, a
20 horizontal axis represents the incident angle θ, and a
vertical axis represents the detection signal level in
the photodiode 121e.
[0174]
Since the waveforms indicated by the solid line and
25 the dotted line indicating the detection signal levels
corresponding to the incident angle θ illustrated in the
upper part of Fig. 10 can be changed according to the
range of the light-shielding film 121b, it thereby
becomes possible to give (set) different incident angle
30 directivities from each other in pixel units. Note that
the solid-line waveform in the upper part of Fig. 10
69
corresponds to solid-line arrows indicating that incident
light is collected while changing the incident angle θ in
the middle left part and a lower left part of Fig. 10.
Further, the dotted waveform in the upper part of Fig. 10
corresponds to dotted-line arrows 5 indicating that
incident light is collected while changing the incident
angle θ in the middle right part and a lower right part
of Fig. 10.
[0175]
10 The incident angle directivity mentioned here is a
characteristic (light receiving sensitivity
characteristic) of the detection signal level of each
pixel corresponding to the incident angle θ, but in a
case of the example of the middle part of Fig. 10, it can
15 be said that this is a characteristic of light shielding
value according to the incident angle θ. That is, the
light-shielding film 121b shields incident light in a
specific direction at a high level, but cannot
sufficiently shield incident light from directions other
20 than the specific direction. This change in lightshielding
capability level generates different detection
signal levels corresponding to the incident angle θ as
illustrated in the upper part of Fig. 10. Therefore, if
a direction in which light can be shielded at a highest
25 level in each pixel is defined as a light shielding
direction of each pixel, having different incident angle
directivities from each other in pixels units means, in
other words, having different light shielding directions
from each other in pixel units.
30 [0176]
Furthermore, as illustrated in the lower left part
70
of Fig. 10, by providing two photodiodes 121f-1, 121f-2
for one on-chip lens 121c-11 (the pixel output unit
includes two photodiodes 121f-1, 121f-2) so as to use the
detection signal of the photodiode 121f-1 only in the
left part of the diagram, it is possible 5 to obtain the
same detection signal level as in a state that the right
side of the photodiode 121e-11 in the middle left part of
Fig. 10 is shielded from light.
[0177]
10 Specifically, if the incident angle θ, which is an
angle formed by the incident light, increases with
respect to the dotted and dashed line that is at the
center position of the on-chip lens 121c and is
perpendicular to each of them (if the incident angle θ
15 increases in the positive direction), the light is
collected in a range of the photodiode 121f-1 where the
detection signal is read out, and thus the detection
signal level increases. Conversely, the smaller the
incident angle θ (the larger the incident angle θ is in
20 the negative direction), the more the light is collected
in a range of the photodiode 121f-2 where the detection
value is not read out, and thus the detection signal
level decreases.
[0178]
25 Further, similarly, as illustrated in the lower
right part of Fig. 10, by providing two photodiodes 121f-
11, 121f-12 for one on-chip lens 121c-12 so as to use the
detection signal of the photodiode 121f-12 only in the
right part of the diagram, it is possible to obtain the
30 detection signal of the output pixel unit of the same
detection signal level as in a state that the left side
71
of the photodiode 121e-12 in the middle right part of
Fig. 10 is shielded from light.
[0179]
Specifically, if the incident angle θ, which is an
angle formed by the incident light, 5 increases with
respect to the dotted and dashed line that is at the
center position of the on-chip lens 121c and is
perpendicular to each of them (if the incident angle θ
increases in the positive direction), the light is
10 collected in a range of the photodiode 121f-11 where the
detection signal does not contribute to a detection
signal of the output pixel unit, and thus the detection
signal level of the detection signal of the output pixel
unit decreases. Conversely, the smaller the incident
15 angle θ (the larger the incident angle θ is in the
negative direction), the more the light is collected in a
range of the photodiode 121f-12 where the detection
signal contributes to the detection signal of the output
pixel unit, and thus the detection signal level of the
20 detection signal of the output pixel unit increases.
[0180]
Note that it is desirable that the incident angle
directivity has high randomness. This is because, for
example, if adjacent pixels have the same incident angle
25 directivity, the equations (1) to (3) described above or
equations (4) to (6) described later may become the same
equations as each other, the relationship between the
number of unknowns and the number of equations to be
solutions of simultaneous equations cannot be satisfied,
30 and pixel values constituting the restored image may not
be obtained. Further, in the configuration illustrated
72
in the middle part of Fig. 10, one photodiode 121e-11 and
one photodiode 121e-12 are formed in the pixel 121a. On
the other hand, in the configuration illustrated in the
lower part of Fig. 10, two photodiodes 121f-1 and 121f-2
and photodiodes 121f-11 and 121f-12 5 are formed in the
pixel 121a. Therefore, for example, in the lower part of
Fig. 10, a single photodiode 121f does not constitute one
pixel.
[0181]
10 Further, as illustrated in the lower part of Fig.
10, in a case where one pixel output unit includes a
plurality of photodiodes 121f, it can be considered that
an output pixel value of the pixel output unit is
modulated according to the incident angle. Therefore,
15 the characteristic of the output pixel value (incident
angle directivity) can be made different for each pixel
output unit, and an incident angle directivity in one
pixel output unit is set. Furthermore, in a case where
one pixel output unit includes a plurality of photodiodes
20 121f, one on-chip lens 121c is a necessary component for
one pixel output unit in order to generate an incident
angle directivity in one pixel output unit.
[0182]
Further, as illustrated in the upper part of Fig.
25 10, in a case where each of one photodiode 121e-11 or
photodiode 121e-12 constitutes one pixel output unit, by
modulating incident light to one photodiode 121e-11 or
photodiode 121e-12 constituting one pixel output unit
according to the incident angle, an output pixel value is
30 modulated as a result. Therefore, the characteristics of
the output pixel value (incident angle directivities) can
73
be made different, and the incident angle directivity in
one pixel output unit is set. Furthermore, in a case
where each of one photodiode 121e-11 or photodiode 121e-
12 constitutes one pixel output unit, the incident angle
directivity is set independently, by the 5 light-shielding
film 121b provided for each pixel output unit in
manufacturing.
[0183]
Further, as illustrated in the lower part of Fig.
10 10, in a case where one pixel output unit includes a
plurality of photodiodes 121f, the number of the
plurality of photodiodes 121f for setting the incident
angle directivity in each one pixel output unit (divided
number of photodiodes 121f constituting one pixel output
15 unit) and positions thereof are set independently in one
pixel output unit in manufacturing, and further which
photodiode 121f among these is used to set the incident
angle directivity can be switched at a time of imaging.
[0184]
20
For example, as illustrated in an upper part of
Fig. 11, a setting range of the light-shielding film 121b
is a range from a left end to a position A in the
horizontal direction in the pixel 121a, and a range from
25 an upper end to a position B in the vertical direction.
[0185]
In this case, a weight Wx of 0 to 1 in the
horizontal direction serving as an index of incident
angle directivity is set according to an incident angle
30 θx (deg) from a center position in the horizontal
direction of each pixel. More specifically, assuming
74
that the weight Wx is 0.5 for the incident angle θx = θa
corresponding to the position A, a weight Wh is set so
that the weight Wx is 1 for the incident angle θx < θa -
α, the weight Wx is (-(θx - θa)/2α + 1/2) for θa - α ≤
the incident angle θx ≤ θa + α, and 5 the weight Wx is 0
for the incident angle θx > θa + α. Note that an example
in which the weight Wh is 0, 0.5, 1 will be described
here, and the weight Wh becomes 0, 0.5, 1 when an ideal
condition is satisfied.
10 [0186]
Similarly, a weight Wy of 0 to 1 in the vertical
direction serving as an index of incident angle
directivity is set according to an incident angle θy
(deg) from a center position in the vertical direction of
15 each pixel. More specifically, assuming that a weight Wv
is 0.5 for the incident angle θy = θb corresponding to
the position B, the weight Wy is set so that the weight
Wy is 0 for the incident angle θy < θb - α, the weight Wy
is ((θy - θb)/2α + 1/2) for θb - α ≤ the incident angle
20 θy ≤ θb + α, and the weight Wy is 1 for the incident
angle θy > θb + α.
[0187]
Then, by using the weights Wx, Wy thus obtained,
the incident angle directivity of each pixel 121a, that
25 is, a coefficient (coefficient set) corresponding to the
light receiving sensitivity characteristic can be
obtained.
[0188]
Further, at this time, an inclination (1/2α)
30 indicating a change in weight in the range where the
weight Wx in the horizontal direction and the weight Wy
75
in the vertical direction are around 0.5 can be set by
using an on-chip lens 121c having a different focal
length.
[0189]
In other words, different focal 5 lengths can be
provided by using an on-chip lens 121c having a different
curvature.
[0190]
For example, by using the on-chip lens 121c having
10 a different curvature, when the light is collected so
that the focal length is on the light-shielding film 121b
as illustrated by solid lines in a lower part of Fig. 11,
the inclination (1/2α) becomes steep. That is, the
weight Wx in the horizontal direction and the weight Wy
15 in the vertical direction in the upper part of Fig. 11
suddenly change to 0 or 1 near a boundary at the incident
angle θx = θa in the horizontal direction and the
incident angle θy = θb in the vertical direction where
they are close to 0.5.
20 [0191]
Further, for example, by using the on-chip lens
121c having a different curvature, when the light is
collected so that the focal length is on the photodiode
121e as illustrated by dotted lines in the lower part of
25 Fig. 11, the inclination (1/2α) becomes gentle. That is,
the weight Wx in the horizontal direction and the weight
Wy in the vertical direction are close to 0.5 in the
upper part of Fig. 11 gently change to 0 or 1 near the
boundary at the incident angle θx = θa in the horizontal
30 direction and the incident angle θy = θb in the vertical
direction.
76
[0192]
As described above, different incident angle
directivities, that is, different light receiving
sensitivity characteristics can be obtained by using the
on-chip lenses 121c having different 5 curvatures to have
different focal lengths.
[0193]
Therefore, the incident angle directivity of the
pixel 121a can be set to a different value by making the
10 range in which the photodiode 121e is shielded from light
by the light-shielding film 121b and the curvature of the
on-chip lens 121c different. Note that the curvature of
the on-chip lens may be the same for all pixels in the
imaging element 121, or may be different for part of
15 pixels.
[0194]

As described above, the imaging element 121 does
not need an imaging lens. However, the on-chip lens 121c
20 is necessary at least in a case where the incident angle
directivity is achieved using a plurality of photodiodes
in a pixel as described with reference to Fig. 8. The
on-chip lens 121c and the imaging lens have different
physical operations.
25 [0195]
The imaging lens has a light-collecting function
for causing incident light entering from the same
direction to be incident on a plurality of pixels
adjacent to each other. On the other hand, a light
30 passing through the on-chip lens 121c is incident only on
the light receiving surface of the photodiode 121e or
77
121f constituting one corresponding pixel. In other
words, the on-chip lens 121c is provided for each pixel
output unit, and collects object light incident on itself
on only the corresponding pixel output unit. That is,
the on-chip lens 121c does not have 5 a light-collecting
function for causing diffused light emitted from a
virtual point light source to be incident on a plurality
of pixels adjacent to each other.
[0196]
10
Next, the relationship of a distance between an
object plane and the imaging element 121 will be
described with reference to Fig. 12.
15 [0197]
As illustrated in an upper left part of Fig. 12, in
a case where the object distance between the imaging
element 121 and the object plane 131 is a distance d1,
for example, when point light sources PA, PB, PC on the
20 object plane 131 are set, it is assumed that detection
signal levels DA, DB, DC at positions Pa, Pb, Pc on the
corresponding imaging element 121 can be expressed by the
same equations as the equations (1) to (3) described
above.
25 [0198]
DA = α1 × a + β1 × b + γ1 × c
... (1)
DB = α2 × a + β2 × b + γ2 × c
... (2)
30 DC = α3 × a + β3 × b + γ3 × c
... (3)
78
[0199]
On the other hand, as illustrated in a lower left
part of Fig. 12, in a case where the object distance from
the imaging element 121 is an object plane 131’ having a
distance d2 larger than the distance d1 5 by d, that is, in
a case of the object plane 131’ deeper than the object
plane 131 as viewed from the imaging element 121, the
detection signal levels are similar for all of the
detection signal levels DA, DB, DC as illustrated in an
10 upper center part and a lower center part of Fig. 12.
[0200]
However, in this case, light beams having light
intensities a’, b’, c’ from the point light sources PA’,
PB’, PC’ on the object plane 131’ are received by each
15 pixel of the imaging element 121. At this time, incident
angles of the light beams having the light intensities
a’, b’, c’ received on the imaging element 121 are
different (change) and thus respective different
coefficient sets are necessary, and the detection signal
20 levels DA, DB, DC at the respective positions Pa, Pb, Pc
are expressed as described in the following equations (4)
to (6), for example.
[0201]
DA = α11 × a' + β11 × b' + γ11 × c'
25 ... (4)
DB = α12 × a' + β12 × b' + γ12 × c'
... (5)
DC = α13 × a' + β13 × b' + γ13 × c'
... (6)
30 [0202]
Here, a coefficient set group including a
79
coefficient set α11, β11, γ11, a coefficient set α12,
β12, γ12, and a coefficient sets α13, β13, γ13 is a
coefficient set group of the object plane 131’ that
corresponds to a coefficient set α1, β1, γ1, a
coefficient set α2, β2, γ2, and a coefficient 5 set α3, β3,
γ3, respectively, on the object plane 131.
[0203]
Therefore, by solving the equations (4) to (6)
using the preset coefficient set group α11, β11, γ11,
10 α12, β12, γ12, α13, β13, γ13, they can be obtained as the
light intensities (a', b', c') of the light beams from
the point light sources PA', PB', PC' as illustrated in a
lower right part of Fig. 12 in a similar manner to the
method of obtaining the light intensities (a, b, c) of
15 the light beams at the point light sources PA, PB, PC in
a case of the object plane 131 illustrated in an upper
right part of Fig. 12, and as a result, a restored image
of the object on the object plane 131' can be obtained.
[0204]
20 That is, in the imaging device 101 in Fig. 4, it is
possible to obtain a restored image of the object plane
at various object distances on the basis of one detection
image by storing coefficient set groups for respective
distances from the imaging element 121 to the object
25 plane in advance, composing simultaneous equations by
switching the coefficient set groups, and solving the
composed simultaneous equations.
[0205]
In other words, by simply imaging the detection
30 image once, it is also possible to generate a restored
image at an arbitrary distance in a subsequent process by
80
switching the coefficient set group according to the
distance to the object plane and obtaining the restored
image.
[0206]
Further, in a case where it is 5 desired to obtain
characteristics of an object of image recognition or a
visible image, or an object other than a visible image,
it is also possible to apply machine learning such as
deep learning on a detection signal of the imaging
10 element and perform image recognition or the like using
the detection signal itself, without performing image
recognition or the like on the basis of a restored image
after obtaining the restored image.
[0207]
15 Further, in a case where an object distance and an
angle of view can be specified, a restored image may be
generated using a detection image formed by a detection
signal of a pixel having an incident angle directivity
suitable for imaging an object plane corresponding to the
20 specified object distance and angle of view, without
using all the pixels. In this manner, a restored image
can be obtained using a detection signal of a pixel
suitable for imaging the object plane corresponding to
the specified object distance and angle of view.
25 [0208]
For example, a pixel 121a shielded from light by
the light-shielding film 121b by a width d1 from each end
of the four sides as illustrated in an upper part of Fig.
13, and a pixel 121a’ shielded from light by the light30
shielding film 121b by a width d2 (> d1) from each end of
the four sides as illustrated in a lower part of Fig. 13
81
will be considered.
[0209]
The pixel 121a is used, for example, for restoring
an image I1 in Fig. 13 corresponding to an angle of view
SQ1 including an entire person H101 5 as an object, as
illustrated in an upper part of Fig. 14. On the other
hand, the pixel 121a’ is used, for example, for restoring
an image I2 in Fig. 13 corresponding to an angle of view
SQ2 in which a periphery of the face of the person H101
10 as the object is zoomed up, as illustrated in the upper
part of Fig. 14.
[0210]
The pixel 121a in Fig. 13 is used because it has an
incident possible angle range A of incident light with
15 respect to the imaging element 121 as illustrated in a
left part of Fig. 15, and thus incident light by an
amount of an object width W1 can be received in a
horizontal direction on the object plane 131.
[0211]
20 On the other hand, the pixel 121a’ in Fig. 13 is
used because it has a wider light-shielded range than the
pixel 121a in Fig. 13 and hence has an incident possible
angle range B (< A) of incident light with respect to the
imaging element 121 as illustrated in the left part of
25 Fig. 15, and thus incident light by an amount of an
object width W2 (< W1) is received in the horizontal
direction on the object plane 131.
[0212]
That is, the pixel 121a in Fig. 13 with a narrow
30 light-shielding range is a wide angle-of-view pixel
suitable for imaging a wide range on the object plane
82
131, whereas the pixel 121a’ in Fig. 13 with a wide
light-shielding range is a narrow angle-of-view pixel
suitable for imaging a narrow range on the object plane
131. Note that the wide angle-of-view pixel and the
narrow angle-of-view pixel mentioned here 5 are expressions
to compare both the pixels 121a, 121a’ in Fig. 13, and
are not limited to this for comparing pixels having other
angles of views.
[0213]
10 Note that Fig. 15 illustrates the relationship
between a position on the object plane 131 and the
incident angle of incident light from each position with
respect to the center position C1 of the imaging element
121. Further, in Fig. 15, the relationship between the
15 position on the object plane 131 and the incident angle
of incident light from each position on the object plane
131 with respect to the horizontal direction is
illustrated, but a similar relationship applies to the
vertical direction. Moreover, a right part of Fig. 15
20 illustrates the pixels 121a, 121a’ in Fig. 13.
[0214]
With such a configuration, as illustrated in a
lower part of Fig. 14, in a case where it is configured
by collecting in the imaging element 121 a predetermined
25 number of pixels of each of the pixel 121a of Fig. 13 in
a range ZA surrounded by a dotted line and the pixel
121a’ of Fig. 13 in a range ZB surrounded by a dotted and
dashed line, when it is attempted to restore the image
with the angle of view SQ1 corresponding to the object
30 width W1, using the pixel 121a of Fig. 13 that images the
angle of view SQ1 enables to appropriately restore the
83
image with the object width W1 of the object plane 131.
[0215]
Similarly, when it is attempted to restore the
image with the angle of view SQ2 corresponding to the
object width W2, using the detection signal 5 level of the
pixel 121a’ of Fig. 13 that images the angle of view SQ2
enables to appropriately restore the image with the
object width W2.
[0216]
10 Note that in the lower part of Fig. 14, although a
configuration in which a predetermined number of pixels
121a’ is provided on a left side in the drawing and a
predetermined number of pixels 121a is provided on a
right side is illustrated, this is illustrated as an
15 example for simplicity of description, and the pixels
121a and the pixels 121a’ are desirably arranged in a
random mixture.
[0217]
In this manner, since the angle of view SQ2 is
20 narrower than the angle of view SQ1, in a case of
restoring images of the angle of view SQ2 and the angle
of view SQ1 with the same predetermined number of pixels,
a restored image with higher image quality can be
obtained by restoring the image of the angle of view SQ2
25 that is narrower than the image of the angle of view SQ1.
[0218]
That is, in a case where it is considered to obtain
a restored image using the same number of pixels, a
restored image with higher image quality can be obtained
30 by restoring an image with a narrower angle of view.
[0219]
84
Note that in a case of obtaining an image with a
wide angle of view as a restored image, all of the wide
angle-of-view pixels may be used, or a part of the wide
angle-of-view pixels may be used. Further, in a case of
obtaining an image with a narrow 5 angle of view as a
restored image, all of the narrow angle-of-view pixels
may be used, or a part of the narrow angle-of-view pixels
may be used.
[0220]
10 By using the imaging element 121 as described
above, consequently, an imaging lens, an optical element
including a diffraction grating and the like, a pinhole,
or the like is not necessary. Thus, the degree of
freedom in designing the device can be increased, and it
15 is possible to achieve size reduction of the device with
respect to an incident direction of incident light, and
to reduce manufacturing cost. Further, a lens
corresponding to an imaging lens for forming an optical
image, such as a focus lens, becomes unnecessary.
20 [0221]
Moreover, by using the imaging element 121, it is
possible to generate a restored image at various object
distances and with angles of view only by obtaining a
detection image, and thereafter obtaining a restored
25 image by solving simultaneous equations composed by
selectively using a coefficient set group according to an
object distance and an angle of view.
[0222]
Moreover, since the imaging element 121 can have an
30 incident angle directivity in pixel units, pixel
multiplication can be implemented, and a restored image
85
with high resolution and high angular resolution can be
obtained, compared to an optical filter including a
diffraction grating and a conventional imaging element,
or the like. On the other hand, in an imaging device
including an optical filter and a conventional 5 imaging
element, it is difficult to achieve a high resolution or
the like of a restored image because it is difficult to
miniaturize the optical filter even if the pixels are
miniaturized.
10 [0223]
Further, since the imaging element 121 does not
require an optical filter or the like including a
diffraction grating, deformation of an optical filter by
heat or the like due to high temperatures of the use
15 environment does not occur. Therefore, by using such an
imaging element 121, it is possible to achieve a device
with high environmental resistance.
[0224]

20 In the right part of Fig. 6, as a configuration of
the light-shielding film 121b in each pixel 121a of the
imaging element 121, there is provided an example in
which an overall light-shielding is provided in the
vertical direction and a light shielding width and
25 position in the horizontal direction are changed, so as
to have a difference in an incident angle directivity in
the horizontal direction. However, the configuration of
the light-shielding film 121b is not limited to this
example. For example, an overall light-shielding may be
30 provided in the horizontal direction and a width (height)
and position in the vertical direction are changed, so as
86
to have a difference in an incident angle directivity in
the vertical direction.
[0225]
Note that as in the example illustrated in the
right part of Fig. 6, the light-shielding 5 film 121b that
shields the entire pixel 121a from light in the vertical
direction and shields the pixel 121a from light by a
predetermined width in the horizontal direction will be
referred to as a horizontal band type light-shielding
10 film 121b. On the other hand, the light-shielding film
121b that shields the entire pixel 121a from light in the
horizontal direction and shields the pixel 121a from
light by a predetermined height in the vertical direction
will be referred to as a vertical band type light15
shielding film 121b.
[0226]
Further, as in an example illustrated in a left
part of Fig. 16, the pixel 121a may be provided with an
L-shaped light-shielding film 121b by combining vertical
20 band type and horizontal band type light-shielding films
121b. In the left part of Fig. 16, a part illustrated in
black is a light-shielding film 121b. That is, lightshielding
films 121b-21 to 121b-24 are light-shielding
films of the pixels 121a-21 to 121a-24, respectively.
25 [0227]
Each of these pixels (pixels 121a-21 to 121a-24)
has an incident angle directivity as illustrated in a
right part of Fig. 16. A graph illustrated in the right
part of Fig. 16 illustrates light receiving sensitivity
30 in each pixel. A horizontal axis represents an incident
angle θx in the horizontal direction (x direction) of
87
incident light, and a vertical axis represents an
incident angle θy in the vertical direction (y direction)
of the incident light. Then, light receiving sensitivity
within a range C4 is higher than that outside the range
C4, light receiving sensitivity within 5 a range C3 is
higher than that outside the range C3, light receiving
sensitivity within a range C2 is higher than that outside
the range C2, and light receiving sensitivity within a
range C1 is higher than that outside the range C1.
10 [0228]
Therefore, it is illustrated that, for each pixel,
the incident light that satisfies conditions of the
incident angle θx in the horizontal direction (x
direction) and the incident angle θy in the vertical
15 direction (y direction), which is within the range C1,
has the highest detection signal level, and the detection
signal level decreases in the order of the conditions in
the range C2, the range C3, the range C4, and the range
other than the range C4. The intensity of such light
20 receiving sensitivity is determined by the range shielded
from light by the light-shielding film 121b.
[0229]
Further, in the left part of Fig. 16, an alphabet
in each pixel 121a indicates the color of a color filter
25 (illustrated for convenience of description and is not
actually written). The pixel 121a-21 is a G pixel in
which a green color filter is disposed, the pixel 121a-22
is an R pixel in which a red color filter is disposed,
the pixel 121a-23 is a B pixel in which a blue color
30 filter is disposed, and the pixel 121a-24 is a G pixel in
which a green color filter is disposed. That is, these
88
pixels form a Bayer array. Of course, this is an
example, and the arrangement pattern of the color filters
is arbitrary. The arrangement of the light-shielding
film 121b and the color filter are irrelevant. For
example, in part or all of the pixels, 5 a filter other
than the color filter may be provided, or no filter may
be provided.
[0230]
In the left part of Fig. 16, although an example in
10 which the “L-shaped” light-shielding film 121b shields a
left side and a lower side of the pixel 121a in the
diagram from light is illustrated, the direction of this
“L-shaped” light-shielding film 121b is arbitrary and is
not limited to the example in Fig. 16. For example, the
15 “L-shaped” light-shielding film 121b may shield the lower
side and a right side of the pixel 121a in the diagram
from light, may shield the right side and an upper side
of the pixel 121a in the diagram from light, or may
shield the upper side and the left side of the pixel 121a
20 in the diagram from light. Of course, the direction of
the light-shielding film 121b can be set independently
for each pixel. Note that the “L-shaped” light-shielding
film 121b will also be collectively referred to as an “L
type light-shielding film 121b”.
25 [0231]
Although the light-shielding film has been
described above, the description of this example can also
be applied to a case where an incident angle directivity
is given by selectively using a plurality of photodiodes
30 disposed in a pixel. That is, for example, by
appropriately setting a dividing position (size and shape
89
of each partial area), a position, a size, a shape, and
the like of each photodiode, or appropriately selecting a
photodiode, incident light directivity equivalent to the
incident light directivity by the L type light-shielding
film 121b described above 5 can be achieved.
[0232]

In the above, although the horizontal band type,
the vertical band type, and the L type light-shielding
10 films have been described with respect to an example in
which a light-shielded range is arranged in each pixel in
a randomly changing manner, for example, as illustrated
by an imaging element 121’ in Fig. 17, there may be
formed a light-shielding film 121b (range illustrated in
15 black in the diagram) that shields a portion other than
the range in the vicinity of a position where a light
beam is received in each of the pixels from light in a
case where a rectangular opening is provided.
[0233]
20 That is, the light-shielding film 121b may be
provided so as to have an incident angle directivity to
receive, in a case where a rectangular opening is
provided for each pixel, only a light beam that is
transmitted through the rectangular opening to be
25 received, out of light beams from a point light source
that form an object plane at a predetermined object
distance.
[0234]
Note that in Fig. 17, for example, the width in the
30 horizontal direction of the light-shielding film 121b
changes to widths dx1, dx2, ... dxn with respect to a
90
pixel arrangement in the horizontal direction, and the
widths are in the relationship of dx1 < dx2 < ... < dxn.
Similarly, the height in the vertical direction of the
light-shielding film 121b changes to heights dy1, dy2 ...
dym with respect to a pixel arrangement 5 in the vertical
direction, and the heights are in the relationship of dy1
< dy2 < ... < dxm. Further, intervals of respective
changes in the width in the horizontal direction and the
width in the vertical direction of the light-shielding
10 film 121b depend on object resolution (angular
resolution) to be restored.
[0235]
In other words, it can be said that the
configuration of each pixel 121a in the imaging element
15 121’ in Fig. 17 has an incident angle directivity such
that the light-shielding range changes so as to
correspond to the pixel arrangement in the imaging
element 121’ in the horizontal direction and the vertical
direction.
20 [0236]
More specifically, the light-shielding range of
each pixel 121a in Fig. 17 is determined according to
rules described using a pixel 121a illustrated on a left
part in Fig. 18, for example.
25 [0237]
Note that a right part of Fig. 18 illustrates the
same configuration of the imaging element 121’ as that of
Fig. 17. Further, the left part of Fig. 18 illustrates a
configuration of the pixel 121a of the imaging element
30 121’ in the right part of Fig. 18 (same as Fig. 17).
[0238]
91
As illustrated in the left part of Fig. 18, the
light-shielding film 121b shields light by width dx1 from
each of upper and lower side ends of the pixel 121a
toward the inside of the pixel 121a, and the lightshielding
film 121b shields light by height 5 dy1 from each
of left and right side ends toward the inside of the
pixel 121a. Note that in Figs. 18 and 19, the lightshielding
film 121b is a range illustrated in black.
[0239]
10 A range shielded from light by the light-shielding
film 121b formed in this manner in the left part of Fig.
18 will be hereinafter referred to as a main lightshielding
portion Z101 of the pixel 121a (black part in
the left part of Fig. 18), and a square-shaped range
15 other than that will be referred to as a range Z102.
[0240]
A rectangular opening Z111 that is not shielded
from light by the light-shielding film 121b is provided
in the range Z102 in the pixel 121a. Therefore, in the
20 range Z102, the range other than the rectangular opening
Z111 is shielded from light by the light-shielding film
121b.
[0241]
The pixel arrangement in the imaging element 121’
25 in Fig. 17 is such that in the pixel 121a-1 at a left end
portion and an upper end portion, a rectangular opening
Z111 is arranged so that a left side thereof is at a
distance of width dx1 from the left side of the pixel
121a, and an upper side thereof is at a distance of dy1
30 from the upper side of the pixel 121a, as illustrated in
the right part of Fig. 18 (same as Fig. 17).
92
[0242]
Similarly, in the pixel 121a-2 on a right side of
the pixel 121a-1, a rectangular opening Z111 is arranged
so that a left side thereof is at a distance of width dx2
from the left side of the pixel 121a, 5 and an upper side
thereof is at a distance of height dy1 from the upper
side of the pixel 121a, and a range other than the
rectangular opening Z111 is shielded from light by the
light-shielding film 121b.
10 [0243]
Thereafter, similarly, in a pixel 121a adjacent in
the horizontal direction, the right side of the
rectangular opening Z111 moves by the widths dx1, dx2 ...
dxn from the right side of the pixel 121a as its
15 arrangement moves toward a right side in the diagram.
Note that a rectangular portion of a dotted line of an
upper right portion in the range Z102 of Fig. 18
illustrates a state that the rectangular opening Z111 is
arranged so that a left side thereof is at a distance of
20 the width dxn from the left side of the pixel 121a, and
an upper side thereof is at the distance of height dy1
from the upper side of the pixel 121a. Further, each of
intervals of the widths dx1, dx2 ... dxn is a value
obtained by dividing a width obtained by subtracting the
25 width of the rectangular opening Z111 from the width in
the horizontal direction of the range Z102 by the number
of pixels n in the horizontal direction. That is, an
interval of change in the horizontal direction is
determined by dividing by the number of pixels n in the
30 horizontal direction.
[0244]
93
Further, a position in the horizontal direction of
the rectangular opening Z111 in the pixel 121a in the
imaging element 121’ is the same in the pixels 121a
(pixels 121a in the same column) at the same positions in
the horizontal direction in the imaging 5 element 121’.
[0245]
Moreover, in the pixel 121a-3 adjacent on an
immediately lower side of the pixel 121a-1, a rectangular
opening Z111 is arranged so that a left side thereof is
10 at a distance of width dx1 from the left side of the
pixel 121a, and an upper side thereof is at a distance of
height dy2 from the upper side of the pixel 121a, and a
range other than the rectangular opening Z111 is shielded
from light by the light-shielding film 121b.
15 [0246]
Thereafter, similarly, in a pixel 121a adjacent in
the vertical direction, the upper side of the rectangular
opening Z111 moves by the heights dy1, dy2 ... dyn from
the upper side of the pixel 121a as its arrangement moves
20 toward a lower side in the diagram. Note that a
rectangular portion of a dotted line of a lower left
portion in the range Z102 of Fig. 18 illustrates a state
that the rectangular opening Z111 is arranged so that a
left side thereof is at the distance of width dx1 from
25 the left side of the pixel 121a, and an upper side
thereof is at a distance of the height dym from the upper
side of the pixel 121a. Further, each of intervals of
the heights dy1, dy2 ... dym is a value obtained by
dividing a height obtained by subtracting the height of
30 the rectangular opening Z111 from the height in the
vertical direction of the range Z102 by the number of
94
pixels m in the vertical direction. That is, an interval
of change in the vertical direction is determined by
dividing by the number of pixels m in the vertical
direction.
5 [0247]
Further, a position in the vertical direction of
the rectangular opening Z111 in the pixel 121a in the
imaging element 121’ is the same in the pixels 121a
(pixels 121a in the same row) at the same positions in
10 the vertical direction in the imaging element 121’.
[0248]
Moreover, the angle of view can be changed by
changing the main light-shielding portion Z101 and the
rectangular opening Z111 of each pixel 121a constituting
15 the imaging element 121’ illustrated in Fig. 18 (Fig.
17).
[0249]
A right part of Fig. 19 illustrates a configuration
of an imaging element 121’ in a case where the angle of
20 view is wider than that of the imaging element 121’ of
Fig. 18 (Fig. 17). Further, a left part of Fig. 19
illustrates a configuration of the pixel 121a of the
imaging element 121’ in the right part of Fig. 19.
[0250]
25 That is, as illustrated in the left part of Fig.
19, for example, a main light-shielding portion Z151
(black part in the left part of Fig. 19) having a lightshielding
range narrower than the main light-shielding
portion Z101 in Fig. 18 is set in the pixel 121a, and a
30 range other than that is set as a range Z152. Moreover,
a rectangular opening Z161 having a larger opening area
95
than the rectangular opening Z111 is set in the range
Z152.
[0251]
More specifically, as illustrated in the left part
of Fig. 19, the light-shielding film 5 121b shields light
by width dx1’ (< dx1) from each of upper and lower side
ends of the pixel 121a toward the inside of the pixel
121a, and the light-shielding film 121b shields light by
height dy1’ (< dy1) from each of left and right side ends
10 toward the inside of the pixel 121a, thereby forming the
rectangular opening Z161.
[0252]
Here, as illustrated in the right part of Fig. 19,
in the pixel 121a-1 at a left end portion and at an upper
15 end portion, a rectangular opening Z161 is arranged so
that a left side thereof is at a distance of width dx1’
from the left side of the pixel 121a, and an upper side
thereof is at a distance of height dy1’ from the upper
side of the pixel 121a, thereby light-shielding a range
20 other than the rectangular opening Z161 by the lightshielding
film 121b.
[0253]
Similarly, in the pixel 121a-2 on a right side of
the pixel 121a-1, a rectangular opening Z161 is arranged
25 so that a left side thereof is at a width dx2’ from the
left side of the pixel 121a, and an upper side thereof is
at a height dy1’ from the upper side of the pixel 121a,
and a range other than the rectangular opening Z161 is
shielded from light by the light-shielding film 121b.
30 [0254]
Thereafter, similarly, in a pixel 121a adjacent in
96
the horizontal direction, the right side of the
rectangular opening Z161 moves by the widths dx1’,
dx2’ ... dxn’ from the right side of the pixel 121a as
its arrangement moves toward a right side in the diagram.
Here, each of intervals of the widths dx1’, 5 dx2’ ... dxn’
is a value obtained by dividing a width obtained by
subtracting the width in the horizontal direction of the
rectangular opening Z161 from the width in the horizontal
direction of the range Z152 by the number of pixels n in
10 the horizontal direction. That is, an interval of change
in the vertical direction is determined by dividing by
the number of pixels n in the horizontal direction.
Therefore, intervals of changes in the widths dx1’,
dx2’ ... dxn’ are larger than intervals of changes in the
15 widths dx1, dx2 ... dxn.
[0255]
Further, a position in the horizontal direction of
the rectangular opening Z161 in the pixel 121a in the
imaging element 121’ of Fig. 19 is the same in the pixels
20 121a (pixels 121a in the same column) at the same
positions in the horizontal direction in the imaging
element 121’.
[0256]
Moreover, in the pixel 121a-3 adjacent on an
25 immediately lower side of the pixel 121a-1, a rectangular
opening Z161 is arranged so that a left side thereof is
at a distance of width dx1’ from the left side of the
pixel 121a, and an upper side thereof is at a distance of
height dy2’ from the upper side of the pixel 121a, and a
30 range other than the rectangular opening Z161 is shielded
from light by the light-shielding film 121b.
97
[0257]
Thereafter, similarly, in a pixel 121a adjacent in
the vertical direction, the upper side of the rectangular
opening Z161 moves by the heights dy1’, dy2’ ... dym’
from the upper side of the pixel 121a 5 as its arrangement
moves toward a lower side in the diagram. Here,
intervals of changes in the heights dy1’, dy2’ ... dym’
are a value obtained by dividing a height obtained by
subtracting the height in the vertical direction of the
10 rectangular opening Z161 from the height in the vertical
direction of the range Z152 by the number of pixels m in
the vertical direction. That is, an interval of change
in the vertical direction is determined by dividing by
the number of pixels m in the vertical direction.
15 Therefore, intervals of changes in the heights dy1’,
dy2’ ... dym’ are larger than intervals of changes in the
width heights dy1, dy2 ... dym.
[0258]
Further, a position in the vertical direction of
20 the rectangular opening Z161 in the pixel 121a in the
imaging element 121’ of Fig. 19 is the same in the pixels
121a (pixels 121a in the same row) at the same positions
in the vertical direction in the imaging element 121’.
[0259]
25 In this manner, by changing the combination of the
light-shielding range of the main light-shielding portion
and the opening range of the opening, the imaging element
121’ including pixels 121a having various angles of view
(having various incident angle directivities) can be
30 achieved.
[0260]
98
Moreover, the imaging element 121 may be achieved
by combining not only the pixels 121a having the same
angle of view but also the pixels 121a having various
angles of view.
5 [0261]
For example, as illustrated in Fig. 20, four pixels
including two pixels × two pixels indicated by dotted
lines are defined as one unit U, and each unit U includes
a wide angle-of-view pixel 121a-W, a medium angle-of-view
10 pixel 121a-M, a narrow angle-of-view pixel 121a-N, and a
very narrow angle-of-view pixel 121a-AN.
[0262]
In this case, for example, in a case where the
number of pixels of all the pixels 121a is X, it is
15 possible to restore the restored image using detection
images of every X/4 pixels for each of the four types of
angles of view. At this time, four different types of
coefficient sets are used for every angle of view, and
restored images having different angles of view are
20 restored respectively by four different simultaneous
equations.
[0263]
Thus, by restoring the restored image of an angle
of view to be restored using a detection image obtained
25 from pixels suitable for imaging with the angle of view
to be restored, an appropriate restored image
corresponding to the four types of angles of view can be
restored.
[0264]
30 Further, an image with an intermediate angle of
view among the four types of angles of view or with an
99
angle of view before or after it may be generated by
interpolation from images of the four types of angles of
view, and pseudo optical zoom may be achieved by
seamlessly generating images with various angles of view.
5 [0265]
Although the light-shielding film has been
described above, the description of this example can also
be applied to a case where an incident angle directivity
is given by selectively using a plurality of photodiodes
10 disposed in a pixel. That is, for example, by
appropriately setting a dividing position (size and shape
of each partial area), a position, a size, a shape, and
the like of each photodiode, or appropriately selecting a
photodiode, incident light directivity equivalent to the
15 incident light directivity by the light-shielding film
121b having a rectangular opening described above can be
achieved. Of course, also in this case, the imaging
element 121 can be achieved also by combining pixels 121a
having various angles of view. Further, an image with an
20 intermediate angle of view or with an angle of view
before or after it may be generated by interpolation from
images with a plurality of types of angles of view, and
pseudo optical zoom may be achieved by seamlessly
generating images of various angles of view.
25 [0266]

Incidentally, in a case where ranges shielded from
light by the light-shielding films 121b of pixels 121a in
the imaging element 121 have randomness, the larger the
30 randomness of differences in the ranges shielded from
light by the light-shielding films 121b, the larger the
100
load of processing by a restoration unit 321 and the
like. Accordingly, part of the differences in the ranges
shielded from light by the light-shielding films 121b of
the pixels 121a may be regularized to reduce this
randomness of differences, to 5 thereby reduce the
processing load.
[0267]
For example, horizontal band type light-shielding
films 121b having the same widths are combined in a
10 predetermined column direction, and vertical band type
light-shielding films 121b having the same heights are
combined in a predetermined row direction, so as to form
L type light-shielding films 121b combining a vertical
band type and a horizontal band type. In this manner,
15 light-shielding ranges of the light-shielding films 121b
of respective pixels 121a are set to different values at
random in pixel units while having regularity in the
column direction and the row direction. As a result,
randomness of differences in light-shielding ranges of
20 the light-shielding films 121b of respective pixels 121a,
that is, differences in incident angle directivities of
respective pixels can be reduced, and a processing load
outside the imaging element 121, such as that of the
restoration unit 321, can be reduced.
25 [0268]
For example, in a case of the imaging element 121''
in Fig. 21, horizontal band type light-shielding films
121b having the same widths X0 are used for all pixels in
the same column indicated by a range Z130, vertical band
30 type light-shielding films 121b having the same heights
Y0 are used for pixels in the same row indicated by a
101
range Z150, and L type light-shielding films 121b in
which they are combined are set for pixels 121a
identified by respective rows and columns.
[0269]
Similarly, horizontal band type 5 light-shielding
films 121b having the same widths X1 are used for all
pixels in the same column indicated by a range Z131
adjacent to the range Z130, vertical band type lightshielding
films 121b having the same heights Y1 are used
10 for pixels in the same row indicated by a range Z151
adjacent to the range Z150, and L type light-shielding
films 121b in which they are combined are set for pixels
121a identified by respective rows and columns.
[0270]
15 Moreover, horizontal band type light-shielding
films having the same widths X2 are used for all pixels
in the same column indicated by a range Z132 adjacent to
the range Z131, vertical band type light-shielding films
having the same heights Y2 are used for pixels in the
20 same row indicated by a range Z152 adjacent to the range
Z151, and L type light-shielding films 121b in which they
are combined are set for pixels 121a identified by
respective rows and columns.
[0271]
25 In this manner, while the light-shielding films
121b have regularity in widths and positions in the
horizontal direction as well as heights and positions in
the vertical direction, the ranges of the light-shielding
films can be set to different values in pixel units, and
30 thus it is possible to suppress the randomness of
differences in the incident angle directivities.
102
Consequently, patterns of coefficient sets can be
reduced, and a processing load of arithmetic processing
in a subsequent stage (for example, the restoration unit
321, and the like) can be reduced.
5 [0272]
More specifically, in a case where a restored image
with N × N pixels is obtained from a detection image Pic
with N pixels × N pixels as illustrated in an upper right
part of Fig. 22, a relationship illustrated in a left
10 part of Fig. 22 is established by a vector X having pixel
values of respective pixels of a restored image with N ×
N rows and one column as elements, a vector Y having
pixel values of respective pixels of a detection image
with N × N rows and one column as elements, and a matrix
15 A with N × N rows and N × N columns including coefficient
sets.
[0273]
That is, Fig. 22 illustrates that a result of
multiplying each element of the matrix A with N × N rows
20 and N × N columns including coefficient sets and the
vector X with N × N rows and one column that represents
the restored image becomes the vector Y with N × N rows
and one column that represents a detection image, and a
simultaneous equation is obtained from this relationship.
25 [0274]
Note that Fig. 22 illustrates that each element of
a first column indicated by a range Z201 of the matrix A
corresponds to an element of a first row of the vector X,
and each element of an N × N-th column indicated by a
30 range Z202 of the matrix A corresponds to an element of
an N × N-th row of the vector X.
103

CLAIMS
1. An imaging device comprising:
an imaging element that includes a plurality of
pixel output units that receives incident 5 light entering
without passing through either an imaging lens or a
pinhole, and that each outputs one detection signal
indicating an output pixel value modulated by an incident
angle of the incident light; and
10 an output processing unit that outputs a detection
image formed by a detection signal obtained in the pixel
output units of the imaging element, without associating
with a restoration matrix including coefficients used
when a restored image is restored from the detection
15 image.
2. The imaging device according to claim 1, wherein
the plurality of pixel output units has a
configuration in which an incident angle directivity
20 indicating a directivity of the output pixel value with
respect to an incident angle of incident light from an
object is independently settable in each of the pixel
output units.
25 3. The imaging device according to claim 1, wherein
the plurality of pixel output units has a
configuration in which an incident angle directivity
indicating a directivity with respect to an incident
angle of incident light from an object is independently
30 settable in each of the pixel output units.
184
4. The imaging device according to claim 1, wherein
the plurality of pixel output units has a
configuration in which, by having different photodiodes
(PD) from each other that contribute to output, an
incident angle directivity indicating 5 a directivity of
the output pixel value with respect to an incident angle
of incident light from an object is independently
settable in each of the pixel output units.
10 5. The imaging device according to claim 1, further
comprising
a readout control unit that controls reading out of
the detection signal from each of the pixel output units
of the imaging element and regularly or irregularly
15 switches the pixel output unit from which the detection
signal is read out.
6. The imaging device according to claim 1, further
comprising
20 a restoration matrix setting unit that sets the
restoration matrix in a case where a predetermined
condition is satisfied, wherein
the output processing unit is configured to output
the restoration matrix set by the restoration matrix
25 setting unit.
7. The imaging device according to claim 6, further
comprising
an encryption unit that encrypts the restoration
30 matrix set by the restoration matrix setting unit,
wherein
185
the output processing unit is configured to output
the restoration matrix encrypted by the encryption unit.
8. The imaging device according to claim 1, wherein
the detection image is an image 5 in which an object
is visually unrecognizable, and
the restored image is an image in which the object
is visually recognizable.
10 9. The imaging device according to claim 1, further
comprising
a restoration unit that restores the restored image
from the detection image using the restoration matrix,
wherein
15 the output processing unit is configured to display
the restored image restored by the restoration unit.
10. An imaging method comprising:
imaging an object by an imaging element including a
20 plurality of pixel output units that receives incident
light entering without passing through either an imaging
lens or a pinhole, and that each outputs one detection
signal indicating an output pixel value modulated by an
incident angle of the incident light; and
25 outputting a detection image obtained by the
imaging and formed by a detection signal obtained in the
pixel output units of the imaging element, without
associating with a restoration matrix including
coefficients used when a restored image is restored from
30 the detection image.
186
11. An image processing device comprising
a restoration unit that restores a restored image
from a detection image obtained by an external device
using a restoration matrix including coefficients used
when a restored image is restored 5 from the detection
image that is obtained by imaging an object by an imaging
element including a plurality of pixel output units and
is formed by a detection signal obtained in the pixel
output units, the plurality of pixel output units
10 receiving incident light entering without passing through
either an imaging lens or a pinhole, and each outputting
one detection signal indicating an output pixel value
modulated by an incident angle of the incident light.
15 12. The image processing device according to claim 11,
further comprising
a restoration matrix setting unit that sets a
restoration matrix used to restore the restored image by
the restoration unit, wherein
20 the restoration unit is configured to restore the
restored image from the detection image using the
restoration matrix set by the restoration matrix setting
unit.
25 13. The image processing device according to claim 12,
wherein
the restoration matrix setting unit regularly or
irregularly switches the restoration matrix used to
restore the restored image.
30
14. The image processing device according to claim 12,
187
wherein
the restoration matrix setting unit generates the
restoration matrix used to restore the restored image.
15. The image processing device according 5 to claim 12,
further comprising
a restoration matrix storage unit that stores
candidate restoration matrices, wherein
the restoration matrix setting unit is configured
10 to set the restoration matrix used to restore the
restored image from the restoration matrices stored in
the restoration matrix storage unit.
16. The image processing device according to claim 15,
15 further comprising
a restoration matrix communication unit that
obtains a restoration matrix from the external device by
communication, wherein
the restoration matrix storage unit is configured
20 to store the restoration matrix obtained by the
restoration matrix communication unit.
17. The image processing device according to claim 16,
further comprising
25 a decryption unit that decrypts an encrypted
restoration matrix obtained by the restoration matrix
communication unit, wherein
the restoration matrix storage unit is configured
to store the restoration matrix decrypted by the
30 decryption unit.
188
18. The image processing device according to claim 17,
wherein
the restoration matrix communication unit obtains
an encrypted restoration matrix associated with the
5 detection image.
19. The image processing device according to claim 16,
further comprising
an authentication unit that performs an
10 authentication process of itself for the external device,
wherein
the restoration matrix communication unit is
configured to obtain a restoration matrix from the
external device in a case where authentication by the
15 authentication unit is successful.
20. An image processing method comprising
restoring a restored image from a detection image
obtained by an external device using a restoration matrix
20 including coefficients used when a restored image is
restored from the detection image that is obtained by
imaging an object by an imaging element including a
plurality of pixel output units and is formed by a
detection signal obtained in the pixel output units, the
25 plurality of pixel output units receiving incident light
entering without passing through either an imaging lens
or a pinhole, and each outputting one detection signal
indicating an output pixel value modulated by an incident
angle of the incident light.