FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION (See section 10, rule 13)
“ELECTRONIC APPARATUS”
SONY CORPORATION, of 1-7-1, Konan, Minato-ku, Tokyo 108-0075, Japan
The following specification particularly describes the invention and the manner in which it is to
be performed.

the frame 412. [0276]
In the example in B of Fig. 27, the imaging element 121 is arranged near a bridge on a back surface of the frame 412. [0277]
In the example in C of Fig. 27, the imaging element 121-1 is arranged so as to surround the circumference of the left lens 411L on a back surface of the frame 412, and the imaging element 121-2 is arranged so as to surround the circumference of the right lens 411R on a back surface of the frame 412. [0278]
As described above, the imaging element 121 can be arranged in an empty space of the frame 412 of the wearable device 401 while a growth in size and a deterioration in design are suppressed. Furthermore, the imaging element 121 is arranged at a position where an area near both eyes of the user wearing the wearable device 401 is capturable, and can image the periphery of both eyes of this user. [0279]
Note that it is desirable that the incident angle directivity of each pixel 121a of each imaging element 121 in A to C of Fig. 27 be set such that the light receiving sensitivity with respect to the directions of the user’s eyes in proximity to the left lens 411L and the right lens 411R when the wearable device 401 is worn is higher. With such setting, in each pixel 121a, the light receiving sensitivity with respect to incident light from the directions of the user’s eyes, which are

the main imaging targets, is heightened, and the user’s
eyes can be imaged more clearly.
[0280]
For example, A and B of Fig. 28 schematically illustrate an example of the incident angle directivity of the imaging element 121-1 in C of Fig. 27. A and B of Fig. 28 schematically illustrate the positional relationship between a left eye 421L of the user in a state of wearing the wearable device 401, the left lens 411L, and the imaging element 121-1. A of Fig. 28 illustrates a diagram of the left lens 411L as viewed from the inside (the side of a surface facing the user’s eyes), and B of Fig. 28 illustrates a cross-sectional view of the left lens 411L as viewed from the side. [0281]
Note that the arrows in Fig. 28 indicate the tendency of the incident angle directivities of the pixels 121a in each region of the imaging element 121-1. Furthermore, hereinafter, in A and B of Fig. 28, a direction from the left lens 421L toward the left eye 421 (the right direction in B of Fig. 28) is assumed as a backward direction. [0282]
The pixel 121a in a region on an upper side of the imaging element 121-1 has an incident angle directivity set in a diagonally downward backward direction, and the light receiving sensitivity with respect to the direction of the left eye 421L is made higher. The pixel 121a in a region on a left side of the imaging element 121-1 has an incident angle directivity set in a diagonally backward right direction, and the light receiving sensitivity with

respect to the direction of the left eye 421L is made higher. The pixel 121a in a region on a right side of the imaging element 121-1 has an incident angle directivity set in a diagonally backward left direction, and the light receiving sensitivity with respect to the direction of the left eye 421L is made higher. The pixel 121a in a region on a lower side of the imaging element 121-1 has an incident angle directivity set in a diagonally upward backward direction, and the light receiving sensitivity with respect to the direction of the left eye 421L is made higher. [0283]
With such setting, in the imaging element 121-1, the light receiving sensitivity with respect to incident light from the direction of the user’s left eye 421L, which is the main imaging target, is heightened, and the left eye 421L can be imaged more clearly. [0284]
Note that, although illustration is omitted, also in the imaging element 121-2 arranged around the right lens 411R, the incident angle directivity of each pixel 121a is set in the direction of the user’s right eye 421R, which is the main imaging target. With such setting, the light receiving sensitivity with respect to incident light from the user’s right eye 421R is heightened, and the right eye 421R can be imaged more clearly. [0285]
Note that, among examples in A to C of Fig. 27, from the viewpoint of the image quality of the restoration image, it is more desirable to arrange the

imaging element 121 as in C of Fig. 27, and making variations in the incident angle directivities of the respective pixels 121a. This is because making variations in the incident angle directivities of the respective pixel 121a heightens the diversity of coefficient set groups for the simultaneous equations used to restore the restoration image, and improve the restoration accuracy. [0286]
In addition, the incident angle directivities of all the pixels 121a need not necessarily be set such that the light receiving sensitivity with respect to the directions of the user’s eyes is higher. For example, it is only required that the number of pixels 121a having incident angle directivity that heightens the light receiving sensitivity with respect to the directions of the user’s eyes is larger than the number of pixels 121a having other incident angle directivities. [0287]
A to C of Fig. 29 schematically illustrate a camera 431 as an example of the electronic instrument 301. Note that A to C of Fig. 29 illustrate only the periphery of a viewfinder 441 of the camera 431; for example, a display unit such as a display or a touch panel and a user operation unit are provided on a rear surface of the camera 431 (a surface on the same side as the viewfinder 441). [0288]
Furthermore, A to C of Fig. 29 illustrate examples in which the respective imaging elements 121 (respective pixel 121a of the respective imaging elements 121) are

arranged around the viewfinder 441, which is an eyepiece
unit of the camera 431.
[0289]
Specifically, in the example in A of Fig. 29, the imaging element 121-1 is arranged in a longitudinally long rectangular region on a left side of the viewfinder 441, and the imaging element 121-2 is arranged in a longitudinally long rectangular region on a right side of the viewfinder 441. [0290]
In the example in B of Fig. 29, the imaging element 121 is arranged in a laterally long rectangular region above the viewfinder 441. [0291]
In the example in C of Fig. 29, the imaging element 121 is arranged in an L-shaped region above and on a left side of the viewfinder 441. [0292]
In the example in D of Fig. 29, the imaging element 121-1 is arranged in a laterally long rectangular region above the viewfinder 441, and the imaging element 121-2 is arranged in a laterally long rectangular region below the viewfinder 441. [0293]
In the example in E of Fig. 29, the imaging element 121 is arranged so as to surround the circumference of the viewfinder 441. [0294]
As described above, the imaging element 121 can be arranged in an empty space around the viewfinder 441 of the camera 431 while a growth in size and a deterioration

in design are suppressed. Furthermore, the imaging element 121 is arranged at a position where the periphery of the user’s eye is capturable in a case where this user looks into the viewfinder 441 in order to capture an image using the camera 431, and can image the periphery of the eye of this user. [0295]
Note that it is desirable that the incident angle directivity of each pixel 121a of each imaging element 121 in A to E of Fig. 29 be set such that the light receiving sensitivity with respect to the direction of the user’s eye in proximity to the viewfinder 441 when the viewfinder 441 is looked into (for example, ahead of the viewfinder 441) is higher. [0296]
A to C of Fig. 30 schematically illustrate a part of a goggles-type head mounted display 461 that is worn so as to cover the eyes of the user, as an example of the electronic instrument 301. Note that A to C of Fig. 30 illustrate an inner surface of the head mounted display 461, that is, a surface facing the user’s face when the user wears the head mounted display 461, and illustrate an example in which the respective imaging element 121 (respective pixels 121a of the respective imaging elements 121) are arranged around a left lens 471L and a right lens 471R, which form an eyepiece unit of the head mounted display 461. [0297]
Specifically, in the example in A of Fig. 30, the imaging element 121-1 is arranged in a rectangular region on a left side of the left lens 471L, and the imaging

element 121-2 is arranged in a rectangular region on a
right side of the right lens 471R.
[0298]
In the example in B of Fig. 30, the imaging element 121-1 is arranged in a laterally long rectangular region above the left lens 471L, and the imaging element 121-2 is arranged in a laterally long rectangular region above the right lens 471R. [0299]
In the example in C of Fig. 30, the imaging element 121-1 is arranged in a laterally long rectangular region below the left lens 471L, and the imaging element 121-2 is arranged in a rectangular region below the right lens 471R. [0300]
As described above, the imaging element 121 can be arranged in an empty space around the left lens 471L and the right lens 471R of the head mounted display 461 while a growth in size and a deterioration in design are suppressed. Then, for example, the periphery of both eyes of the user wearing the head mounted display 461 can be imaged. [0301]
Note that, similarly to the wearable device 401 in Fig. 27, it is desirable that the incident angle directivity of each pixel 121a of each imaging element 121 in A to C of Fig. 30 be set such that the light receiving sensitivity with respect to the directions of the user’s eyes in a state in which the head mounted display 461 is worn is higher. [0302]

A to C of Fig. 31 schematically illustrate a notebook personal computer (PC) 491 as an example of the electronic instrument 301. [0303]
In the PC 491, a lid portion provided with a display 501, which is a display unit, and a bottom portion provided with a keyboard 503, which is a user operation unit, are rotatably connected by a hinge portion to allow the lid portion to be opened and closed. Then, the display 501 and the keyboard 503 are exposed to the outside in a state in which the lid portion is opened as illustrated in A to C of Fig. 31, and the user can use the PC 491. Note that, for example, by providing a touch panel on the display 501, the display 501 (strictly, the touch panel) can be employed as a user operation unit. [0304]
Furthermore, A to C of Fig. 31 illustrate examples in which the imaging elements 121 (respective pixel 121a of the imaging elements 121) are arranged around the display 501. [0305]
In the example in A of Fig. 31, the imaging element 121 is arranged on an upper side of a bezel 502 around the display 501. [0306]
In the example in B of Fig. 31, the imaging element 121-1 is arranged on a left side of the bezel 502, and the imaging element 121-2 is arranged on a right side of the bezel 502. [0307]
In the example in C of Fig. 31, the imaging element

121 is arranged on four sides of the bezel 502 so as to
enclose the display 501.
[0308]
As described above, the imaging element 121 can be arranged in an empty space of the bezel 502, and a narrower bezel can be implemented while a growth in size and a deterioration in design are suppressed. Furthermore, the imaging element 121 is arranged at a position directly in front of the user using the keyboard 503 while looking at the display 501 of the PC 491, where the periphery of the user’s face is capturable, and can image the periphery of the face of this user. [0309]
Note that it is desirable that the incident angle directivity of each pixel 121a of each imaging element 121 in A to C of Fig. 31 be set such that the light receiving sensitivity with respect to the direction of the face of the user using the keyboard 503 while looking at the display 501 of the PC 491 (for example, ahead of the display 501) is higher. [0310]

Next, a user imaging control process executed by the electronic instrument 301 will be described with reference to a flowchart in Fig. 32. [0311]
In step S101, each imaging element 121 of the electronic instrument 301 images the user by a process similar to the process in step S1 in Fig. 20. [0312]
For example, in the case of the wearable device 401

in Fig. 27, the periphery of both eyes of the user wearing the wearable device 401 is imaged. [0313]
In the case of the camera 431 in Fig. 29, the periphery of the eye of the user looking into the viewfinder 441 is imaged. [0314]
In the case of the head mounted display 461 in Fig. 30, the periphery of both eyes of the user wearing the head mounted display 461 is imaged. [0315]
In the case of the PC 491 in Fig. 31, the periphery of the face of the user looking at the display 501 is imaged. [0316]
Each imaging element 121 supplies a detection signal set including a detection signal of each pixel 121a to the association unit 325. [0317]
In step S102, the restoration unit 321 finds a coefficient used for image restoration. Specifically, the restoration unit 321 sets the subject distance by the process similar to the process by the restoration unit 122 of the imaging apparatus 101 in step S2 in Fig. 20. Then, the restoration unit 321 reads a coefficient set group in correspondence with the set subject distance from the storage unit 327. [0318]
In step S103, the restoration unit 321 restores an image using the detection signal set and the coefficients. That is, the restoration unit 321 restores

the restoration image using the detection signal set output from each imaging element 121 and the coefficient set group found in the process in step S102, by a process similar to the process by the restoration unit 122 of the imaging apparatus 101 in step S3 in Fig. 20. [0319]
In step S104, the electronic instrument 301 performs various processes on the restoration image. For example, the restoration unit 321 performs a demosaic process, γ correction, white balance adjustment, a conversion process to a predetermined compression format, and the like on the restoration image as necessary. Furthermore, the restoration unit 321, for example, supplies the restoration image to the output unit 326 to display, or supplies the restoration image to the recording/playback unit 328 to record on the recording medium 329, or outputs the restoration image to other instruments via the communication unit 330, as necessary. [0320]
In step S105, the electronic instrument 301 executes an application process using the restoration image (that is, the user’s image). [0321]
For example, in the case of the wearable device 401 in Fig. 27, the control unit 322 detects the user’s line of sight on the basis of the image of the user’s eyes in the restoration image. Note that an arbitrary method can be used for line-of-sight detection. Then, for example, the control unit 322 generates an operation command corresponding to the movement of the user’s line of sight, and transmits the generated operation command to

another electronic instrument (not illustrated) via the communication unit 330. Therefore, the user can operate the another electronic instrument merely by wearing the wearable device 401 and moving his/her line of sight. [0322]
For example, in the case of the camera 431 in Fig. 29, the control unit 322 detects the user’s line of sight on the basis of the image of the user’s eye in the restoration image. Note that an arbitrary method can be used for line-of-sight detection. Then, for example, the control unit 322 controls each unit of the camera 431 to move the position of a subject to be focused (focus point) according to the movement of the user’s line of sight, or to perform various processes of the camera 431. Therefore, the user can, for example, set the focus point and perform various operations on the camera 431 merely by looking into the viewfinder 441 and moving his/her line of sight. [0323]
Alternatively, for example, the control unit 322 performs a user recognition process or authentication process on the basis of inside the restoration image. [0324]
Here, the recognition process is, for example, a process of specifying a user or recognizing a user’s features. Meanwhile, the authentication process is a process of, for example, judging whether or not a user is a user who is registered in advance or a legitimate user or the like, for example, by collating the restoration image with an image registered in advance (for example, a face image or an eye image). Note that the recognition

process and the authentication process sometimes partially overlap without being definitely distinguished. Furthermore, an arbitrary method can be used for the user recognition process and authentication process. For example, various types of biometric authentication such as face authentication and iris authentication can be used for the recognition process. [0325]
Then, for example, the control unit 322 displays a user interface screen corresponding to the recognized or authenticated user on the display of the output unit 326 on the basis of the user recognition result or authentication result, or alters the setting of the camera 431, or gives permission to use a specific function (for example, playback and the like). [0326]
For example, in the case of the head mounted display 461 in Fig. 30, the control unit 322 detects the user’s line of sight on the basis of the image of the user’s eyes in the restoration image. Note that an arbitrary method can be used for line-of-sight detection. Then, for example, the control unit 322 controls each unit of the head mounted display 461 to perform various processes of the head mounted display 461 according to the movement of the user’s line of sight. Therefore, the user can perform various operations on the head mounted display 461 merely by wearing the head mounted display 461 and moving his/her line of sight. [0327]
For example, in the case of the PC 491 in Fig. 31, the control unit 322 performs a user recognition process

or authentication process on the basis of the image of the user’s face in the restoration image. Then, for example, the control unit 322 displays a user interface screen corresponding to the recognized or authenticated user on the display of the output unit 326 on the basis of the user recognition result or authentication result, or alters the setting or the like of the PC 491 (for example, custom settings, image quality settings, and parental control settings), or gives permission to log in to a specific account, to access a specific folder or file, to use a specific function, and the like. [0328]
Thereafter, the user imaging control process ends. [0329]
Note that, in the above, an example in which the restoration image is restored from the detection signal set using the coefficient set group in correspondence with the imaging element 121 and the subject distance has been described; however, as described earlier, for example, a coefficient set group corresponding to the angle of view of the restoration image may be further prepared in addition to a coefficient set group corresponding to the subject distance such that the restoration image is restored using the coefficient set groups corresponding to the subject distance and the angle of view. [0330]
By configuring as described above, the imaging element 121 can be arranged while a growth in size and a deterioration in design of the electronic instrument 301 are suppressed, and a user using the electronic

instrument can be imaged. Then, the image of the user can be restored, and various application processes can be executed on the basis of the restored image. [0331]
<<4. Second Embodiment>>
Next, a second embodiment of the present disclosure will be described with reference to Figs. 33 to 41. [0332]
In the first embodiment described above, an example in which a user using the electronic instrument 301 is imaged and various application processes are executed using the image of the user obtained by the restoration process is indicated. Meanwhile, in the second embodiment, the surroundings of a user using the electronic instrument 301 are imaged and various application processes are executed using the image of the surroundings of the user obtained by a restoration process. [0333]
Note that, in the second embodiment, the electronic instrument 301 in Fig. 26 is used as in the first embodiment. On the other hand, in the second embodiment, unlike the first embodiment, the imaging element 121 is arranged at a position where the surroundings of the user wearing or using the electronic instrument 301 are capturable. [0334]

Here, arrangement examples of the imaging element 121 will be described with reference to Figs. 33 to 40, giving specific examples of the electronic instrument

301. [0335]
A to C of Fig. 33 schematically illustrate a glasses-type wearable device 601 that is worn so as to cover the eyes of the user, as an example of the electronic instrument 301. Furthermore, A to C of Fig. 33 illustrate examples in which the imaging elements 121 (respective pixels 121a of the imaging elements 121) are arranged on a front surface of a frame 612 exposed to the outside in a state in which the wearable device 601 is worn by the user. [0336]
Specifically, in the example in A of Fig. 33, the imaging element 121-1 is arranged above a left lens 611L on a front surface of the frame 612, and the imaging element 121-2 is arranged above a right lens 611R on a front surface of the frame 612. [0337]
In the example in B of Fig. 33, the imaging element 121-1 is arranged on a right side of the left lens 611L on a front surface of the frame 612, and the imaging element 121-2 is arranged on a left side of the right lens 611R on a front surface of the frame 612. [0338]
In the example in C of Fig. 33, the imaging element 121-1 is arranged so as to surround the circumference of the left lens 611L on a front surface of the frame 612, and the imaging element 121-2 is arranged so as to surround the circumference of the right lens 611R on a front surface of the frame 612. [0339]

As described above, the imaging element 121 can be arranged in an empty space of the frame 612 of the wearable device 601 while a growth in size and a deterioration in design are suppressed. Then, for example, an area ahead of the user wearing the wearable device 601 can be imaged. [0340]
A to C of Fig. 34 schematically illustrate a camera 631 as an example of the electronic instrument 301. Furthermore, examples in which the imaging elements 121 (respective pixels 121a of the imaging elements 121) are arranged on a forward surface of a casing of the camera 631 are illustrated. [0341]
Specifically, in the example in A of Fig. 34, the imaging element 121 is arranged on a forward surface of a main body portion of the camera 631 in a longitudinally long rectangular region on a left side of a mount 641 near a left end of the camera 631. [0342]
In the example in B of Fig. 34, the respective imaging elements 121-1 to 121-4 are arranged on a forward surface of the main body portion of the camera 631 in four rectangular regions near four corners outside the mount 641. [0343]
In the example in C of Fig. 34, the imaging element 121 is arranged on a forward surface of a flash built-in unit 642 in which a flash of the camera 631 is built. [0344]
In the example in D of Fig. 34, the imaging element

121 is arranged in a ring-shaped region along an outer
periphery of the mount 641 of the camera 631.
[0345]
As described above, the imaging element 121 can be arranged in an empty space on a forward surface of the casing of the camera 631 while a growth in size and a deterioration in design are suppressed. Then, for example, an area in an imaging direction of the camera 631 can be imaged. [0346]
A to D of Fig. 35 schematically illustrate a goggles-type head mounted display 661 that is worn so as to cover the eyes of the user, as an example of the electronic instrument 301. Furthermore, A to D of Fig. 35 illustrate examples in which the imaging elements 121 (respective pixels 121a of the imaging elements 121) are arranged on a forward surface of a casing exposed to the outside in a state in which the head mounted display 661 is worn by the user. [0347]
Specifically, in the example in A of Fig. 35, the imaging element 121 is arranged in a laterally long rectangular region on a lower side of a forward surface of a main body portion 671. [0348]
In the example in B of Fig. 35, the imaging element 121 is arranged in a laterally long region at an upper end of a forward surface of the main body portion 671. [0349]
In the example in C of Fig. 35, the imaging element 121 is arranged in a rectangular region on a forward

surface of a head pad 672. [0350]
In the example in D of Fig. 35, the imaging elements 121-1 and 121-2 are arranged in rectangular regions on the left and right of a forward surface of the main body portion 671. [0351]
As described above, the imaging element 121 can be arranged in an empty space on a forward surface of the casing of the head mounted display 661 while a growth in size and a deterioration in design are suppressed. Then, for example, an area ahead of the user wearing the head mounted display 661 can be imaged. [0352]
A of Fig. 36 to D of Fig. 37 schematically illustrate overhead headphones 691 as an example of the electronic instrument 301. A and B of Fig. 36 illustrate perspective views of the headphones 691 as viewed diagonally from the front, and C and D of Fig. 37 illustrate perspective views of the headphones 691 as viewed diagonally from behind. [0353]
In the example in A of Fig. 36, the imaging element 121-1 is arranged near a forward center of a side surface of a housing 701L on the left side, and the imaging element 121-2 is arranged near a forward center of a side surface of a housing 701R on the right side. [0354]
In the example in B of Fig. 36, the imaging element 121 is arranged in a region along a forward surface of a headset 702.

[0355]
In the example in C of Fig. 37, the imaging element 121-1 is arranged near a backward center of a side surface of the housing 701L on the left side, and the imaging element 121-2 is arranged near a backward center of a side surface of the housing 701R on the right side. [0356]
In the example in D of Fig. 37, the imaging element 121 is arranged in a region along a backward surface of the headset 702. [0357]
As described above, the imaging element 121 can be arranged in an empty space on a surface exposed to the outside in a state in which the headphones 691 are worn by the user, while a growth in size and a deterioration in design are suppressed. Then, for example, an area ahead of or behind the user wearing the headphones 691 can be imaged. [0358]
A and B of Fig. 38 schematically illustrate neckband headphones 721 as an example of the electronic instrument 301. [0359]
In the example in A of Fig. 38, the imaging element 121-1 is arranged in a forward portion of a side surface of a housing 731L on the left side. Furthermore, although illustration is omitted, the imaging element 121-2 is arranged in a forward portion of a side surface of a housing 731R on the right side. [0360]
In the example in B of Fig. 38, the imaging element

121 is arranged near a backward portion of a neckband
732.
[0361]
As described above, the imaging element 121 can be arranged in an empty space on a surface exposed to the outside in a state in which the headphones 721 are worn by the user, while a growth in size and a deterioration in design are suppressed. Then, for example, an area ahead of or behind the user wearing the headphones 721 can be imaged. [0362]
Note that, in the above examples, an example in which an area ahead of or behind the user is imaged is indicated; however, the respective imaging elements 121 may be installed so as to image other directions around the user (for example, areas on sides of, above, and below the user). [0363]

Next, a user surrounding imaging control process executed by the electronic instrument 301 will be described with reference to a flowchart in Fig. 39. [0364]
In step S201, each imaging element 121 of the electronic instrument 301 images the surroundings of the user by a process similar to the process in step S1 in Fig. 20. [0365]
For example, in the case of the wearable device 601 in Fig. 33, an area ahead of the user wearing the wearable device 601 is imaged.

[0366]
In the case of the camera 631 in Fig. 34, an area in the imaging direction of the camera 631 is imaged. [0367]
In the case of the head mounted display 661 in Fig. 35, an area ahead of the user wearing the head mounted display 661 is imaged. [0368]
In the case of the headphones 691 in Figs. 36 and 37, an area ahead of or behind the user wearing the headphones 691 is imaged. [0369]
In the case of the headphones 721 in Fig. 38, an area ahead of or behind the user wearing the headphones 721 is imaged. [0370]
Each imaging element 121 supplies a detection signal set including a detection signal of each pixel 121a to the association unit 325. [0371]
In step S202, the restoration unit 321 finds a coefficient used for image restoration. Specifically, the restoration unit 321 sets the subject distance by the process similar to the process by the restoration unit 122 of the imaging apparatus 101 in step S2 in Fig. 20. Then, the restoration unit 321 reads a coefficient set group in correspondence with the set subject distance from the storage unit 327. [0372]
Note that, in a case where the imaging element 121 is arranged in a deformable part like the headset 702 of

the headphones 691 in Figs. 36 and 37 and the neckband 732 of the headphones 721 in Fig. 38, the relative position between the respective pixels 121a changes as the deformation occurs. [0373]
Meanwhile, the coefficient set group of simultaneous equations used for restoring the restoration image described above is set on the assumption that the relative position between the respective pixels 121a does not change. [0374]
Thus, in a case where the imaging element 121 is arranged in a deformable part of the electronic instrument 301, a coefficient set group according to the deformation status may be prepared in advance, and additionally, the deformation status may be detected such that a coefficient set group according to the detected deformation status is used. [0375]
Here, the deformable part is a part that can vary its shape in a case where, for example, the electronic instrument 301 is used or worn. Note that the type of deformation is not particularly limited, and includes, for example, stretching, shrinking, bending, shifting, twisting, separation, and the like. Furthermore, the deformable part includes, for example, a part that is deformed by a force imparted from the outside, and a part that actively varies its shape by an actuator or the like. [0376]
For example, a bending sensor may be provided in

the headset 702 of the headphones 691 illustrated in Figs. 36 and 37 such that a coefficient set group corresponding to the bending status of the headset detected by the bending sensor is used. [0377]
For example, coefficient set groups corresponding to three patterns of bending statuses of a bending sensor 751 in A to C of Fig. 40 are prepared in advance. Then, the restoration image is restored using a coefficient set group corresponding to a bending status closest to the detection result of the bending sensor 751 in a case where the user wears the headphones 691. [0378]
In step S203, an image is restored using the detection signal set and the coefficients, as in the process in step S103 in Fig. 32. [0379]
In step S204, as in the process in step S104 in Fig. 32, various processes are executed on the restoration image. [0380]
In step S205, the electronic instrument 301 executes an application process using the restoration image (that is, an image of the surroundings of the user). [0381]
For example, in the case of the wearable device 601 in Fig. 33, the control unit 322 performs a recognition process for the surroundings of the user on the basis of the restoration image. Note that an arbitrary method can be used for the recognition process for the surroundings

of the user. Then, for example, the control unit 322 controls the output unit 326 to superimpose an image and information corresponding to the environment around the user obtained by the recognition process, onto an area within the field of view visually recognized by the user via the left lens 611L and the right lens 611R, thereby implementing augmented reality (AR). [0382]
Alternatively, for example, the control unit 322 controls the recording/playback unit 328 to record the restoration image on the recording medium 329 as a user life log. [0383]
For example, in the case of the camera 631 in Fig. 34, the control unit 322 performs subject tracking, scene recognition, and the like by performing the recognition process for the surroundings of the user on the basis of the restoration image. Note that an arbitrary method can be used for the recognition process for the surroundings of the user. Therefore, for example, the subject can be tracked even during the working of a shutter of an imaging element (not illustrated) that performs imaging via a lens 643 of the camera 631. Furthermore, for example, subject tracking and scene recognition can be performed by sensing a range wider than the lens 643. [0384]
For example, in the case of the head mounted display 661 in Fig. 35, the control unit 322 controls the output unit 326 to superimpose a part or the whole of the restoration image onto an image that the user is visually recognizing.

[0385]
For example, in the case of the headphones 691 in Figs. 36 and 37 or the headphones 721 in Fig. 38, the control unit 322 performs the recognition process for the surroundings of the user on the basis of the restoration image. Note that an arbitrary method can be used for the recognition process for the surroundings of the user. Then, on the basis of the recognition result, the control unit 322 controls the output unit 326 and the like to, for example, assist the user. For example, a danger such as a vehicle coming closer is notified by vibration, sound, or the like in a case where the danger is detected, support by sound and the like is provided for the visually impaired, and the name of a recognized person is notified by sound. [0386]
Alternatively, for example, the control unit 322 controls the recording/playback unit 328 to record the restoration image on the recording medium 329 as a user life log. [0387]
Thereafter, the user surrounding imaging control process ends. [0388]
Note that, in the above, an example in which the restoration image is restored from the detection signal set using the coefficient set group in correspondence with the imaging element 121 and the subject distance has been described; however, as described earlier, for example, a coefficient set group corresponding to the angle of view of the restoration image may be further

prepared in addition to a coefficient set group corresponding to the subject distance such that the restoration image is restored using the coefficient set groups corresponding to the subject distance and the angle of view. [0389]
By configuring as described above, the imaging element 121 can be arranged while a growth in size and a deterioration in design of the electronic instrument 301 are suppressed, and the surroundings of a user using the electronic instrument can be imaged. Then, the image of the surroundings of the user can be restored, and various application processes can be executed on the basis of the restored image. [0390]
Note that, in the above, an example in which the restoration image is recorded as a life log is indicated; however, the detection signal set may be recorded instead of the restoration image such that an image is restored as necessary. [0391]
In this case, for example, the detection signal set acquired by each imaging element 121 is associated with metadata corresponding to each detection signal set. [0392]
Note that the metadata may or may not include the coefficient set group at the time of restoration, for example. In the latter case, for example, the subject distance, angle of view, deformation information on a part where the imaging element 121 is arranged, and the like used at the time of restoration are included in the

metadata. [0393]
Furthermore, the method for associating the detection signal set with metadata is not particularly limited as long as the correspondence between the detection signal set and the metadata can be specified. For example, the detection signal set is associated with metadata by adding metadata to data including the detection signal set, giving the same ID to the detection signal set and metadata, or recording the detection signal set and metadata on the same recording medium 329. In addition, metadata may be associated with each detection signal set individually, or metadata may be associated with data in which the detection signal sets are gathered into one piece. [0394]
<<5. Variations>>
Hereinafter, variations of the above-described embodiments of the present disclosure will be described. [0395]

Fig. 26 illustrates an example in which one electronic instrument 301 executes the imaging process for the user or the surroundings of the user, the restoration process for the restoration image, and the application process using the restoration image; however, these processes may be divided and allocated to two or more apparatuses. [0396]
For example, an information processing system 801

in Fig. 41 includes an electronic instrument 811 and a signal processing apparatus 812. Then, the electronic instrument 811 and the signal processing apparatus 812 may perform the imaging process, the restoration process, and the application process by each taking a share thereof. [0397]
For example, the electronic instrument 811 may perform the imaging process, and the signal processing apparatus 812 may perform the restoration process and application process. Alternatively, for example, the electronic instrument 811 may perform the imaging process and application process, and the signal processing apparatus 812 may perform the restoration process. Alternatively, the electronic instrument 811 and the signal processing apparatus 812 may perform some processes out of the imaging process, the restoration process, and the application process in coordination. [0398]
Furthermore, the present disclosure can be applied to an electronic instrument other than the above-described electronic instruments, having a function of imaging a user or the surroundings of the user. In an electronic instrument having a function of imaging the user, for example, as in the above-described examples, the imaging element 121 is arranged around the eyepiece unit or the display unit. In addition, in an electronic instrument having a function of imaging the surroundings of the user, for example, as in the above-described examples, the imaging element 121 is arranged on a surface exposed to the outside in a state in which the

user is wearing the electronic instrument. Note that the arrangement of the imaging element 121 is not limited to the above-described examples, and can be appropriately altered according to the main imaging target and the like. [0399]
In addition, for example, the imaging element 121 also can be arranged not around the eyepiece unit or around the display unit of the electronic instrument but in the eyepiece unit or the display unit. For example, the imaging elements 121 can be arranged on front surfaces of the left lens 411L and the right lens 411R in Fig. 27, a front surface of the viewfinder 441 in Fig. 29, front surfaces of the left lens 471L and the right lens 471R in Fig. 30, a front surface of the display 501 in Fig. 31, and the like. [0400]

Furthermore, for example, the shape of the light shielding film 121b of each pixel 121a can adopt a shape other than the above-described lateral belt type, longitudinal belt type, L-shaped type, or type having a rectangular opening portion. [0401]
Moreover, for example, in the imaging element 121 described above with reference to Fig. 5, an example in which four photodiodes 121f in 2 rows × 2 columns are provided in one pixel 121a is indicated; however, the number and arrangement of the photodiodes 121f are not limited to this example.

[0402]
For example, as illustrated in Fig. 42, nine photodiodes 121f-111 to 121f-119 placed in 3 rows × 3 columns may be provided in one pixel 121a for one on-chip lens 121c. That is, one pixel output unit may include nine photodiodes 121f. [0403]
Then, for example, by not reading signals from five pixels, namely, the photodiodes 121f-111, 121f-114, 121f-117 to 121f-119, an incident angle characteristic substantially similar to an incident angle characteristic of a pixel 121a including the L-shaped-type light shielding film 121b in which the light shielding film 121b is set in a range of the photodiodes 121f-111, 121f-114, 121f-117 to 121f-119 can be obtained. [0404]
By configuring in this manner, an incident angle characteristic similar to a case where the light shielding film 121b is provided can be obtained without providing the light shielding film 121b. Furthermore, by switching the pattern of the photodiodes 121f from which no signal is read, the incident angle directivity can be changed as in a case where the position and range shielded by the light shielding film 121b are varied. [0405]
In addition, in the above description, an example in which one pixel output unit is constituted by one pixel 121a is indicated; however, one pixel output unit also can be constituted by a plurality of pixels 121a. [0406]
For example, as illustrated in Fig. 43, one pixel

output unit 851b can be constituted by pixels 121a-111 to 121a-119 placed in 3 rows × 3 columns. Note that each of the pixels 121a-111 to 121a-119 includes, for example, one photodiode and does not include an on-chip lens. [0407]
For example, by adding pixel signals from the respective pixels 121a, a detection signal for one pixel of the detection image is generated, and additionally by stopping the output of pixel signals from some pixels 121a or not adding pixel signals from some pixels 121a, the incident angle directivity of the pixel output unit 851b can be implemented. For example, by adding pixel signals of the pixels 121a-112, 121a-113, and 121a-115, and pixel 121a-116 to generate a detection signal, an incident angle directivity similar to a case where the L-shaped-type light shielding film 121b is provided in the range of the pixels 121a-111, pixel 121a-114, and 121a-117 to 121a-119 can be obtained. [0408]
In addition, by switching the pattern of the pixels 121a whose pixel signals are to be added to the detection signal, the incident angle directivity can be set to a different value as in a case where the position and the range shielded by the light shielding film 121b are varied. [0409]
Furthermore, in this case, for example, the range of the pixel output unit can be altered by altering the combination of the pixels 121a. For example, a pixel output unit 851s can be constituted by pixels 121a in 2 rows × 2 columns including pixels 121a-111, 121a-112,

121a-114, and 121a-115. [0410]
Moreover, for example, the range of the pixel output unit can be set later by recording the pixel signals of all the pixels 121a and setting the combination of the pixels 121a later. Besides, by selecting a pixel 121a whose pixel signal to be added to the detection signal, from among the pixels 121a in the set pixel output unit, the incident angle directivity of the pixel output unit can be set later. [0411]
In addition, Fig. 4 illustrates an example in which different incident angle directivities are provided for each pixel by using the light shielding film 121b as a modulation element or altering the combination of photodiodes that contribute to the output; in the present disclosure, however, for example, as illustrated in Fig. 44, an optical filter 902 that covers the light receiving surface of an imaging element 901 also can be used as a modulation element such that each pixel is provided with the incident angle directivity. [0412]
Specifically, the optical filter 902 is arranged at a predetermined interval from a light receiving surface 901A of the imaging element 901 so as to cover the entire surface of the light receiving surface 901A. Light from the subject surface 31 is modulated by the optical filter 902 and then is incident on the light receiving surface 901A of the imaging element 901. [0413]
For example, as the optical filter 902, an optical

filter 902BW having a grating-like black and white pattern illustrated in Fig. 45 can be used. In the optical filter 902BW, a white pattern portion that transmits light and a black pattern portion that provides shielding from light are randomly arranged. The size of each pattern is set independently of the pixel size of the imaging element 901. [0414]
Fig. 46 illustrates the light receiving sensitivity characteristics of the imaging element 901 with respect to light from point light sources PA and PB on the subject surface 31 in a case where the optical filter 902BW is used. Each ray of light from the point light sources PA and PB is modulated by the optical filter 902BW, and then is incident on the light receiving surface 901A of the imaging element 901. [0415]
For example, the light receiving sensitivity characteristic of the imaging element 901 with respect to the light from the point light source PA is as a waveform Sa. That is, since a shadow is produced by the black pattern portion of the optical filter 902BW, a shade pattern is produced in a figure on the light receiving surface 901A for the light from the point light source PA. Similarly, the light receiving sensitivity characteristic of the imaging element 901 with respect to the light from the point light source PB is as a waveform Sb. That is, since a shadow is produced by the black pattern portion of the optical filter 902BW, a shade pattern is produced in a figure on the light receiving surface 901A for the light from the point light source

PB. [0416]
Note that the light from the point light source PA and the light from the point light source PB have different incident angles with respect to each white pattern portion of the optical filter 902BW, such that a shift is produced in a way how a shade pattern appears on the light receiving surface. Accordingly, each pixel of the imaging element 901 is provided with an incident angle directivity with respect to each point light source on the subject surface 31. [0417]
Details of this scheme are disclosed, for example, in Non-Patent Document 1 described above. [0418]
Note that an optical filter 902HW in Fig. 47 may be used instead of the optical filter 902BW. The optical filter 902HW includes a linear polarizing element 911A and a linear polarizing element 911B having equal polarization directions, and a half-wave plate 912, where the half-wave plate 912 is sandwiched between the linear polarizing elements 911A and 911B. The half-wave plate 912 is provided with a polarizing portion indicated by hatching instead of the black pattern portion of the optical filter 902BW, and the white pattern portion and the polarizing portion are randomly arranged. [0419]
The linear polarizing element 911A transmits only light in a predetermined polarization direction out of approximately non-polarized light emitted from the point light source PA. Hereinafter, it is assumed that the

linear polarizing element 911A transmits only light whose polarization direction is parallel to the drawing. The polarization direction of the polarized light transmitted through the polarizing portion of the half-wave plate 912, out of the polarized light transmitted through the linear polarizing element 911A, is changed in a direction perpendicular to the drawing because the polarization plane is rotated. On the other hand, the polarization direction of the polarized light transmitted through the white pattern portion of the half-wave plate 912, out of the polarized light transmitted through the linear polarizing element 911A, remains unchanged in a direction parallel to the drawing. Then, the linear polarizing element 911B transmits the polarized light that has been transmitted through the white pattern portion, and hardly transmits the polarized light that has been transmitted through the polarizing portion. Accordingly, the light amount of the polarized light transmitted through the polarizing portion is lessened, as compared with the polarized light transmitted through the white pattern portion. Therefore, a shade pattern approximately similar to a case where the optical filter BW is used is produced on the light receiving surface 901A of the imaging element 901. [0420]
Furthermore, as illustrated in A of Fig. 48, an optical interference mask can be used as an optical filter 902LF. The light receiving surface 901A of the imaging element 901 is irradiated with light emitted from the point light sources PA and PB on the subject surface 31 via the optical filter 902LF. As illustrated in the

enlarged view on a lower side of A of Fig. 48, for example, a light incident surface of the optical filter 902LF is provided with irregularities to an extent equivalent to the wavelength. In addition, the optical filter 902LF maximizes the transmission of light having a specific wavelength radiated from a perpendicular direction. When a change in the incident angle (tilt with respect to the perpendicular direction) of light having a specific wavelength emitted from the point light sources PA and PB on the subject surface 31 with respect to the optical filter 902LF increases, the optical path length changes. Here, when the optical path length is an odd multiple of the half wavelength, the light is weakened with each other, and when the optical path length is an even multiple of the half wavelength, the light is intensified with each other. That is, the intensity of the transmitted light having a specific wavelength that has been emitted from the point light sources PA and PB and transmitted through the optical filter 902LF is modulated according to the incident angle with respect to the optical filter 902LF, as illustrated in B of Fig. 48, and is incident on the light receiving surface 901A of the imaging element 901. Accordingly, the detection signal output from each pixel output unit of the imaging element 901 is given as a signal obtained by merging the modulated light intensities of respective point light sources for each pixel output unit. [0421]
Details of this scheme are disclosed, for example, in Patent Document 1 described above. [0422]

Note that, in the schemes of Patent Document 1 and Non-Patent Document 1, the incident angle directivity cannot be set independently in units of pixels 121a without affecting adjacent pixels, like the imaging element 121 using the pixel 121a in Fig. 4 or the pixel 121a in Fig. 5 described above. Accordingly, for example, when the pattern of the optical filter 902BW or the pattern of the diffraction grating of the optical filter 902LF is made different, the incident angle directivities of at least a plurality of adjacent pixels of the imaging element 901 are in turn made different together with each other. Furthermore, the pixels 121a located at close positions have incident angle directivities close to each other. [0423]
In addition, the present disclosure can also be applied to an imaging apparatus or an imaging element that images light of a wavelength other than visible light, such as infrared light. In this case, the restoration image is not given as an image in which the user can recognize the subject by viewing but an image in which the user cannot visually recognize the subject. Note that, since it is difficult for a normal imaging lens to transmit far-infrared light, the present technology is effective in a case where far-infrared light is imaged, for example. [0424]

In the above description, an example in which biometric authentication is performed on the basis of an image obtained by imaging the face or eye of the user is

indicated; however, the present disclosure can also be applied to a case where biometric authentication is performed on the basis of an image obtained by imaging other parts of the user, such as fingerprint authentication. [0425]
Furthermore, the present disclosure can also be applied to a case where the movement or state or the like of the user’s eye other than the user’s line of sight is detected. For example, the present disclosure can also be applied to a case where blink detection, sleep detection, or the like is performed on the basis of an image obtained by imaging the user’s eye or face. Moreover, for example, the present disclosure can also be applied to a case where wearing or the like of the electronic instrument 301 such as a wearable device is detected. [0426]
In addition, for example, by applying machine learning such as deep learning, it is also possible to perform image recognition or the like using the detection image and detection signal set before restoration without using the restoration image after restoration. [0427]
<<6. Application Examples>>
The technology according to the present disclosure can be applied to a variety of products. For example, the technology according to the present disclosure can be applied to an imaging unit of a medical instrument such as an endoscopic surgery system or a capsular endoscope. [0428]

Fig. 49 is a diagram illustrating an example of an outline of the configuration of an endoscopic surgery system 5000 to which the technology according to the present disclosure can be applied. Fig. 49 illustrates a scene in which an operating surgeon (surgeon) 5067 is performing a surgery on a patient 5071 on a patient bed 5069 using the endoscopic surgery system 5000. As illustrated in Fig. 49, the endoscopic surgery system 5000 is constituted by an endoscope 5001, other surgical tools 5017, a support arm apparatus 5027 supporting the endoscope 5001, and a cart 5037 in which various apparatuses for surgery through the endoscope are equipped. [0429]
In endoscopic surgery, instead of cutting the abdominal wall and opening the abdomen, a plurality of cylindrical puncture tools called trocars 5025a to 5025d is punctured into the abdominal wall. Then, a lens barrel 5003 of the endoscope 5001 and other surgical tools 5017 are inserted into the body cavity of the patient 5071 through the trocars 5025a to 5025d. In the illustrated example, a pneumoperitoneum tube 5019, an energy treatment instrument 5021, and a forceps 5023 are inserted into the body cavity of the patient 5071 as the other surgical tools 5017. Furthermore, the energy treatment instrument 5021 is a treatment instrument that performs incision and detachment of tissue, sealing of a blood vessel, and the like by high-frequency current or ultrasonic vibration. However, the illustrated surgical tools 5017 are merely an example and various surgical tools generally used for surgery through the endoscope,

such as a thumb forceps and a retractor, can be used as
the surgical tools 5017, for example.
[0430]
The image of the surgical site in the body cavity of the patient 5071 captured by the endoscope 5001 is displayed on a display apparatus 5041. The operating surgeon 5067 performs treatment such as cutting off the affected part, for example, using the energy treatment instrument 5021 and the forceps 5023 while viewing the image of the surgical site displayed on the display apparatus 5041 in real time. Note that, although illustration is omitted, the pneumoperitoneum tube 5019, the energy treatment instrument 5021, and the forceps 5023 are supported by the operating surgeon 5067 or an assistant or the like during surgery. [0431]
(Support Arm Apparatus)
The support arm apparatus 5027 includes an arm portion 5031 extending from a base portion 5029. In the illustrated example, the arm portion 5031 is constituted by joint portions 5033a, 5033b, and 5033c, and links 5035a and 5035b and is driven under the control of an arm control apparatus 5045. The endoscope 5001 is supported by the arm portion 5031 such that the position and orientation of the endoscope 5001 are controlled. With this configuration, fixing of the position of the endoscope 5001 can be implemented in a stable manner. [0432]
(Endoscope)
The endoscope 5001 is constituted by the lens barrel 5003 of which a region of a predetermined length

from the distal end is inserted into the body cavity of the patient 5071, and a camera head 5005 connected to the proximal end of the lens barrel 5003. In the illustrated example, the endoscope 5001 configured as a so-called rigid endoscope having the rigid lens barrel 5003 is illustrated; however, the endoscope 5001 may be configured as a so-called flexible endoscope having the flexible lens barrel 5003. [0433]
An opening portion into which an objective lens is fitted is provided at the distal end of the lens barrel 5003. A light source apparatus 5043 is connected to the endoscope 5001; light generated by this light source apparatus 5043 is guided to the distal end of the lens barrel 5003 by a light guide provided extending inside this lens barrel, and is radiated toward an object to be observed in the body cavity of the patient 5071 via the objective lens. Note that the endoscope 5001 may be a forward-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope. [0434]
An optical system and an imaging element are provided inside the camera head 5005 and reflected light (observation light) from the object to be observed is converged on this imaging element by this optical system. The observation light is photoelectrically converted by the imaging element and an electrical signal corresponding to the observation light, that is, an image signal corresponding to the observation image is generated. This image signal is transmitted as RAW data to a camera control unit (CCU) 5039. Note that the

camera head 5005 is equipped with a function of adjusting the magnification and focal length by appropriately driving the optical system thereof. [0435]
In addition, for example, in order to cope with stereoscopic viewing (three-dimensional (3D) display) or the like, a plurality of imaging elements may be provided in the camera head 5005. In this case, a plurality of relay optical systems is provided inside the lens barrel 5003 in order to guide the observation light to each of the plurality of imaging elements. [0436]
(Various Apparatuses Equipped in Cart)
The CCU 5039 is constituted by a central processing unit (CPU), graphics processing unit (GPU), or the like and comprehensively controls working of the endoscope 5001 and the display apparatus 5041. Specifically, the CCU 5039 carries out various image processes for displaying an image based on the image signal accepted from the camera head 5005, such as a developing process (demosaic process), on the accepted image signal, for example. The CCU 5039 provides the image signal on which these image processes have been carried out to the display apparatus 5041. Furthermore, the CCU 5039 transmits a control signal to the camera head 5005 and controls driving of the camera head 5005. This control signal can include information regarding imaging conditions such as magnification and focal length. [0437]
The display apparatus 5041 displays an image based on the image signal on which the image processes have

been carried out by the CCU 5039, under the control of the CCU 5039. In a case where the endoscope 5001 is compatible with high resolution capturing such as 4K capturing (the number of horizontal pixels 3840 × the number of vertical pixels 2160), or 8K capturing (the number of horizontal pixels 7680 × the number of vertical pixels 4320), and/or is compatible with 3D display, a display apparatus capable of high resolution display and/or capable of 3D display can be used as the display apparatus 5041 so as to cope with each case, for example. In a case where the endoscope 5001 is compatible with high resolution capturing such as 4K or 8K capturing, a more immersive feeling can be obtained by using a display apparatus 5041 having a size of 55 inches or more. Furthermore, a plurality of display apparatuses 5041 with different resolutions and sizes may be provided depending on the practical usage. [0438]
The light source apparatus 5043 is constituted by a light source such as a light emitting diode (LED) and the like and supplies irradiation light used when the surgical site is captured to the endoscope 5001. [0439]
The arm control apparatus 5045 is constituted, for example, by a processor such as a CPU and works in accordance with a predetermined program to control driving of the arm portion 5031 of the support arm apparatus 5027 in accordance with a predetermined control scheme. [0440]
The input apparatus 5047 is an input interface to

the endoscopic surgery system 5000. The user can input various types of information and input instructions to the endoscopic surgery system 5000 via the input apparatus 5047. For example, the user inputs various types of information regarding surgery, such as body information of the patient and information about the surgical procedure of the surgery, via the input apparatus 5047. Furthermore, for example, the user inputs an instruction to drive the arm portion 5031, an instruction to alter the imaging conditions (the type of irradiation light, the magnification, the focal length, and the like) for the endoscope 5001, an instruction to drive the energy treatment instrument 5021, and the like via the input apparatus 5047. [0441]
The type of the input apparatus 5047 is not limited and the input apparatus 5047 can be various known input apparatuses. For example, a mouse, a keyboard, a touch panel, a switch, a foot switch 5057, a lever, and/or the like, can be applied as the input apparatus 5047. In a case where a touch panel is used as the input apparatus 5047, the touch panel may be provided on the display surface of the display apparatus 5041. [0442]
Alternatively, the input apparatus 5047 is, for example, a device worn by the user, such as a glasses-type wearable device or a head mounted display (HMD), and various inputs are made according to the gesture and the line of sight of the user detected by these devices. Furthermore, the input apparatus 5047 includes a camera capable of detecting the movement of the user and various

inputs are made according to the gesture and the line of sight of the user detected from a moving picture captured by the camera. Moreover, the input apparatus 5047 includes a microphone capable of picking up the voice of the user and various inputs are made by sound via the microphone. In this manner, by configuring the input apparatus 5047 so as to be able to input various types of information in a non-contact manner, particularly a user (for example, the operating surgeon 5067) belonging to a clean area is allowed to operate an instrument belonging to a non-clean area in a non-contact manner. In addition, since the user can operate the instrument without releasing his/her hand from the holding surgical tool, the user’s convenience is improved. [0443]
A treatment instrument control apparatus 5049 controls driving of the energy treatment instrument 5021 for cauterization and incision of tissue, sealing of a blood vessel, or the like. For the purpose of securing a field of view through the endoscope 5001 and securing a working space for the operating surgeon, a pneumoperitoneum apparatus 5051 delivers a gas into the body cavity of the patient 5071 via the pneumoperitoneum tube 5019 in order to inflate the body cavity. A recorder 5053 is an apparatus capable of recording various types of information regarding surgery. A printer 5055 is an apparatus capable of printing various types of information regarding surgery in various formats such as text, image, or graph. [0444]
Hereinafter, a particularly characteristic

configuration of the endoscopic surgery system 5000 will
be described in more detail.
[0445]
(Support Arm Apparatus)
The support arm apparatus 5027 includes the base portion 5029 as a pedestal and the arm portion 5031 extending from the base portion 5029. In the illustrated example, the arm portion 5031 is constituted by the plurality of joint portions 5033a, 5033b, and 5033c, and the plurality of links 5035a and 5035b coupled by the joint portion 5033b; in Fig. 49, however, for the sake of simplicity, the configuration of the arm portion 5031 is illustrated in a simplified manner. Actually, the shapes, the number, and the arrangement of the joint portions 5033a to 5033c and the links 5035a and 5035b, as well as the directions of the rotation axes of the joint portions 5033a to 5033c, and the like can be appropriately set such that the arm portion 5031 has a desired degree of freedom. For example, the arm portion 5031 can be suitably configured so as to have degrees of freedom equal to or greater than six degrees of freedom. This allows the endoscope 5001 to freely move within the movable range of the arm portion 5031, such that the lens barrel 5003 of the endoscope 5001 can be inserted into the body cavity of the patient 5071 in a desired direction. [0446]
Actuators are provided in the joint portions 5033a to 5033c and the joint portions 5033a to 5033c are configured so as to be rotatable around predetermined rotation axes by driving of these actuators. Driving of

the actuators is controlled by the arm control apparatus 5045, such that the rotation angle of each of the joint portions 5033a to 5033c is controlled and then driving of the arm portion 5031 is controlled. With this configuration, the control of the position and orientation of the endoscope 5001 can be implemented. At this time, the arm control apparatus 5045 can control driving of the arm portion 5031 by various known control schemes such as force control or position control. [0447]
For example, the position and orientation of the endoscope 5001 can be controlled in such a manner that the operating surgeon 5067 appropriately makes an operation input via the input apparatus 5047 (including the foot switch 5057) and in turn the arm control apparatus 5045 appropriately controls driving of the arm portion 5031 according to the operation input. With this control, the endoscope 5001 at the distal end of the arm portion 5031 can be moved from an arbitrary position to another arbitrary position and thereafter can be fixedly supported at that position after the movement. Note that the arm portion 5031 may be operated by a so-called master slave scheme. In this case, the arm portion 5031 can be remotely operated by the user via the input apparatus 5047 installed at a place away from the surgery room. [0448]
Furthermore, in a case where the force control is applied, the arm control apparatus 5045 may perform so-called power assist control in which the actuators of the respective joint portions 5033a to 5033c are driven in

response to an external force from the user such that the arm portion 5031 moves smoothly in proportion to the external force. With this control, when the user moves the arm portion 5031 while directly touching the arm portion 5031, the arm portion 5031 can be moved with a relatively light force. Accordingly, it becomes possible to more intuitively move the endoscope 5001 with a simpler operation and user’s convenience can be improved. [0449]
Here, in general surgery through the endoscope, the endoscope 5001 has been supported by a surgeon called a scopist. In contrast to this, using the support arm apparatus 5027 makes it possible to reliably fix the position of the endoscope 5001 without manual operation, such that the image of the surgical site can be stably obtained and surgery can be performed smoothly. [0450]
Note that the arm control apparatus 5045 is not necessarily provided in the cart 5037. Furthermore, the arm control apparatus 5045 is not necessarily one apparatus. For example, the arm control apparatus 5045 may be provided in each of the joint portions 5033a to 5033c of the arm portion 5031 of the support arm apparatus 5027 such that control of driving of the arm portion 5031 is implemented by the plurality of arm control apparatuses 5045 collaborating with each other. [0451]
(Light Source Apparatus)
The light source apparatus 5043 supplies the endoscope 5001 with irradiation light when the surgical site is captured. The light source apparatus 5043 is

constituted by, for example, an LED, a laser light source, or a white light source constituted by a combination thereof. At this time, in a case where the white light source is configured by a combination of RGB laser light sources, the output intensity and the output timing of each color (each wavelength) can be controlled with high accuracy and accordingly the white balance of the captured image can be adjusted in the light source apparatus 5043. Furthermore, in this case, by irradiating an object to be observed with a laser beam from each of the RGB laser light sources by time sharing and controlling driving of the imaging element of the camera head 5005 in synchronization with the irradiation timing, it is also possible to capture an image corresponding to each of RGB by time sharing. According to this method, a color image can be obtained without providing a color filter in the imaging element. [0452]
In addition, driving of the light source apparatus 5043 may be controlled such that the intensity of light to be output is altered for every predetermined time span. By controlling driving of the imaging element of the camera head 5005 in synchronization with the timing of the alteration of the intensity of light to acquire images by time sharing and merging these images, an image of a high dynamic range without so-called blocked up shadows and blown out highlights can be generated. [0453]
Furthermore, the light source apparatus 5043 may be configured such that light of a predetermined wavelength band compatible with special light observation can be

supplied from the light source apparatus 5043. In the special light observation, for example, by utilizing the wavelength dependence of light absorption in body tissue, so-called narrow band light observation (narrow band imaging) is performed in which, by radiating narrow band light compared with the irradiation light at the time of ordinary observation (in other words, white light), predetermined tissue such as a blood vessel at a mucosal surface layer is captured with high contrast. Alternatively, in the special light observation, fluorescence observation for obtaining an image by fluorescence generated by radiating excitation light may be performed. In the fluorescence observation, for example, fluorescence observation in which body tissue is irradiated with excitation light and fluorescence from the body tissue is observed (autofluorescence observation), or fluorescence observation in which a reagent such as indocyanine green (ICG) is locally administered to body tissue and at the same time, the body tissue is irradiated with excitation light corresponding to a fluorescence wavelength of the reagent to obtain a fluorescent image can be performed. The light source apparatus 5043 can be configured such that narrow band light and/or excitation light compatible with such special light observation can be supplied from the light source apparatus 5043. [0454]
(Camera Head and CCU)
Referring to Fig. 50, functions of the camera head 5005 and the CCU 5039 of the endoscope 5001 will be described in more detail. Fig. 50 is a block diagram

illustrating an example of functional configurations of the camera head 5005 and the CCU 5039 illustrated in Fig. 49. [0455]
Referring to Fig. 50, the camera head 5005 has a lens unit 5007, an imaging unit 5009, a driving unit 5011, a communication unit 5013, and a camera head control unit 5015 as functions thereof. Furthermore, the CCU 5039 has a communication unit 5059, an image processing unit 5061, and a control unit 5063 as functions thereof. The camera head 5005 and the CCU 5039 are connected via a transmission cable 5065 so as to be able to communicate bidirectionally. [0456]
First, the functional configuration of the camera head 5005 will be described. The lens unit 5007 is an optical system provided at a connecting portion with the lens barrel 5003. The observation light taken in from the distal end of the lens barrel 5003 is guided to the camera head 5005 and is incident on the lens unit 5007. The lens unit 5007 is constituted by combining a plurality of lenses including a zoom lens and a focus lens. The optical characteristics of the lens unit 5007 are adjusted so as to converge the observation light on a light receiving surface of the imaging element of the imaging unit 5009. Furthermore, the zoom lens and the focus lens are configured such that their positions on an optical axis can move in order to adjust the magnification and focus of the captured image. [0457]
The imaging unit 5009 is constituted by the imaging

element and is arranged at a succeeding stage of the lens unit 5007. The observation light having passed through the lens unit 5007 is converged on the light receiving surface of the imaging element and an image signal corresponding to the observation image is generated by photoelectric conversion. The image signal generated by the imaging unit 5009 is provided to the communication unit 5013. [0458]
For example, an image sensor of the complementary metal oxide semiconductor (CMOS) type capable of capturing a color image having a Bayer array is used as the imaging element constituting the imaging unit 5009. Note that, for example, an imaging element capable of coping with capturing of a high resolution image of 4K or more may be used as the imaging element. Since the image of the surgical site is obtained with high resolution, the operating surgeon 5067 can grasp how the surgical site is in more detail and can progress the surgery more smoothly. [0459]
Furthermore, the imaging element constituting the imaging unit 5009 is configured such that the imaging element constituting the imaging unit 5009 has a pair of imaging elements for separately acquiring image signals for right eye and left eye compatible with the 3D display. Owing to that the 3D display is performed, the operating surgeon 5067 can more precisely grasp the depth of the living tissue in the surgical site. Note that, in a case where the imaging unit 5009 is configured as a multi-plate type, the lens units 5007 are also provided

as a plurality of systems corresponding to respective
imaging elements.
[0460]
Furthermore, the imaging unit 5009 is not necessarily provided in the camera head 5005. For example, the imaging unit 5009 may be provided inside the lens barrel 5003 immediately behind the objective lens. [0461]
The driving unit 5011 is constituted by an actuator and moves the zoom lens and the focus lens of the lens unit 5007 by a predetermined distance along the optical axis under the control of the camera head control unit 5015. With this movement, the magnification and the focus of the captured image by the imaging unit 5009 can be appropriately adjusted. [0462]
The communication unit 5013 is constituted by a communication apparatus for transmitting and receiving various types of information to and from the CCU 5039. The communication unit 5013 transmits the image signal obtained from the imaging unit 5009 as RAW data to the CCU 5039 via the transmission cable 5065. At this time, in order to display the captured image of the surgical site with low latency, it is preferred that the image signal be transmitted by optical communication. This is because the operating surgeon 5067 performs surgery while observing the state of the affected part with the captured image and thus, for more safe and reliable surgery, the moving image of the surgical site is required to be displayed in real time as much as possible during surgery. In a case where optical communication is

performed, a photoelectric conversion module that converts the electrical signal into an optical signal is provided in the communication unit 5013. The image signal is converted into an optical signal by the photoelectric conversion module and then transmitted to the CCU 5039 via the transmission cable 5065. [0463]
Furthermore, the communication unit 5013 receives the control signal for controlling driving of the camera head 5005 from the CCU 5039. This control signal includes, for example, information regarding imaging conditions, such as information to designate a frame rate of the captured image, information to designate an exposure value at the time of imaging, and/or information to designate the magnification and focus of the captured image. The communication unit 5013 provides the received control signal to the camera head control unit 5015. Note that the control signal from the CCU 5039 may also be transmitted by optical communication. In this case, the communication unit 5013 is provided with a photoelectric conversion module that converts the optical signal into an electrical signal; the control signal is converted into an electrical signal by this photoelectric conversion module and then provided to the camera head control unit 5015. [0464]
Note that the above-mentioned imaging conditions such as the frame rate, exposure value, magnification, and focus are automatically set by the control unit 5063 of the CCU 5039 on the basis of the acquired image signal. In other words, so-called auto exposure (AE)

function, auto focus (AF) function, and auto white balance (AWB) function are equipped in the endoscope 5001. [0465]
The camera head control unit 5015 controls driving of the camera head 5005 on the basis of the control signal received from the CCU 5039 via the communication unit 5013. For example, the camera head control unit 5015 controls driving of the imaging element of the imaging unit 5009 on the basis of information to designate the frame rate of the captured image and/or information to designate exposure at the time of imaging. Furthermore, for example, the camera head control unit 5015 appropriately moves the zoom lens and the focus lens of the lens unit 5007 via the driving unit 5011 on the basis of information to designate the magnification and focus of the captured image. The camera head control unit 5015 may further include a function of storing information for identifying the lens barrel 5003 and the camera head 5005. [0466]
Note that, by arranging constituents such as the lens unit 5007 and the imaging unit 5009 in a hermetically sealed structure having high airtightness and waterproofness, the camera head 5005 is enabled to have resistance to an autoclave sterilization process. [0467]
Next, the functional configuration of the CCU 5039 will be described. The communication unit 5059 is constituted by a communication apparatus for transmitting and receiving various types of information to and from

the camera head 5005. The communication unit 5059 receives the image signal transmitted from the camera head 5005 via the transmission cable 5065. At this time, as described above, the image signal can be suitably transmitted by optical communication. In this case, a photoelectric conversion module that converts the optical signal into an electrical signal is provided in the communication unit 5059 so as to be compatible with the optical communication. The communication unit 5059 provides the image signal converted into the electrical signal to the image processing unit 5061. [0468]
Furthermore, the communication unit 5059 transmits the control signal for controlling driving of the camera head 5005 to the camera head 5005. This control signal can also be transmitted by optical communication. [0469]
The image processing unit 5061 carries out various image processes on the image signal, which is RAW data transmitted from the camera head 5005. Examples of these image processes include various known signal processes such as a developing process, a high image quality process (a band enhancement process, a super resolution process, a noise reduction (NR) process, a camera shake correction process, and/or the like), and/or an enlargement process (electronic zoom process). Furthermore, the image processing unit 5061 performs a wave sensing process on the image signal for performing AE, AF, and AWB. [0470]
The image processing unit 5061 is constituted by a

processor such as a CPU and a GPU and the above-described image processes and wave sensing process can be performed by this processor working in accordance with a predetermined program. Note that, in a case where the image processing unit 5061 is constituted by a plurality of GPUs, the image processing unit 5061 appropriately divides the information related to the image signal and performs the image processes in parallel by the plurality of GPUs. [0471]
The control unit 5063 performs various types of control relating to imaging of the surgical site by the endoscope 5001 and display of the captured image obtained by the imaging. For example, the control unit 5063 generates a control signal for controlling driving of the camera head 5005. At this time, in a case where the imaging conditions have been input by the user, the control unit 5063 generates the control signal on the basis of the input by the user. Alternatively, in a case where the AE function, the AF function, and the AWB function are equipped in the endoscope 5001, the control unit 5063 appropriately calculates the optimum exposure value, focal length, and white balance according to the result of the wave sensing process by the image processing unit 5061 and generates the control signal. [0472]
Furthermore, the control unit 5063 displays the image of the surgical site on the display apparatus 5041 on the basis of the image signal on which the image processes have been carried out by the image processing unit 5061. At this time, the control unit 5063

recognizes various objects in the image of the surgical site using various image recognition technologies. For example, the control unit 5063 detects the shape, color, and the like of the edge of an object included in the image of the surgical site, thereby being able to recognize a surgical tool such as the forceps, a specific living body part, bleeding, a mist at the time of using the energy treatment instrument 5021, and the like. When displaying the image of the surgical site on the display apparatus 5041, the control unit 5063 displays various types of surgery support information superimposed onto this image of the surgical site using results of the recognition. Since the surgery support information is displayed superimposed and presented to the operating surgeon 5067, surgery can be advanced more safely and reliably. [0473]
The transmission cable 5065 connecting the camera head 5005 and the CCU 5039 is an electrical signal cable compatible with communication of electrical signals, an optical fiber compatible with optical communication, or a composite cable thereof. [0474]
Here, in the illustrated example, communication is performed by wire using the transmission cable 5065; however, communication between the camera head 5005 and the CCU 5039 may be performed wirelessly. In a case where the communication between the camera head 5005 and the CCU 5039 is performed wirelessly, it is no longer necessary to lay the transmission cable 5065 in the surgery room, such that a difficulty that the movement of

the medical staff in the surgery room is hindered by the
transmission cable 5065 can be eliminated.
[0475]
An example of the endoscopic surgery system 5000 to which the technology according to the present disclosure can be applied has been described thus far. Note that, although the endoscopic surgery system 5000 has been described here as an example, a system to which the technology according to the present disclosure can be applied is not limited to such an example. For example, the technology according to the present disclosure may be applied to an examination flexible endoscopic system or a microscopic surgery system. [0476]
The technology according to the present disclosure can be suitably applied to the imaging unit 5009 in the configuration described above. By applying the technology according to the present disclosure to the imaging unit 5009, the imaging unit 5009 can be downsized. Furthermore, since the degree of freedom in arrangement of each pixel of the imaging element is higher, it becomes easier to obtain the image of the surgical site at a desired position, and it is possible to perform surgery more safely and reliably. [0477]
Fig. 51 is a diagram illustrating an example of an outline of the configuration of an in-vivo information acquisition system 5400 to which the technology according to the present disclosure can be applied. Referring to Fig. 51, the in-vivo information acquisition system 5400 is constituted by a capsule endoscope 5401 and an

external control apparatus 5423 that comprehensively controls working of the in-vivo information acquisition system 5400. During the examination, the capsule endoscope 5401 is swallowed by a patient. The capsule endoscope 5401 has an imaging function and a wireless communication function, and moves inside organs such as the stomach and intestine by peristaltic motion and the like until the capsule endoscope 5401 is spontaneously discharged from the patient, while sequentially capturing images inside these organs (hereinafter also referred to as in-vivo images) at predetermined intervals, and sequentially transmitting information about the captured in-vivo images wirelessly to the external control apparatus 5423 outside the body. The external control apparatus 5423 generates image data for displaying the captured in-vivo images on a display apparatus (not illustrated) on the basis of the received information about the in-vivo images. By configuring in this manner, in the in-vivo information acquisition system 5400, an image obtained by imaging how the inside of the patient’s body is can be obtained at any time from when the capsule endoscope 5401 is swallowed until it is discharged. [0478]
The configurations and functions of the capsule endoscope 5401 and the external control apparatus 5423 will be described in more detail. As illustrated in Fig. 51, the capsule endoscope 5401 is configured in such a manner that functions of a light source unit 5405, an imaging unit 5407, an image processing unit 5409, a wireless communication unit 5411, a power feeding unit 5415, a power supply unit 5417, a state detection unit

5419, and a control unit 5421 are equipped in a capsule
type casing 5403.
[0479]
The light source unit 5405 is constituted by a light source such as a light emitting diode (LED), and irradiates the imaging field of view of the imaging unit 5407 with light. [0480]
The imaging unit 5407 is constituted by an imaging element and an optical system including a plurality of lenses provided at the preceding stage of the imaging element. Reflected light (hereinafter referred to as observation light) of light radiated on body tissue to be observed is converged by the optical system and is incident on the imaging element. The imaging element receives the observation light and photoelectrically converts the received observation light to generate an electrical signal corresponding to the observation light, that is, an image signal corresponding to the observation image. The image signal generated by the imaging unit 5407 is provided to the image processing unit 5409. Note that various known imaging elements such as a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor can be used as the imaging element of the imaging unit 5407. [0481]
The image processing unit 5409 is constituted by a processor such as a central processing unit (CPU) or a graphics processing unit (GPU), and performs various signal processes on the image signal generated by the imaging unit 5407. The various signal processes only

need to include minimal processes for transmitting the image signal to the external control apparatus 5423 (for example, image data compression, frame rate conversion, data rate conversion, format conversion, and/or the like). The image processing unit 5409 can be implemented with a smaller size and lower consumed power by configuring such that the image processing unit 5409 performs only the minimum necessary processes, and therefore is favorable for the capsule endoscope 5401. However, in a case where there is a margin in space in the casing 5403 and consumed power, additional signal processes (for example, noise removal process, other high image quality processes, or the like) may be performed in the image processing unit 5409. The image processing unit 5409 provides the image signal on which the signal processes have been carried out to the wireless communication unit 5411 as RAW data. Note that, in a case where information about the state (movement, orientation, and the like) of the capsule endoscope 5401 is acquired by the state detection unit 5419, the image processing unit 5409 may provide the image signal to the wireless communication unit 5411 in link with the acquired information. With this configuration, the position in the body where the image was captured, the imaging direction of the image, and the like can be associated with the captured image. [0482]
The wireless communication unit 5411 is constituted by a communication apparatus capable of transmitting and receiving various types of information to and from the external control apparatus 5423. This communication

apparatus is constituted, for example, by an antenna 5413, a processing circuit that performs a modulation process for signal transmission and reception, and the like. The wireless communication unit 5411 performs a predetermined process such as the modulation process on the image signal on which the signal processes have been carried out by the image processing unit 5409, and transmits the resulting image signal to the external control apparatus 5423 via the antenna 5413. Furthermore, the wireless communication unit 5411 receives a control signal relating to drive control of the capsule endoscope 5401 from the external control apparatus 5423 via the antenna 5413. The wireless communication unit 5411 provides the received control signal to the control unit 5421. [0483]
The power feeding unit 5415 is constituted by an antenna coil for receiving electric power, a power regeneration circuit that regenerates electric power from a current generated in the antenna coil, a booster circuit, and the like. In the power feeding unit 5415, electric power is generated using a so-called non-contact charging principle. Specifically, when a magnetic field (electromagnetic wave) having a predetermined frequency is imparted from the outside to the antenna coil of the power feeding unit 5415, an induced electromotive force is generated in the antenna coil. This electromagnetic wave may be a carrier wave transmitted from the external control apparatus 5423 via the antenna 5425, for example. Electric power is regenerated from the generated induced electromotive force by the power regeneration circuit,

and the potential of the regenerated electric power is appropriately adjusted in the booster circuit, whereby electric power for accumulation is generated. The electric power generated by the power feeding unit 5415 is accumulated in the power supply unit 5417. [0484]
The power supply unit 5417 is constituted by a secondary battery, and accumulates the electric power generated by the power feeding unit 5415. In Fig. 51, in order to avoid making the drawing complicated, illustration of an arrow or the like indicating the supply destination of the electric power from the power supply unit 5417 is omitted; however, the electric power accumulated in the power supply unit 5417 is supplied to the light source unit 5405, the imaging unit 5407, the image processing unit 5409, the wireless communication unit 5411, the state detection unit 5419, and the control unit 5421, and can be used for driving these units. [0485]
The state detection unit 5419 is constituted by a sensor for detecting the state of the capsule endoscope 5401, such as an acceleration sensor and/or a gyro sensor. The state detection unit 5419 can acquire information about the state of the capsule endoscope 5401 from the result of the detection by the sensor. The state detection unit 5419 provides the acquired information about the state of the capsule endoscope 5401 to the image processing unit 5409. In the image processing unit 5409, as described above, this information about the state of the capsule endoscope 5401 can be linked with the image signal.

[0486]
The control unit 5421 is constituted by a processor such as a CPU, and comprehensively controls working of the capsule endoscope 5401 by working in accordance with a predetermined program. The control unit 5421 appropriately controls driving of the light source unit 5405, the imaging unit 5407, the image processing unit 5409, the wireless communication unit 5411, the power feeding unit 5415, the power supply unit 5417, and the state detection unit 5419 in accordance with control signals transmitted from the external control apparatus 5423, thereby implementing the function in each unit as described above. [0487]
The external control apparatus 5423 can be a processor such as a CPU or a GPU, or a microcomputer or a control board or the like in which a processor and a storage element such as a memory are jointly mounted. The external control apparatus 5423 includes an antenna 5425, and is configured such that various types of information can be transmitted to and received from the capsule endoscope 5401 via the antenna 5425. Specifically, the external control apparatus 5423 controls working of the capsule endoscope 5401 by transmitting a control signal to the control unit 5421 of the capsule endoscope 5401. For example, light irradiation conditions for an object to be observed in the light source unit 5405 can be altered by the control signal from the external control apparatus 5423. Furthermore, imaging conditions (for example, the frame rate or exposure value in the imaging unit 5407) can be

altered by the control signal from the external control apparatus 5423. In addition, the content of the process in the image processing unit 5409 and conditions for the wireless communication unit 5411 to transmit the image signal (for example, the transmission interval, the number of transmission images, and the like) may be altered by the control signal from the external control apparatus 5423. [0488]
Besides, the external control apparatus 5423 carries out various image processes on the image signal transmitted from the capsule endoscope 5401, and generates image data for displaying the captured in-vivo image on the display apparatus. For example, various known signal processes can be performed as these image processes, such as a developing process (demosaic processing), a high image quality process (a band enhancement process, a super resolution process, a noise reduction (NR) process, a camera shake correction process, and/or the like), and/or an enlargement process (electronic zoom process). The external control apparatus 5423 controls driving of the display apparatus (not illustrated) to display an in-vivo image captured on the basis of the generated image data. Alternatively, the external control apparatus 5423 may cause a recording apparatus (not illustrated) to record the generated image data or may cause a printing apparatus (not illustrated) to output and print the generated image data. [0489]
An example of the in-vivo information acquisition system 5400 to which the technology according to the

present disclosure can be applied has been described thus far. The technology according to the present disclosure can be suitably applied to the imaging unit 5407 in the configuration described above. By applying the technology according to the present disclosure to the imaging unit 5407, the capsule endoscope 5401 can be further downsized, and thus the burden on the patient can be further mitigated. [0490]
<<7. Others>>
A series of the above-described processes can be executed by hardware as well and also can be executed by software. In a case where the series of the processes is executed by software, a program constituting the software is installed in a computer. Here, the computer includes, for example, a computer incorporated in dedicated hardware (for example, the control unit 123 and the like). [0491]
For example, the program executed by the computer can be provided by being recorded in a recording medium serving as a package medium or the like (for example, the recording medium 130 and the like). Furthermore, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. [0492]
Note that, the program executed by the computer may be a program in which the processes are performed along the time series in accordance with the order described in the present description, or alternatively, may be a

program in which the processes are performed in parallel or at a necessary timing, for example, when called. [0493]
In addition, in the present description, a system refers to a collection of a plurality of constituent elements (e.g., apparatuses and modules (parts)) and whether or not all the constituent elements are arranged within the same casing is not regarded as important. Accordingly, a plurality of apparatuses accommodated in separate casings so as to be connected to one another via a network and one apparatus of which a plurality of modules is accommodated within one casing are both deemed as systems. [0494]
Moreover, the embodiments according to the present technology are not limited to the above-described embodiments and various alterations can be made without departing from the scope of the present technology. [0495]
For example, the present technology can employ a cloud computing configuration in which one function is divided and allocated to a plurality of apparatuses so as to be processed in coordination thereamong via a network. [0496]
Furthermore, the respective steps described in the aforementioned flowcharts can be executed by a plurality of apparatuses each taking a share thereof as well as executed by a single apparatus. [0497]
Moreover, in a case where a plurality of processes is included in one step, the plurality of processes

included in one step can be executed by a plurality of apparatuses each taking a share thereof as well as executed by a single apparatus. [0498]
Note that the present disclosure can be also configured as described below. [0499]
(1)
An electronic instrument worn or used by a user, the electronic instrument including
an imaging unit arranged at a position where at least a part of the user wearing or using the electronic instrument is capturable, the imaging unit including two or more pixel output units that each receive incident light from a subject incident not via either an imaging lens or a pinhole and output one detection signal indicating an output pixel value modulated depending on an incident angle of the incident light.
(2)
The electronic instrument according to (1) above, in which
each of the pixel output units is arranged in an eyepiece unit of the electronic instrument or around the eyepiece unit.
(3)
The electronic instrument according to (2) above, in which
the electronic instrument is worn on a head of the user.
(4)
The electronic instrument according to (3) above,

in which
the electronic instrument is of a glasses type or a goggles type.
(5)
The electronic instrument according to (2) above, in which
the electronic instrument includes a camera, and
the eyepiece unit includes a viewfinder of the camera.
(6)
The electronic instrument according to any one of (2) to (5) above, in which
at least some of the pixel output units have higher light receiving sensitivity with respect to a direction of an eye of the user in proximity to the eyepiece unit.
(7)
The electronic instrument according to (1) above, in which
each of the pixel output units is arranged in a display unit of the electronic instrument or around the display unit.
(8)
The electronic instrument according to (7) above, in which
at least some of the pixel output units have higher light receiving sensitivity with respect to a direction of a face of the user viewing the display unit.
(9)
The electronic instrument according to any one of (1) to (8) above, further including
a restoration unit that restores a restoration

image using a plurality of the detection signals from respective ones of the pixel output units.
(10)
The electronic instrument according to (9) above, further including
a control unit that executes a predetermined process on the basis of the restoration image.
(11)
The electronic instrument according to (10) above, in which
the imaging unit is arranged at a position where at least a part of an eye of the user wearing or using the electronic instrument is capturable, and
the control unit detects a movement or a state of the eye of the user on the basis of the restoration image.
(12)
The electronic instrument according to (10) above, in which
the imaging unit is arranged at a position where at least a part of a face of the user wearing or using the electronic instrument is capturable, and
the control unit performs a recognition process or an authentication process for the user on the basis of the restoration image.
(13)
The electronic instrument according to any one of (1) to (12) above, in which
the imaging unit includes one or more imaging elements, and
a plurality of the pixel output units is provided

in the imaging elements.
(14)
The electronic instrument according to any one of (1) to (13) above, in which
respective ones of the pixel output units are arranged in two or more regions away from each other.
(15)
The electronic instrument according to any one of (1) to (14) above, in which
the plurality of pixel output units has a configuration in which the incident angle directivity can be set independently.
(16)
The electronic instrument according to (15) above, in which
the plurality of pixel output units each includes: one photodiode; and
a light shielding film that blocks a part of the incident light from being incident on the photodiode.
(17)
The electronic instrument according to (15) above, in which
at least two pixel output units out of the plurality of pixel output units include a plurality of photodiodes, and have the incident angle directivities different from each other by making photodiodes that contribute to output different from each other.
(18)
The electronic instrument according to any one of (1) to (17) above, in which
the imaging unit has a configuration for causing

the incident angle directivities of output pixel values with respect to the incident light of at least two pixel output units out of the plurality of pixel output units to have characteristics different from each other. [0500]
Note that the effects described in the present description merely serve as examples and not construed to be limited. There may be another effect.
REFERENCE SIGNS LIST
[0501]
101 Imaging apparatus
111 Signal processing control unit
121 Imaging element
121a, 121a' Pixel
121A Light receiving surface 121b Light shielding film 121c On-chip lens 121e, 121f Photodiode
122 Restoration unit
123 Control unit

125 Detection unit
126 Association unit
301 Electronic instrument
311 Imaging unit
312 Signal processing control unit

321 Restoration unit
322 Control unit

325 Association unit
326 Output unit
328 Recording/playback unit

401 Wearable device
411L, 411R Lens
412 Frame
431 Camera
441 Viewfinder
461 Head mounted display
471L, 471R Lens
491 PC
501 Display
502 Bezel
601 Wearable device 611L, 611R Lens 612 Frame 631 Camera
641 Mount
642 Flash built-in unit
643 Lens
661 Head mounted display
671 Main body portion
672 Head pad 691 Headphones 701L, 701R Housing 702 Headset
721 Headphones
731L, 731R Housing
732 Neckband
751 Bending sensor
801 Information processing system
811 Electronic instrument
812 Signal processing apparatus 901b, 901s Pixel output unit

CLAIMS
1. An electronic instrument worn or used by a user,
the electronic instrument comprising
an imaging unit arranged at a position where at least a part of the user wearing or using the electronic instrument is capturable, the imaging unit including two or more pixel output units that each receive incident light from a subject incident not via either an imaging lens or a pinhole and output one detection signal indicating an output pixel value modulated depending on an incident angle of the incident light.
2. The electronic instrument according to claim 1,
wherein
each of the pixel output units is arranged in an eyepiece unit of the electronic instrument or around the eyepiece unit.
3. The electronic instrument according to claim 2,
wherein
the electronic instrument is worn on a head of the user.
4. The electronic instrument according to claim 3,
wherein
the electronic instrument is of a glasses type or a goggles type.
5. The electronic instrument according to claim 2,
wherein

the electronic instrument includes a camera, and the eyepiece unit includes a viewfinder of the camera.
6. The electronic instrument according to claim 2,
wherein
at least some of the pixel output units have higher light receiving sensitivity with respect to a direction of an eye of the user in proximity to the eyepiece unit.
7. The electronic instrument according to claim 1,
wherein
each of the pixel output units is arranged in a display unit of the electronic instrument or around the display unit.
8. The electronic instrument according to claim 7,
wherein
at least some of the pixel output units have higher light receiving sensitivity with respect to a direction of a face of the user viewing the display unit.
9. The electronic instrument according to claim 1,
further comprising
a restoration unit that restores a restoration image using a plurality of the detection signals from respective ones of the pixel output units.
10. The electronic instrument according to claim 9,
further comprising
a control unit that executes a predetermined

process on a basis of the restoration image.
11. The electronic instrument according to claim 10,
wherein
the imaging unit is arranged at a position where at least a part of an eye of the user wearing or using the electronic instrument is capturable, and
the control unit detects a movement or a state of the eye of the user on a basis of the restoration image.
12. The electronic instrument according to claim 10,
wherein
the imaging unit is arranged at a position where at least a part of a face of the user wearing or using the electronic instrument is capturable, and
the control unit performs a recognition process or an authentication process for the user on a basis of the restoration image.
13. The electronic instrument according to claim 1,
wherein
the imaging unit includes one or more imaging elements, and
a plurality of the pixel output units is provided in the imaging elements.
14. The electronic instrument according to claim 1,
wherein
respective ones of the pixel output units are arranged in two or more regions away from each other.

15. The electronic instrument according to claim 1,
wherein
the plurality of pixel output units has a configuration in which the incident angle directivity can be set independently.
16. The electronic instrument according to claim 15,
wherein
the plurality of pixel output units each includes: one photodiode; and
a light shielding film that blocks a part of the incident light from being incident on the photodiode.
17. The electronic instrument according to claim 15,
wherein
at least two pixel output units out of the plurality of pixel output units include a plurality of photodiodes, and have the incident angle directivities different from each other by making photodiodes that contribute to output different from each other.
18. The electronic instrument according to claim 1,
wherein
the imaging unit has a configuration for causing the incident angle directivities of output pixel values with respect to the incident light of at least two pixel output units out of the plurality of pixel output units to have characteristics different from each other.