FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION (See section 10, rule 13)
“SIGNAL PROCESSING DEVICE AND IMAGING DEVICE”
SONY CORPORATION, of 1-7-1, Konan, Minato-ku, Tokyo 108-0075, Japan

The following specification particularly describes the invention and the manner in which it is to
be performed.
31 is an angle A1 as illustrated in a left part of Fig. 15. Accordingly, the pixel 121a may receive the incident light by a width W1 in the horizontal direction of the subject surface 31. [0174]
In contrast, in a case where the pixel 121a' in Fig. 14 is arranged in the central position C1 of the imaging element 121, the pixel 121a' has a wider light-shielding range than the pixel 121a, so that the range of the incident angle of the incident light on the pixel 121a' from the subject surface 31 is an angle A2 (< A1) as illustrated in the left part of Fig. 15. Accordingly, the pixel 121a' may receive the incident light by a width W2 (< W1) in the horizontal direction of the subject surface 31. [0175]
That is, the pixel 121a having a narrow light-shielding range is a wide-angle pixel suitable for imaging a wide range on the subject surface 31, whereas the pixel 121a' having a wide light-shielding range is a narrow-angle pixel suitable for imaging a narrow range on the subject surface 31. Note that, the wide-angle pixel and the narrow-angle pixel here are expressions that compare both the pixels 121a and 121a' in Fig. 14, and are not limited when comparing pixels of other angles of view. [0176]
Therefore, for example, the pixel 121a is used to restore an image I1 in Fig. 14. The image I1 is the image having an angle of view SQ1 corresponding to the subject width W1 including an entire person H101 as the

subject in an upper stage of Fig. 16. In contrast, for example, the pixel 121a' is used to restore an image I2 in Fig. 14. The image I2 is the image having an angle of view SQ2 corresponding to the subject width W2 in which a periphery of a face of the person H101 in the upper stage of Fig. 16 is zoomed up. [0177]
Furthermore, for example, as illustrated in a lower stage of Fig. 16, it is considered to arrange a predetermined number of pixels 121a in Fig. 14 in a range ZA enclosed by a dotted line of the imaging elements 121 and arrange a predetermined number of pixels 121a' in a range ZB enclosed by a dashed-dotted line. Then, for example, when restoring the image of the angle of view SQ1 corresponding to the subject width W1, the image of the angle of view SQ1 may be appropriately restored by using the detection signal of each pixel 121a in the range ZA. On the other hand, when restoring the image of the angle of view SQ2 corresponding to the subject width W2, the image of the angle of view SQ2 may be appropriately restored by using the detection signal of each pixel 121a' in the range ZB. [0178]
Note that, since the angle of view SQ2 is narrower than the angle of view SQ1, in a case of restoring the images of the angle of view SQ2 and the angle of view SQ1 with the same number of pixels, it is possible to obtain the restored image with a higher image quality when restoring the image of the angle of view SQ2 than when restoring the image of the angle of view SQ1. [0179]

That is, in a case where it is considered to obtain the restored image using the same number of pixels, it is possible to obtain the restored image with a higher image quality when restoring the image with the narrower angle of view. [0180]
For example, a right part of Fig. 17 illustrates a configuration example in the range ZA of the imaging element 121 in Fig. 16. A left part of Fig. 17 illustrates a configuration example of the pixel 121a in the range ZA. [0181]
In Fig. 17, a range in black represents the light-shielding film 121b, and the light-shielding range of each pixel 121a is determined, for example, according to rules illustrated in the left part of Fig. 17. [0182]
A main light-shielding portion Z101 in the left part of Fig. 17 (black part in the left part of Fig. 17) is a range that is shielded in common in each pixel 121a. Specifically, the main light-shielding portion Z101 has a range of a width dx1 from left and right sides of the pixel 121a into the pixel 121a, and a range of a height dy1 from upper and lower sides of the pixel 121a into the pixel 121a, respectively. Then, in each pixel 121a, a rectangular opening Z111 that is not shielded by the light-shielding film 121b is provided in a range Z102 inside the main light-shielding portion Z101. Accordingly, in each pixel 121a, a range other than the opening Z111 is shielded by the light-shielding film 121b.

[0183]
Here, the openings Z111 of the respective pixels 121a are regularly arranged. Specifically, a position in the horizontal direction of the opening Z111 in each pixel 121a is the same in the pixels 121a in the same column in the vertical direction. Furthermore, a position in the vertical direction of the opening Z111 in each pixel 121a is the same in the pixels 121a in the same row in the horizontal direction. [0184]
On the other hand, the position in the horizontal direction of the opening Z111 in each pixel 121a is shifted at a predetermined interval according to the position in the horizontal direction of the pixel 121a. That is, as the position of the pixel 121a advances rightward, a left side of the opening Z111 moves to a position shifted rightward by widths dx1, dx2, ..., and dxn from the left side of the pixel 121a. An interval between the widths dx1 and dx2, an interval between the widths dx2 and dx3, ..., and an interval between the widths dxn-1 and dxn is a value obtained by dividing a length obtained by subtracting the width of the opening Z111 from the width in the horizontal direction of the range Z102 by the number of pixels n-1 in the horizontal direction. [0185]
Furthermore, the position in the vertical direction of the opening Z111 in each pixel 121a is shifted at a predetermined interval according to the position in the vertical direction of the pixel 121a. That is, as the position of the pixel 121a advances downward, an upper

side of the opening Z111 moves to a position shifted downward by widths dy1, dy2, ..., and dyn from the upper side of the pixel 121a. An interval between the heights dy1 and dy2, an interval between the heights dy2 and dy3, ..., and an interval between the heights dyn-1 and dyn is a value obtained by dividing a length obtained by subtracting the height of the opening Z111 from the height in the vertical direction of the range Z102 by the number of pixels m-1 in the vertical direction. [0186]
A right part of Fig. 18 illustrates a configuration example within the range ZB of the imaging element 121 in Fig. 16. A left part of Fig. 18 illustrates a configuration example of the pixel 121a' in the range ZB. [0187]
In Fig. 18, a range in black represents the light-shielding film 121b', and the light-shielding range of each pixel 121a' is determined, for example, according to rules illustrated in the left part of Fig. 18. [0188]
A main light-shielding portion Z151 in the left part of Fig. 18 (black part in the left part of Fig. 18) is a range that is shielded in common in each pixel 121a'. Specifically, the main light-shielding portion Z151 has a range of a width dx1' from left and right sides of the pixel 121a' into the pixel 121a', and a range of a height dy1' from upper and lower sides of the pixel 121a' into the pixel 121a', respectively. Then, in each pixel 121a', a rectangular opening Z161 that is not shielded by the light-shielding film 121b' is provided in a range Z152 inside the main light-shielding portion

Z151. Accordingly, in each pixel 121a', a range other than the opening Z161 is shielded by the light-shielding film 121b'. [0189]
Here, the openings Z161 of the respective pixels 121a' are regularly arranged in a manner similar to that of the openings Z111 of the respective pixels 121a in Fig. 17. Specifically, a position in the horizontal direction of the opening Z161 in each pixel 121a' is the same in the pixels 121a' in the same column in the vertical direction. Furthermore, a position in the vertical direction of the opening Z161 in each pixel 121a' is the same in the pixels 121a' in the same row in the horizontal direction. [0190]
On the other hand, the position in the horizontal direction of the opening Z161 in each pixel 121a' is shifted at a predetermined interval according to the position in the horizontal direction of the pixel 121a'. That is, as the position of the pixel 121a' advances rightward, a left side of the opening Z161 moves to a position shifted rightward by widths dx1', dx2', ..., and dxn' from the left side of the pixel 121a'. An interval between the widths dx1' and dx2', an interval between the widths dx2' and dx3', ..., and an interval between the widths dxn-1' and dxn' is a value obtained by dividing a length obtained by subtracting the width of the opening Z161 from the width in the horizontal direction of the range Z152 by the number of pixels n-1 in the horizontal direction. [0191]

Furthermore, the position in the vertical direction of the opening Z161 in each pixel 121a' is shifted at a predetermined interval according to the position in the vertical direction of the pixel 121a'. That is, as a position of the pixel 121a' advances downward, an upper side of the opening Z161 moves to a position shifted downward by widths dy1', dy2', ..., and dyn' from the upper side of the pixel 121a'. An interval between the heights dy1' and dy2', an interval between the heights dy2' and dy3', ..., and an interval between the heights dyn-1' and dyn' is a value obtained by dividing a length obtained by subtracting the height of the opening Z161 from the height in the vertical direction of the range Z152 by the number of pixels m-1 in the vertical direction. [0192]
Here, the length obtained by subtracting the width of the opening Z111 from the width in the horizontal direction of the range Z102 of the pixel 121a in Fig. 17 is larger than the width obtained by subtracting the width of the opening Z161 from the width in the horizontal direction of the range Z152 of the pixel 121a' in Fig. 18. Accordingly, a change interval between the widths dx1, dx2, ..., and dxn in Fig. 17 is larger than the change interval between the widths dx1', dx2', ..., and dxn' in Fig. 18. [0193]
Furthermore, the length obtained by subtracting the height of the opening Z111 from the height in the vertical direction of the range Z102 of the pixel 121a in Fig. 17 is larger than the length obtained by subtracting

the height of the opening Z161 from the height in the vertical direction of the range Z152 of the pixel 121a' in Fig. 18. Accordingly, a change interval between the heights dy1, dy2, ..., and dyn in Fig. 17 is larger than the change interval between the heights dy1', dy2', and dyn' in Fig. 18. [0194]
As described above, the change interval of the positions in the horizontal and vertical directions of the opening Z111 of the light-shielding film 121b of each pixel 121a in Fig. 17 is different from the change interval of the positions in the horizontal and vertical directions of the opening Z161 of the light-shielding film 121b' of each pixel 121a' in Fig. 18. Then, this difference in interval is the difference in subject resolution (angular resolution) in the restored image. That is, the change interval of the positions in the horizontal and vertical directions of the opening Z161 of the light-shielding film 121b' of each pixel 121a' in Fig. 18 is narrower than the change interval of the positions in the horizontal and vertical directions of the opening Z111 of the light-shielding film 121b of each pixel 121a in Fig. 17. Accordingly, the restored image restored by using the detection signal of each pixel 121a' in Fig. 18 has higher subject resolution and higher image quality than the restored image restored by using the detection signal of each pixel 121a in Fig. 17. [0195]
In this manner, by changing the combination of the light-shielding range of the main light-shielding portion and the opening range of the opening, the imaging element

121 including pixels having various angles of view (having various incident angle directivities) may be realized. [0196]
Note that, although the example in which the pixels 121a and the pixels 121a' are arranged separately in the range ZA and the range ZB, respectively, is described above, this is for the sake of simplicity, and the pixels 121a corresponding to different angles of view are desirably mixedly arranged in the same region. [0197]
For example, as illustrated in Fig. 19, four pixels each including two pixels × two pixels indicated by a dotted line are made one unit U, and each unit U includes a wide-angle pixel 121a-W, a medium-angle pixel 121a-M, a narrow-angle pixel 121a-N, and an extremely narrow-angle pixel 121a-AN. [0198]
In this case, for example, in a case where the number of all the pixels 121a is X, it becomes possible to restore the restored image using the detection images of X/4 pixels for each of the four types of view angles. At that time, four types of coefficient set groups different for each angle of view are used, and the restored images having different angles of view are restored by four different simultaneous equations. [0199]
Therefore, by restoring the restored image using the detection image obtained from the pixel suitable for imaging the angle of view of the restored image to be restored, it becomes possible to obtain an appropriate

restored image corresponding to the four types of angles
of view.
[0200]
Furthermore, it is also possible to interpolate to generate images of the angle of view between the four types of angles of view and the angle of view around the same from the images of the four types of angles of view, and realize pseudo optical zooming by seamlessly generating the images of the various angles of view. [0201]
Note that, for example, in a case where the image having the wide angle of view is obtained as the restored image, all the wide-angle pixels may be used, or a part of the wide-angle pixels may be used. Furthermore, for example, in a case where the image having the narrow angle of view is obtained as the restored image, all the narrow-angle pixels may be used, or a part of the narrow-angle pixels may be used. [0202]

Next, imaging processing by the imaging device 101 in Fig. 2 is described with reference to a flowchart in Fig. 20. [0203]
At step S1, the imaging element 121 images the subject. Therefore, the detection signal indicating the detection signal level corresponding to the amount of incident light from the subject is output from each pixel 121a of the imaging element 121 having different incident angle directivities, and the imaging element 121 supplies the detection image including the detection signal of

each pixel 121a to the restoration unit 122. [0204]
At step S2, the restoration unit 122 obtains the coefficient used for the image restoration. Specifically, the restoration unit 122 sets the distance to the subject surface 31 to be restored, that is, the subject distance. Note that, an arbitrary method may be adopted as a method of setting the subject distance. For example, the restoration unit 122 sets the subject distance input by the user via the input unit 124 or the subject distance detected by the detection unit 125 as the distance to the subject surface 31 to be restored. [0205]
Next, the restoration unit 122 reads the coefficient set group associated with the set subject distance from the storage unit 128. [0206]
At step S3, the restoration unit 122 restores the image using the detection image and the coefficient. Specifically, the restoration unit 122 uses the detection signal level of each pixel of the detection image and the coefficient set group obtained in the processing at step S2 to create the simultaneous equations described with reference to equations (1) to (3) or equations (4) to (6) described above. Next, the restoration unit 122 calculates the light intensity of each point light source on the subject surface 31 corresponding to the set subject distance by solving the created simultaneous equations. Then, by arranging the pixels having the pixel values according to the calculated light intensities according to the arrangement of the

respective point light sources on the subject surface 31, the restoration unit 122 generates the restored image formed as the image of the subject. [0207]
At step S4, the imaging device 101 performs various types of processing on the restored image. For example, the restoration unit 122 performs demosaic processing, γ correction, white balance adjustment, conversion processing to a predetermined compression format and the like on the restored image as necessary. Furthermore, the restoration unit 122 supplies the restored image to the display unit 127 and allows the same to display the image, supplies the restored image to the
recording/playback unit 129 and allows the same to record the image on the recording medium 130, or outputs the restored image to another device via the communication unit 131 as necessary, for example. [0208]
Thereafter, the imaging processing ends. [0209]
Note that, in the description above, the example of restoring the restored image from the detection image using the imaging element 121 and the coefficient set group associated with the subject distance is described; however, for example, it is also possible to further prepare the coefficient set group corresponding to the angle of view of the restored image as described above in addition to the subject distance and restore the restored image by using the coefficient set group according to the subject distance and the angle of view. Note that, the resolution with respect to the subject distance and the

angle of view depends on the number of prepared
coefficient set groups.
[0210]
Furthermore, in the description of the processing using the flowchart in Fig. 20, the example of using the detection signals of all the pixels included in the detection image is described; however, it is also possible to generate the detection image including the detection signal of the pixel having the incident angle directivity corresponding to the specified subject distance and angle of view among the pixels forming the imaging element 121 and restore the restored image by using the same. By such processing, it becomes possible to restore the restored image by the detection image suitable for the subject distance and the angle of view of the restored image to be obtained, and restoration accuracy and image quality of the restored image are improved. That is, in a case where the image corresponding to the specified subject distance and angle of view is the image corresponding to the angle of view SQ1 in Fig. 16, for example, by selecting the pixels 121a having the incident angle directivity corresponding to the angle of view SQ1 and restoring the restored image with the detection image obtained from them, it becomes possible to restore the image of the angle of view SQ1 with high accuracy. [0211]
By the processing described above, it becomes possible to realize the imaging device 101 having the imaging element 121 in which each pixel has incident angle directivity as an indispensable component.

[0212]
As a result, the imaging lens, the pinhole, and the optical filter disclosed in the above-described Patent Document and the like are not necessary, so that the degree of freedom in designing the device may be improved, and an optical element formed separately from the imaging element 121 and assumed to be mounted together with the imaging element 121 in a stage of forming the imaging device becomes not necessary, so that the device may be made compact in the incident angle of the incident light and a manufacturing cost may be decreased. Furthermore, a lens corresponding to an imaging lens for forming an optical image such as a focus lens becomes unnecessary. However, a zoom lens that changes magnification may be provided. [0213]
Note that, in the description above, the processing of restoring the restored image corresponding to the predetermined subject distance immediately after the detection image is captured is described; however, for example, it is also possible to restore the restored image by using the detection image at a desired timing after recording the detection image on the recording medium 130 or output the same to another device via the communication unit 131 without performing the restoring processing immediately. In this case, the restoration of the restored image may be performed by the imaging device 101 or another device. In this case, for example, it is possible to obtain the restored image for the subject surface of arbitrary subject distance and angle of view by obtaining the restored image by solving the

simultaneous equations created by using the coefficient set group according to arbitrary subject distance and angle of view, thereby realizing refocusing and the like. [0214]
For example, in a case where the imaging device including the imaging lens and the conventional imaging element is used, in order to obtain the image with various focal distances and angles of view, it is necessary to image while variously changing the focal distance and angle of view. On the other hand, in the imaging device 101, it is possible to restore the restored image of arbitrary subject distance and angle of view by switching the coefficient set group in this manner, so that processing of repeatedly imaging while variously changing the focal distance (that is, the subject distance) and the angle of view is not necessary. [0215]
In this case, for example, the user may also obtain the restored image of the desired subject distance and angle of view while allowing the display unit 127 to display the restored images that are restored while changing the coefficient set groups corresponding to the different subject distances and angles of view. [0216]
Note that, in a case of recording the detection image, when the subject distance and angle of view at the time of restoration are determined, the metadata used for restoration may be associated with the detection image. For example, by assigning the metadata to image data including the detection image, assigning the same ID to the detection image and the metadata, or recording the

detection image and the metadata on the same recording medium 130, the detection image and the metadata are associated with each other. [0217]
Note that, in a case where the same ID is assigned to the detection image and the metadata, it is possible to record the detection image and the metadata on different recording media or individually output them from the imaging device 101. [0218]
Furthermore, the metadata may include the coefficient set group used for restoration or not. In the latter case, for example, the subject distance and angle of view at the time of restoration are included in the metadata, and the coefficient set group corresponding to the subject distance and angle of view is obtained from the storage unit 128 and the like at the time of restoration. [0219]
Moreover, in a case where the restored image is restored immediately at the time of imaging, for example, an image to be recorded or externally output may be selected from the detection image and the restored image. For example, it is possible to record or externally output both the images or record or externally output only one of the images. [0220]
Furthermore, in a case of capturing a moving image also, it is possible to select whether or not to restore the restored image at the time of imaging, or to select the image to be recorded or externally output. For

example, it is possible to immediately restore the restored image of each frame, and record or externally output both or one of the restored image and the detection image before restoration while capturing the moving image. In this case, it is also possible to display the restored image of each frame as a through image at the time of imaging. Alternatively, for example, it is possible to record or externally output the detection image of each frame without performing restoration processing at the time of imaging. [0221]
Moreover, when capturing the moving image, for example, it is possible to select whether or not to restore the restored image, and select the image to be recorded or externally output for each frame. For example, it is possible to switch whether or not to restore the restored image for each frame. Furthermore, for example, it is possible to individually switch whether or not to record the detection image and whether or not to record the restored image for each frame. Furthermore, for example, it is also possible to record the detection images of all the frames while assigning the metadata to the detection image of a useful frame that may be used later. [0222]
Furthermore, it is also possible to realize an autofocus function as is the case with the imaging device using the imaging lens. For example, the autofocus function may be realized by determining the optimum subject distance by a hill-climbing method similar to a contrast auto focus (AF) method on the basis of the

restored image. [0223]
Moreover, it is possible to generate the restored image using the detection image captured by the imaging element 121 having incident angle directivities in a pixel unit as compared to the imaging device and the like including the optical filter disclosed in above-described Patent Document and the like and the conventional imaging element, so that it becomes possible to realize an increase in pixels or obtain the restored image with high resolution and high angular resolution. In contrast, in the imaging device including the optical filter and the conventional imaging element, it is difficult to realize the high resolution restored image and the like because it is difficult to miniaturize the optical filter even if the pixels are miniaturized. [0224]
Furthermore, in the imaging device 101 of the present disclosure, the imaging element 121 is the indispensable configuration, and does not require, for example, the optical filter and the like disclosed in Patent Document and the like described above, so that the optical filter is not bent by heat due to a high-temperature usage environment, and it is possible to realize the imaging device with high environmental resistance. [0225]
Moreover, the imaging device 101 of the present disclosure does not require the imaging lens, pinhole, and optical filter disclosed in Patent Document and the like described above, so that it becomes possible to

improve the degree of freedom in designing a configuration having an imaging function. [0226]
By the way, in a case where the light-shielding range (that is, the incident angle directivity) of the light-shielding film 121b of each pixel 121a of the imaging element 121 has randomness, as disorder of a difference in the light-shielding range is larger, a load of the processing by the restoration unit 122 is larger. Therefore, it is possible to reduce the disorder, thereby reducing the processing road by making a part of the change in the light-shielding range of the light-shielding film 121b of each pixel 121a regular. [0227]
For example, the L-shaped light-shielding film 121b obtained by combining the longitudinal band-type and the lateral band-type is formed, and the lateral band-type light-shielding films 121b having the same width are combined in a predetermined column direction and the longitudinal band-type light-shielding films 121b having the same height are combined in a predetermined row direction. Therefore, the light-shielding range of the light-shielding film 121b of each pixel 121a changes randomly in a pixel unit while having regularity in the column direction and the row direction. As a result, a difference in the light-shielding range of the light-shielding film 121b of each pixel 121a, that is, the disorder in the difference of the incident angle directivity may be reduced, and the processing load of the restoration unit 122 may be reduced.

[0228]
Specifically, for example, as illustrated in an imaging element 121'' in Fig. 21, lateral band-type light-shielding films 121b having the same width X0 are used for the pixels in the same column indicated by a range Z130, and the longitudinal band-type light-shielding film 121b having the same height Y0 are used for the pixels in the same row indicated by a range Z150. As a result, for the pixel 121a specified by each row and column, the L-shaped light-shielding film 121b obtained by combining them is used. [0229]
Similarly, the lateral band-type light-shielding film 121b having the same width X1 are used for the pixels in the same column indicated by a range Z131 adjacent to the range Z130, and the longitudinal band-type light-shielding film 121b having the same height Y1 are used for the pixels in the same row indicated by a range Z151 adjacent to the range Z150. As a result, for the pixel 121a specified by each row and column, the L-shaped light-shielding film 121b obtained by combining them is used. [0230]
Moreover, the lateral band-type light-shielding film 121b having the same width X2 are used for the pixels in the same column indicated by a range Z132 adjacent to the range Z131, and the longitudinal band-type light-shielding film 121b having the same height Y2 are used for the pixels in the same row indicated by a range Z152 adjacent to the range Z151. As a result, for the pixel 121a specified by each row and column, the L-

shaped light-shielding film 121b obtained by combining
them is used.
[0231]
By doing so, it is possible to set the range of the light-shielding film to different values in a pixel unit while allowing the width and position in the horizontal direction and the height and the position in the vertical direction of the light-shielding film 121b to have regularity, so that it is possible to control the disorder in the change in the incident angle directivity. As a result, it becomes possible to reduce patterns of the coefficient sets and reduce the processing load of arithmetic processing in the restoration unit 122. [0232]
In further detail, as illustrated in an upper right part of Fig. 22, in a case of obtaining a restored image of N×N pixels from a detection image Pic of N pixels × N pixels, a relationship illustrated in a left part of Fig. 22 is established by a vector X having pixel values of the respective pixels of the restored image of (N×N) rows × one column as elements, a vector Y having pixel values of the respective pixels of the detection image of (N×N) rows × one column as elements, and a matrix A of (N×N) rows × (N×N) columns including the coefficient set group. [0233]
That is, Fig. 22 illustrates that a result obtained by multiplying the respective elements of the matrix A of (N×N) rows × (N×N) columns including the coefficient set group by the vector X of (N×N) rows × one column representing the restored image is the vector Y of (N×N) rows × one column representing the detection image.

Then, from this relationship, for example, the simultaneous equations corresponding to equations (1) to (3) or equations (4) to (6) described above are formed. [0234]
Note that, Fig. 22 illustrates that each element of the first column indicated by a range Z201 of the matrix A corresponds to the element of the first row of the vector X, and each element of the N×N-th column indicated by a range Z202 of the matrix A corresponds to the element of the N×N-th row of the vector X. [0235]
Note that, in a case of using the pinhole, and in a case of using a condensing function for allowing the incident light incident in the same direction such as the imaging lens to be incident on both adjacent pixel output units, a relationship between the position of each pixel and the incident angle of the light is uniquely determined, so that the matrix A is a diagonal matrix in which all rightward falling diagonal components are one. On the other hand, in a case where neither the pinhole nor the imaging lens is used as in the imaging device 101 in Fig. 2, the relationship between the position of each pixel and the incident angle of light is not uniquely determined, so that the matrix A is not the diagonal matrix. [0236]
In other words, the restored image may be obtained by solving the simultaneous equations based on a determinant illustrated in Fig. 22 and obtaining each element of the vector X. [0237]

By the way, in general, the determinant in Fig. 22 is transformed as illustrated in Fig. 23 by multiplying both sides by an inverse matrix A-1 of the matrix A from the left, and each element of the vector X being the detection image is obtained by multiplying the vector Y of the detection image by an inverse matrix A-1 from the left.
[0238]
However, in reality, there is a case where the matrix A cannot be obtained correctly, the matrix A cannot be measured correctly, a basis vector of the matrix A nearly linearly dependent and it is not possible to solve, and each element of the detection image includes noise. Then, for any of these reasons or a combination thereof, the simultaneous equations might not be solved.
[0239]
Therefore, for example, considering a robust configuration with respect to various errors, following equation (7) using the concept of the regularized least-square method is used.
[0240]
[Mathematical Expression 1]

[0241]
Here, x with at the top in equation (7) represents the vector X, A represents the matrix A, Y represents the vector Y, γ represents a parameter, ||A|| represents a L2 norm (square-root of sum root squares). Here, a first term on the right side is a norm when

minimizing both sides in Fig. 22, and a second term on
the right side is a regularization term.
[0242]
When this equation (7) is solved for x, following equation (8) is obtained. [0243] [Mathematical Expression 2]

[0244]
Here, At represents a transposed matrix of the matrix A, and I represents a unit matrix. [0245]
However, since the matrix A has an enormous size, a calculation amount and a required memory amount are large. [0246]
Therefore, for example, as illustrated in Fig. 24, the matrix A is decomposed into a matrix AL of N rows × N columns and a matrix ART of N rows × N columns, and they are multiplied from former and latter stages of the matrix X of N rows × N columns representing the restored image, and the matrix Y of N rows × N columns representing the detection image is obtained as a result. Therefore, for the matrix A of the number of elements (N×N)×(N×N), the matrices AL and ART having the number of elements (N×N) are obtained, and the number of elements in each matrix becomes 1/(N×N). As a result, a calculation amount and the required memory amount may be reduced. [0247]

The determinant illustrated in Fig. 24 is realized, for example, by making the matrix in parentheses in equation (8) the matrix AL and making the inverse matrix of the transposed matrix of the matrix A the matrix ART. [0248]
In the calculation illustrated in Fig. 24, as illustrated in Fig. 25, an element group Z222 is obtained by multiplying an element of interest Xp in the matrix X by each element group Z221 of the corresponding column of the matrix AL. Moreover, a two-dimensional response Z224 corresponding to the element of interest Xp is obtained by multiplying the element group Z222 by the elements in the row corresponding to the element of interest Xp of the matrix ART. Then, the matrix Y is obtained by integrating the two-dimensional responses Z224 corresponding to all the elements of the matrix X. [0249]
Therefore, for example, in the element group Z221 of each column of the matrix AL, a coefficient corresponding to the incident angle directivity of the lateral-band type pixel 121a set to have the same width for each column of the imaging elements 121 illustrated in Fig. 21 is used. [0250]
Similarly, for example, in the element group Z223 of each row of the matrix ART, a coefficient corresponding to the incident angle directivity of the longitudinal-band type pixel 121a set to have the same height for each row of the imaging elements 121 illustrated in Fig. 21 is used. [0251]

As a result, since it becomes possible to reduce the matrix used when restoring the restored image on the basis of the detection image, the calculation amount may be reduced, a processing speed may be improved, and power consumption for the calculation may be reduced. Furthermore, since the matrix may be reduced, a capacity of the memory used for the calculation may be reduced, and a device cost may be reduced. [0252]
Note that, although Fig. 21 illustrates the example of changing the light-shielding range (light-receiving range) in a pixel unit while providing predetermined regularity in the horizontal direction and the vertical direction, in the present disclosure, the light-shielding range (light-receiving range) not completely randomly set in a pixel unit but randomly set to a certain degree in this manner is also considered to be randomly set. In other words, in the present disclosure, not only a case where the light-shielding range (light-receiving range) is set completely at random in a unit of pixel, but also a case at random to a certain degree (for example, a case where a part of all the pixels has a range with regularity but other range is at random), or a case apparently not regular to a certain degree (a case of arrangement in which it is not possible to confirm arrangement according to the regularity as described with reference to Fig. 21 among all the pixels) are also considered to be random. [0253]
<<3. Embodiment>>
Next, an embodiment of the present disclosure is

described with reference to Figs. 26 to 35. [0254]
As described above, an imaging element 121 using a pixel having incident angle directivity does not require an imaging lens, an optical filter and the like, so that a degree of freedom in arrangement of respective pixels 121a is high. [0255]
Therefore, in this embodiment, a subject is imaged by a plurality of imaging devices including the imaging element having the incident angle directivity, and a restored image is restored using a detection signal obtained by each imaging device. [0256]

Fig. 26 is a block diagram illustrating a configuration example of an imaging system 301 according to the embodiment of the present disclosure. [0257]
The imaging system 301 includes an imaging device group 311 and a signal processing device 312 configured as separate casings. [0258]
The imaging device group 311 includes two or more n imaging devices 321-1 to 321-n. Note that, hereinafter, in a case where it is not necessary to distinguish the imaging devices 321-1 to 321-n from one another, they are simply referred to as the imaging devices 321. [0259]
Each imaging device 321 is installed in predetermined position and orientation using, a jig and

the like, for example. Alternatively, each imaging device 321 may be appropriately installed, and the position and orientation of each imaging device 321 may be detected by a predetermined method. Furthermore, each imaging device 321 includes the imaging element 121 having the incident angle directivity described above, generates an image file including a detection signal set including a detection signal output from each pixel 121a of each imaging element 121, and transmits the same to the signal processing device 312. [0260]
Here, the image file is a data set that includes image data and includes metadata corresponding to the image data as necessary. The image data includes, for example, at least one of the detection signal set, a detection image, or the restored image. The metadata includes, for example, an ID for identifying each imaging device 321, a coefficient set group corresponding to the image data and the like. [0261]
The signal processing device 312 controls imaging by each imaging device 321, obtains the image file from each imaging device 321, and performs restoration processing and the like of the restored image using the obtained image file. [0262]

Fig. 27 is a block diagram illustrating a configuration example of the imaging device 321 in Fig. 26. Note that, in the drawing, a portion corresponding to that of the imaging device 101 in Fig. 2 is assigned

with the same reference sign, and the description thereof
is omitted as appropriate.
[0263]
The imaging device 321 includes an imaging element 121, a control unit 411, an association unit 413, a storage unit 414, and a communication unit 414. Furthermore, the control unit 411, the association unit 413, the storage unit 414, and the communication unit 414 form a signal processing control unit 401. Note that, the imaging device 321 does not include an imaging lens. [0264]
Furthermore, the imaging element 121, the control unit 411, the association unit 413, the storage unit 414, and the communication unit 414 are connected to one another via a bus B2, and perform transmission, reception and the like of data via the bus B2. Note that, hereinafter, in order to simplify the description, description of the bus B2 in a case where each unit of the imaging device 321 performs the transmission, reception and the like of the data via the bus B2 is omitted. [0265]
The imaging element 121 outputs the detection signal set including the detection signal output from each pixel 121a to the bus B2. [0266]
The control unit 411 includes, for example, various processors and controls each unit of the imaging device 321. [0267]
The input unit 412 includes an input device (for

example, a key, a switch, a button, a dial, a touch panel, a remote controller and the like) for operating the imaging device 321, inputting data used for processing and the like. The input unit 412 outputs an operation signal, the input data and the like to the bus B2. [0268]
The association unit 413 associates the detection signal set obtained by the imaging element 121 with the metadata corresponding to the detection signal set in cooperation with an association unit 504 (Fig. 29) of the signal processing device 312 or alone. [0269]
The storage unit 414 includes one or more storage devices such as a read only memory (ROM), a random access memory (RAM), and a flash memory, and stores, for example, programs, data and the like used for processing of the imaging device 321. The storage unit 414 stores, for example, the ID for uniquely identifying the imaging device 321. [0270]
The communication unit 415 communicates with other devices (for example, other imaging device 321, signal processing device 312 and the like) by a predetermined communication method. Note that, the communication method of the communication unit 415 may be wired or wireless. Furthermore, the communication unit 415 may support a plurality of communication methods. [0271]

Fig. 28 illustrates an example of a pattern of a light-shielding film 121b in a pixel array unit of imaging elements 121-1 to 121-n of imaging devices 321-1 to 321-n, respectively. [0272]
In this example, in each pixel 121a, the light-shielding film 121b is arranged so as to surround four sides, and a rectangular opening is provided in the pixel 121a. The opening is set to have the same size in all the pixels 121a in the same imaging element 121, and is set to have different sizes between the pixels 121a of different imaging elements 121. Furthermore, positions of the openings of the respective pixels 121a in the same imaging element 121 are all different. Accordingly, at least one of the size or position of the openings is different among all the pixels 121a of the imaging elements 121-1 to 121-n, and the incident angle directivities of all the pixels 121a are different from each other. Then, imaging is performed by a plurality of imaging devices 321 having the different incident angle directivities. Therefore, the number of equations forming simultaneous equations used to restore the restored image may be increased, and as a result, the number of pixels of the restored image may be increased, so that resolution of the restored image is improved. [0273]
Note that, the incident angle directivities of all the pixels 121a are not necessarily different, and some of them may be the same. However, as the number of pixels 121a with the same incident angle directivity is smaller, the number of equations forming the simultaneous

equations used to restore the restored image may be increased, and the number of pixels of the restored image may be increased, and as a result, the resolution of the restored image is improved. [0274]

Fig. 29 is a block diagram illustrating a configuration example of the signal processing device 312 in Fig. 26. The signal processing device 312 includes a restoration unit 501, a control unit 502, an input unit 503, an association unit 504, a display unit 505, a storage unit 506, a recording/playback unit 507, a recording medium 508, and a communication unit 509. [0275]
The restoration unit 501, the control unit 502, the input unit 503, the display unit 505, the storage unit 506, the recording/playback unit 507, and the communication unit 509 are connected to one another via a bus B3, and perform transmission, reception and the like of data via the bus B3. Note that, hereinafter, in order to simplify the description, description of the bus B3 in a case where each unit of the signal processing device 312 performs the transmission, reception and the like of the data via the bus B3 is omitted. [0276]
The restoration unit 501 performs the restoration processing and the like of the restored image by processing similar to that of the restoration unit 122 of the imaging device 101 in Fig. 2 by using the detection signal set included in the image file received from each

imaging device 321. The restoration unit 501 outputs the
restored image to the bus B3.
[0277]
The control unit 502 includes, for example, various processors and controls each unit of the signal processing device 312. [0278]
The input unit 503 includes an input device (for example, a key, a switch, a button, a dial, a touch panel, a remote controller and the like) for operating the signal processing device 312, inputting data used for processing and the like. The input unit 503 outputs an operation signal, the input data and the like to the bus B3. [0279]
The association unit 504 associates the detection signal set obtained from each imaging device 321 with the metadata corresponding to each detection signal set in cooperation with the association unit 413 of the imaging device 321 or alone. [0280]
The display unit 505 includes a display, for example, and displays various types of information (for example, the restored image and the like). Note that, the display unit 505 may include an audio output unit such as a speaker to output audio. [0281]
The storage unit 506 includes one or more storage devices such as a ROM, a RAM, and a flash memory, and stores, for example, programs and data used for processing of the signal processing device 312.

[0282]
For example, the storage unit 506 stores installation information regarding an installation state of each imaging device 321. The installation information includes, for example, information regarding a position and orientation of a light-receiving surface of the imaging element 121 of each imaging device 321. [0283]
Note that, for example, the installation information is created when the imaging device 321 is installed and the like, and stored in the storage unit 506. Alternatively, for example, each imaging device 321 may detect its own installation state using a position sensor such as a global navigation satellite system (GNSS), an orientation sensor and the like and transmit a detection result to the signal processing device 312, and the signal processing device 312 may generate the installation information on the basis of the information received from each imaging device 321. Alternatively, for example, the signal processing device 312 or another device may detect the installation state of each imaging device 321 on the basis of an image obtained by imaging each imaging device 321 and the like, and generate the installation information on the basis of a detection result. [0284]
Furthermore, the storage unit 506 stores, for example, a coefficient set group corresponding to each imaging element 121 of each imaging device 321. Note that, a method of setting the coefficient corresponding to each imaging element 121 of each imaging device 321 is

described later with reference to Figs. 30 to 33. [0285]
The recording/playback unit 507 records the data on the recording medium 508 and plays back (reads out) the data recorded on the recording medium 508. For example, the recording/playback unit 507 records the restored image on the recording medium 508 or reads out the same from the recording medium 508. Furthermore, for example, the recording/playback unit 507 records the detection signal set and the corresponding metadata on the recording medium 508 or reads out the same from the recording medium 508. [0286]
The recording medium 508 includes, for example, any one of a HDD, an SSD, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory or the like, a combination thereof or the like. [0287]
The communication unit 509 communicates with another device (for example, each imaging device 321 and the like) by a predetermined communication method. Note that, the communication method of the communication unit 509 may be wired or wireless. Furthermore, the communication unit 509 may support a plurality of communication methods. [0288]

Next, with reference to Figs. 30 to 33, the method of setting the coefficient corresponding to each imaging element 121 of each imaging device 321 is described.

[0289]
Here, first, an example of definition of infinity is described with reference to Fig. 30.
[0290]
In this example, the imaging elements 121-1 to 121-n are arranged such that the light-receiving surfaces thereof are arranged in the same straight line in the same plane. Furthermore, a point light source P of the subject is present at a position away from the center of a column of the light-receiving surfaces of the respective imaging elements 121 by a distance Ds1. Note that, hereinafter, a distance from a left end of the light-receiving surface of the imaging element 121-1 to a right end of the light-receiving surface of the imaging element 121-n in the drawing is set to Dw.
[0291]
In this case, an angle θw indicating a range of incident light incident on each imaging element 121 out of the incident light emitted from the point light source P is expressed by following equation (9).
[0292]

[0293]
Here, assuming that the number of pixels of the restored image to be restored using the detection signal of the pixel 121a of each imaging element 121 is 1,000 pixels vertically × 1,000 pixels horizontally, and an angle of view of the restored image is ±20 degrees, angular resolution between the adjacent pixels of the restored image is 0.04 degrees. In this case, the distance Ds1 at which the angle θw in equation (9) is

0.04 degrees which is the angular resolution between the adjacent pixels of the restored image or smaller may be regarded as infinity. [0294]
Therefore, if the angular resolution between the pixels of the restored image is set to θr, the point light source P may be regarded to be present at infinity if following equation (10) is satisfied. [0295]

[0296]
When equation (10) is solved for the distance Ds1, following equation (11) is obtained. [0297]

[0298]
Furthermore, in a case where the subject is present at infinity, the incident light from each point light source of the subject may be regarded as parallel light, and the incident angle of the incident light from each point light source on each pixel 121a of each imaging element 121 is the same.
[0299]
For example, as illustrated in Fig. 31, the incident light from a point light source PA present at infinity with respect to the pixel 121a-1 and the pixel 121a-2 located at distant positions may be regarded as parallel light, and the incident angle is the same on the pixels 121a-1 and 121a-2. Similarly, the incident light from a point light source PB present at infinity with respect to the pixel 121a-1 and the pixel 121a-2 may be

regarded as parallel light, and the incident angle is the
same on the pixels 121a-1 and 121a-2.
[0300]
Therefore, the incident angle of the incident light from each point light source of the subject present at infinity does not change depending on the position of (the imaging element 121 of) each imaging device 321. [0301]
Note that, a range Ws (angle of view) between the point light source PA and the point light source PB the incident light from which the pixel 121a-1 and the pixel 121a-2 may receive is determined by a light-shielding range of each pixel 121a and the like as described above with reference to Fig. 15. [0302]
In contrast, as illustrated in Fig. 32, the incident angle of the incident light from the point light source PA on a subject surface 31 present at a distance closer than infinity (distance less than infinity) is different depending on the position of the imaging element 121 of each imaging device 321. More precisely, the incident angle of the incident light from the point light source PA differs depending on the position of each pixel 121a of each imaging element 121. Specifically, an incident angle θ1 of the incident light incident on each pixel 121a from the point light source PA is expressed by following equation (12). [0303]

[0304]
Here, Ds2 represents a distance between the subject

surface 31 and the light-receiving surface of each imaging element 121. Xa represents the position of the point light source PA on the subject surface 31, Xp represents the position of the pixel 121a on a plane parallel to the subject surface 31, and Xa-Xp represents a distance between the point light source PA and the pixel 121a in a direction parallel to the subject surface 31. [0305]
Furthermore, in Fig. 32 and Fig. 33 described later, white circles on the subject surface 31 schematically indicate the positions of the point light sources, and shaded circles on the light-receiving surfaces of the imaging element 121-1 and the imaging element 121-n schematically indicate the positions of the pixels 121a. [0306]
As expressed in equation (12), the incident angle θ1 varies depending on the subject distance Ds2 and relative positions (Xa-Xp) between the point light source PA and the pixel 121a. Therefore, the incident angle from each point light source of the subject present at the distance closer than infinity varies depending on the subject distance and the position (of the imaging element 121) of the imaging device 321. [0307]
Furthermore, in a case where the subject is present at infinity and in a case where this is present at the distance closer than infinity, the incident angle of the incident light from the subject varies depending on the orientation of the imaging device 321, more precisely,

the orientation of the light-receiving surface of the imaging element 121 of the imaging device 321 with respect to the subject. [0308]
For example, as illustrated in Fig. 33, in a case where the orientation of the light-receiving surface of the imaging element 121-n is inclined by the angle θr with respect to the subject surface 31, the incident angle θ2 of the incident light from the point light source PA on each pixel 121a of the imaging element 121-n is expressed by following equation (13). [0309]

[0310]
That is, the incident angle θ2 is an angle obtained by adding the inclination θr of the light-receiving surface of the imaging element 121-n with respect to the subject surface 31 to the incident angle θ1 of equation
(12) . This is similar also in a case where the subject is present at infinity.
[0311]
As described above, the coefficient set group corresponding to each imaging element 121 of each imaging device 321 with respect to the subject present at infinity varies depending on the orientation of the light-receiving surface of the imaging element 121 with respect to the subject surface to be restored. In contrast, the coefficient set group corresponding to each imaging element 121 of each imaging device 321 with respect to the subject present at the distance closer from infinity varies depending on the subject distance

from the subject surface to be restored and the position and orientation of the light-receiving surface of each imaging element 121. [0312]
Therefore, in a case where the position and orientation of each imaging device 321 are fixed, the coefficient set group corresponding to each imaging element 121 of each imaging device 321 is set for each combination of the subject distance of the subject surface to be restored and the orientation of the subject surface (hereinafter, referred to as a subject direction). Then, for example, the coefficient set group corresponding to each imaging element 121 is prepared for each combination of the subject distance and subject direction to be stored in the storage unit 506. [0313]
Note that, hereinafter, a case where the position and orientation of each imaging device 321 are fixed is described, and a case where the position and orientation of each imaging device 321 are variable is described later. [0314]
Next, processing of the imaging system 301 is described with reference to Figs. 34 and 35. [0315]
First, processing of the signal processing device 312 is described with reference to a flowchart in Fig. 34. [0316]

At step S101, the signal processing device 312 transmits an imaging command. Specifically, the control unit 502 generates the imaging command and transmits the same to each imaging device 321 via the communication unit 509. [0317]
Each imaging device 321 receives the imaging command at step S151 in Fig. 35 to be described later and transmits the image file at step S153. [0318]
Note that, each image file includes, for example, the ID for identifying each imaging device 321 and the detection signal set obtained by the imaging element 121 of each imaging device 321. [0319]
At step S102, the communication unit 509 receives the image file transmitted from each imaging device 321. The communication unit 509 supplies the received image file to the restoration unit 501. [0320]
At step S103, the restoration unit 501 obtains a coefficient used for image restoration. Specifically, the restoration unit 501 sets a combination of the distance to the subject surface 31 to be restored (subject distance) and orientation (subject direction). Note that, an arbitrary method may be adopted as a method of setting the combination of the subject distance and subject direction. For example, the restoration unit 501 sets the subject distance and subject direction input by a user via the input unit 503 as the combination of the subject distance and subject direction of the subject

surface 31 to be restored. [0321]
Next, the restoration unit 501 reads out the coefficient set group associated with the set combination of the subject distance and subject direction from the storage unit 506. [0322]
At step S104, the restoration unit 501 restores one image using the detection signal set and the coefficient. That is, the restoration unit 501 restores one restored image by using one simultaneous equation using a plurality of detection signal sets included in the image file received from each imaging device 321 and the coefficient set group obtained in the processing at step S103 by the processing similar to that by the restoration unit 122 of the imaging device 101 at step S3 in Fig. 20. [0323]
At step S105, the signal processing device 312 performs various types of processing on the restored image. That is, the signal processing device 312 performs various types of processing on the restored image by the processing similar to that of the imaging device 101 at step S4 in Fig. 20. [0324]
Thereafter, the processing of the signal processing device 312 ends. [0325]
Next, with reference to a flowchart in Fig. 35, processing executed by each imaging device 321 corresponding to the processing of the signal processing device 312 in Fig. 34 is described.

[0326]
At step S151, the communication unit 415 receives the imaging command transmitted from the signal processing device 312 at step S101 in Fig. 34. The communication unit 415 supplies the received imaging command to the control unit 411. [0327]
At step S152, the imaging element 121 images the subject as in the processing at step S1 in Fig. 20. The imaging element 121 supplies the detection signal set including the detection signals of the respective pixels 121a to the association unit 413. [0328]
At step S153, the imaging device 321 transmits the image file. For example, the association unit 413 generates the image file including the ID of the imaging device 321 and the detection signal set obtained from the imaging element 121. Therefore, the detection signal set and the ID of the imaging device 321 are associated with each other. The association unit 413 transmits the generated image file to the signal processing device 312 via the communication unit 415. [0329]
Thereafter, the processing of the imaging device 321 ends. [0330]
In the above-described manner, it is possible to image by a plurality of imaging devices 321 in a shared manner, and restore the restored image using the detection signal set obtained by each imaging device 321. [0331]

As a result, it is possible to increase the number of pixels of the restored image and to improve an image quality of the restored image, more specifically, resolution of the restored image as compared with a case where imaging is performed by only one imaging device 321. [0332]
Furthermore, the number of pixels of each imaging element 121 of each imaging device 321 may be reduced to reduce a size. Alternatively, for example, by increasing the number of imaging devices 321 without increasing the number of pixels of each imaging element 121 of each imaging device 321, it is possible to increase the number of pixels of the restored image. Therefore, for example, introduction of a new manufacturing device and the like associated with an increase in size of the imaging element 121 is not required, and a manufacturing cost may be reduced. Furthermore, each imaging device 321 may be made compact, a degree of freedom of arrangement of each imaging device 321 is improved, and an application range of the imaging device 321 may be expanded in the field of Internet of things (IoT) and the like, for example. [0333]
Note that, for example, it is possible to further prepare the coefficient set group corresponding to the angle of view of the restored image as described above in addition to the subject distance and the subject direction and restore the restored image by further using the coefficient set group according to the angle of view. [0334]
<<4. Variation>>

Hereinafter, a variation of the embodiments of the present disclosure described above is described. [0335]

Although the example in which the pattern of the light-shielding film 121b of the imaging element 121 of each imaging device 321 is different is illustrated in Fig. 28, for example, as illustrated in Fig. 36, it is also possible to use the same imaging element 121 for each imaging device 321 to make the pattern of the light-shielding film 121b of each imaging element 121 the same. In this case, imaging is performed by a plurality of imaging devices 321 having the same incident angle directivity. [0336]
In this case, the coefficient set group corresponding to each of all the imaging elements 121 is the same for a subject present at infinity. Therefore, equations forming simultaneous equations used to restore a restored image overlap, and the number of equations cannot be increased, so that it is not possible to increase the number of pixels of the restored image. However, it is assumed that a detection signal level is different between pixels 121a at the same position of each imaging element 121 due to an effect of noise and the like. As a result, in equations corresponding to the pixels 121a at the same position of each imaging element 121 (for example, equations (1) to (6) described above), even if right sides overlap, a detection signal levels on left sides are different. Then, by solving the

simultaneous equations using the equations in which the right sides overlap, it is possible to inhibit variation in the detection signal level of each pixel 121a due to noise and the like. As a result, noise tolerance of the restored image is improved and the image quality is improved. [0337]
On the other hand, as for the subject present at a distance closer than infinity, an incident angle of incident light from the subject on each imaging element 121 is different, so that the coefficient set group corresponding to each imaging element 121 is different. Therefore, the number of equations forming the simultaneous equations used to restore the restored image increases, so that it is possible to increase the number of pixels of the restored image. [0338]
Note that, for example, the incident angle directivity of some of the imaging elements 121 out of a plurality of imaging elements 121 may be made the same, and some of the imaging elements 121 may have different incident angle directivities. [0339]

Furthermore, for example, in a case where a position and orientation of each imaging device 321 are variable, as described above with reference to Figs. 32 and 33, the coefficient set group corresponding to each imaging element 121 vary depending on the position and orientation of the imaging device 321, more precisely, a position and orientation of a light-receiving surface of

the imaging element 121 of the imaging device 321 in addition to the subject distance and subject direction. Therefore, in a case where the position and orientation of each imaging device 321 are variable, for example, it is possible to prepare the coefficient set group corresponding to each imaging element 121 for each combination of a subject distance and a subject direction, and the position and orientation of the imaging element 121 of each imaging device 321. Then, for example, a restoration unit 501 may read out the coefficient set group corresponding to the imaging element 121 of each imaging device 321 from a storage unit 506 on the basis of the set subject distance and subject direction, and the position and orientation of the imaging element 121 of each imaging device 321, and restore the restored image using the read coefficient set group. [0340]
Alternatively, for example, it is possible that the restoration unit 501 calculates the coefficient corresponding to each pixel 121a of the imaging element 121 of the imaging device 321 by using the characteristics of the weights Wx and Wy described above with reference to Fig. 9 and the like on the basis of the subject distance and subject direction, and the position and orientation of the imaging element 121 of each imaging device 321 without preparing in advance the coefficient set group for each combination described above. [0341]
Note that, for example, each imaging device 321 may

detect an installation state including the position and orientation at the time of imaging of each imaging device 321 and include metadata including a detection result in an image file to transmit to a signal processing device 312. Alternatively, for example, the signal processing device 312 or another device may detect the installation state of each imaging device 321 on the basis of an image obtained by imaging each imaging device 321 and the like. [0342]
Furthermore, for example, in a case where the subject distance, subject direction, and an angle of view at the time of restoration are determined, each imaging device 321 may include the metadata including the coefficient set group corresponding to the subject distance, subject direction, and angle of view in the image file to transmit to the signal processing device 312. [0343]

Next, a variation regarding a system configuration is described with reference to Figs. 37 to 39. [0344]
Fig. 37 illustrates a configuration example of an imaging system 601 that is a first variation of an imaging system. Note that, in the drawing, a portion corresponding to that of the imaging system 301 in Fig. 26 is assigned with the same reference sign, and the description thereof is omitted as appropriate. [0345]
The imaging system 601 is different from the imaging system 301 in Fig. 26 in that an imaging device

group 611 is provided in place of the imaging device group 311 and the signal processing device 312 is deleted. The imaging device group 611 is different from the imaging device group 311 in that an imaging device 621 is added. [0346]
The imaging device 621 has a configuration similar to that of the imaging device 101 in Fig. 2, for example. Then, the imaging device 621 transmits the imaging command to each imaging device 321 as is the case with the signal processing device 312 in Fig. 26 to control imaging of the subject by each imaging device 321 and images the subject by itself, too. Then, the imaging device 621 receives the image file from each imaging device 321 and restores one restored image by using the detection signal set included in the received image file and the detection signal set obtained by imaging by itself. [0347]
Note that, in the imaging system 601, the number of imaging devices 321 may be one. [0348]
Fig. 38 illustrates a configuration example of an imaging system 651 that is a second variation of the imaging system. Note that, in the drawing, a portion corresponding to that of the imaging system 301 in Fig. 26 is assigned with the same reference sign, and the description thereof is omitted as appropriate. [0349]
The imaging system 651 is different from the imaging system 301 in Fig. 26 in that an imaging device

group 661 is provided in place of the imaging device group 311. The imaging device group 661 is different from the imaging device group 311 in that an imaging device 671 is added. [0350]
The imaging device 671 has a configuration similar to that of the imaging device 321 in Fig. 27, for example. Then, the imaging device 671 transmits the imaging command to each imaging device 321 as is the case with the signal processing device 312 in Fig. 26 to control imaging of the subject by each imaging device 321, and images the subject by itself, too. Then, the imaging device 621 receives the image file from each imaging device 321 and transmits the image file received from each imaging device 321 and the image file including the detection signal set obtained by imaging by itself to the signal processing device 312. That is, the imaging device 621 plays a role of relaying the image file from each imaging device 321 to the signal processing device 312. [0351]
The signal processing device 312 restores the restored image using the detection signal data included in each received image file. [0352]
Note that, in the imaging system 651, the number of imaging devices 321 may be one. [0353]
Furthermore, in the above description, the example in which each imaging device 321 images in synchronization according to an external command in the

imaging system 301, the imaging system 601, and the imaging system 651 is described; however, it is also possible that the respective imaging devices 321 communicate with each other to image in synchronization or each imaging device 321 separately image out of synchronization, for example. [0354]
Moreover, in the above description, the example in which the respective imaging elements 121 are provided in different imaging devices is described; however, for example, a plurality of imaging elements 121 may be provided in one imaging device. [0355]
Fig. 39 illustrates a configuration example of an imaging device 701 provided with a plurality of imaging elements 121. Note that, in the drawing, a portion corresponding to that of the imaging device 101 in Fig. 2 is assigned with the same reference sign, and the description thereof is omitted as appropriate. [0356]
The imaging device 701 is different from the imaging device 101 in Fig. 2 in including an imaging unit 711 including two or more n imaging elements 121-1 to 121-n. [0357]
Each imaging element 121 supplies the detection signal set obtained by imaging the subject to the restoration unit 122 or outputs the same to a bus B1. [0358]
The restoration unit 122 restores the restored image using the detection signal set obtained from each

imaging element 121. [0359]
Note that, each imaging element 121 is installed in a casing of the imaging device 701. [0360]
Furthermore, for example, the imaging devices 321-1 to 321-n in Fig. 26, Fig. 37, or Fig. 38, the imaging device 621 in Fig. 37, and the imaging device 671 in Fig. 38 may be provided with a plurality of imaging elements 121. [0361]
Moreover, although the example of restoring the restored image using a plurality of detection signal sets obtained by imaging using a plurality of imaging elements 121 having different installation states is described in the description above, it is also possible, for example, to restore the restored image using a plurality of detection signal sets obtained by imaging by the same imaging element 121 in a plurality of installation states in which at least one of the position or orientation with respect to the subject is different. [0362]
For example, it is possible to restore the restored image by using a plurality of detection signal sets obtained by the imaging element 121 or the imaging device including the imaging element 121 imaging the same subject while moving. [0363]
Alternatively, for example, it is also possible to restore the restored image by using a plurality of detection signal sets obtained by the imaging element 121

or the imaging device including the imaging element 121 imaging in the same position while changing the orientation with respect to the subject. [0364]
Alternatively, for example, it is also possible to restore the restored image by using a plurality of detection signal sets obtained by the imaging element 121 or the imaging device including the imaging element 121 imaging while changing the distance from the subject. [0365]
Furthermore, it is possible that, at the time of imaging, association of each detection signal set with the metadata corresponding to each detection signal set is performed, and the restored image is restored using each detection signal set at a desired timing after imaging. [0366]
Note that, a method of associating the detection signal set with the metadata is not especially limited as long as a correspondence relationship between them may be specified. For example, by putting the detection signal set and the metadata in the same image file, assigning the same ID to the detection signal set and the metadata, or recording the detection signal set and the metadata on the same recording medium, the detection signal set and the metadata are associated with each other. [0367]
Note that, the detection signal set and the metadata may be recorded by each imaging device that images the subject, or may be recorded by the signal processing device or the imaging device that restores the

image. In a case where the detection signal set and metadata are recorded by each imaging device that images the subject, for example, the recorded detection signal set and metadata are supplied to the signal processing device or imaging device that restores the image via a recording medium. [0368]
Furthermore, the metadata may include the coefficient set group used for restoration or not. In the latter case, for example, one or more of the subject distance, the subject direction, and the angle of view at the time of restoration, and the installation state of the imaging device or imaging element at the time of imaging are included in the metadata. [0369]
Moreover, for example, the user may transmit the imaging command to each imaging device 321 using a remote controller and the like. [0370]
Furthermore, for example, in a case where each imaging device 321 captures a still image of the subject that does not move, each imaging device 321 may image at an arbitrary timing without using the imaging command, or an input unit 124 of each imaging device 321 may generate the imaging command separately to supply to a control unit 123 of the same device. [0371]

For example, a shape other than the above-described lateral band-type, longitudinal band-type, L-shaped type, and type provided with a rectangular opening may be

adopted as the shape of the light-shielding film 121b of
each pixel 121a.
[0372]
Furthermore, for example, in the imaging element 121 described above with reference to Fig. 5, the example in which four photodiodes 121f of two rows × two columns are provided in one pixel 121a is illustrated, but the number and arrangement of the photodiodes 121f are not limited to this example. [0373]
For example, as illustrated in Fig. 40, in one pixel 121a, nine photodiodes 121f-111 to 121f-119 arranged in three rows × three columns may be provided for one on-chip lens 121c. That is, one pixel output unit may include the nine photodiodes 121f. [0374]
Then, for example, by not reading out the signals of the five pixels of the photodiodes 121f-111, 121f-114, and 121f-117 to 121f-119, an incident angle characteristic similar to that of the pixel 121a including the L-shaped light-shielding film 121b in which the light-shielding film 121b is set in a range of the photodiodes 121f-111, 121f-114, and 121f-117 to 121f-119 may be substantially obtained. [0375]
In this manner, it is possible to obtain the incident angle characteristic similar to that in a case where the light-shielding film 121b is provided without providing the light-shielding film 121b. Furthermore, by switching a pattern of the photodiodes 121f from which no signal is read out, incident angle directivity may be

changed as in a case where the position and range light-shielded by the light-shielding film 121b are changed. [0376]
Furthermore, in the above description, the example in which one pixel 121a forms one pixel output unit is illustrated; however, it is also possible that a plurality of pixels 121a forms one pixel output unit. [0377]
For example, as illustrated in Fig. 41, the pixels 121a-111 to 121a-119 arranged in three rows × three columns may form one pixel output unit 801b. Note that, each of the pixels 121a-111 to 121a-119 includes, for example, one photodiode and does not include the on-chip lens. [0378]
For example, by adding a pixel signal from each pixel 121a, the detection signal of one pixel of a detection image may be generated, and by stopping or not adding an output of pixel signals from some pixels 121a, the incident angle directivity of a pixel output unit 801b may be realized. For example, by adding the pixel signals of the pixels 121a-112, 121a-113, 121a-115, and 121a-116 to generate the detection signal, the incident angle directivity similar to that in a case of providing the L-shaped light-shielding film 121b in a range of the pixels 121a-111, 121a-114, and 121a-117 to 121a-119 may be obtained. [0379]
Furthermore, by switching the pattern of the pixel 121a the pixel signal of which is added to the detection signal, the incident angle directivity may be set to a

different value as in a case where the position and range light-shielding by the light-shielding film 121b are changed. [0380]
Moreover, in this case, for example, it is possible to change the range of the pixel output unit by changing a combination of the pixels 121a. For example, the pixels 121a of two rows × two columns including the pixels 121a-111, 121a-112, 121a-114, and 121a-115 may form a pixel output unit 801s. [0381]
Furthermore, for example, by recording the pixel signals of all the pixels 121a and later setting the combination of the pixels 121a, it is possible to set the range of the pixel output unit later. Moreover, by selecting the pixel 121a the pixel signal of which is added to the detection signal out of the pixels 121a in the set pixel output unit, the incident angle directivity of the pixel output unit may be set later. [0382]
Furthermore, although the example of providing the different incident angle directivities to the respective pixels by using the light-shielding film 121b as a modulation element and changing the combination of the photodiodes contributing to the output is illustrated in Fig. 4, in the present disclosure, it is also possible to provide the incident angle directivity to each pixel by using an optical filter 902 covering a light-receiving surface of an imaging element 901 as the modulation element as illustrated in Fig. 42, for example. [0383]

Specifically, the optical filter 902 is arranged so as to cover an entire surface of a light-receiving surface 901A at a predetermined interval from the light-receiving surface 901A of the imaging element 901. Light from a subject surface 31 is modulated by the optical filter 902 to be incident on the light-receiving surface 901A of the imaging element 901. [0384]
For example, as the optical filter 902, an optical filter 902BW having a black and white lattice pattern illustrated in Fig. 43 may be used. In the optical filter 902BW, a white pattern portion that transmits light and a black pattern portion that blocks light are randomly arranged. A size of each pattern is set independently of a pixel size of the imaging element 901. [0385]
Fig. 44 illustrates a light-receiving sensitivity characteristic of the imaging element 901 to light from a point light source PA and light from a point light source PB on the subject surface 31 in a case where the optical filter 902BW is used. The light from the point light source PA and the light from the point light source PB are modulated by the optical filter 902BW to be incident on the light-receiving surface 901A of the imaging element 901. [0386]
For example, the light-receiving sensitivity characteristic of the imaging element 901 to the light from the point light source PA is as a waveform Sa. That is, since a shadow is generated by the black pattern portion of the optical filter 902BW, a shaded pattern is

generated in an image on the light-receiving surface 901A for the light from the point light source PA. Similarly, the light-receiving sensitivity characteristic of the imaging element 901 to the light from the point light source PB is as a waveform Sb. That is, since a shadow is generated by the black pattern portion of the optical filter 902BW, a shaded pattern is generated in an image on the light-receiving surface 901A for the light from the point light source PB. [0387]
Note that, the light from the point light source PA and the light from the point light source PB have different incident angles with respect to each white pattern portion of the optical filter 902BW, so that there is a shift in appearance of the shaded pattern on the light-receiving surface. Therefore, each pixel of the imaging element 901 has the incident angle directivity to each point light source on the subject surface 31. [0388]
This method is disclosed in detail, for example, in Non-Patent Document 1 described above. [0389]
Note that, an optical filter 902HW in Fig. 45 may be used in place of the black pattern portion of the optical filter 902BW. The optical filter 902HW includes linear polarizing elements 911A and 911B having the same polarizing direction, and a half-wavelength plate 912, and the half-wavelength plate 912 is interposed between the linear polarizing elements 911A and 911B. The half-wavelength plate 912 includes a polarizing portion

indicated by oblique lines in place of the black pattern portion of the optical filter 902BW, and the white pattern portion and the polarizing portion are randomly arranged. [0390]
The linear polarizing element 911A transmits only light in a predetermined polarizing direction out of substantially non-polarized light emitted from the point light source PA. Hereinafter, it is assumed that the linear polarizing element 911A transmits only light the polarizing direction of which is parallel to the drawing. Out of polarized light transmitted through the linear polarizing element 911A, the polarized light transmitted through the polarizing portion of the half-wavelength plate 912 is such that a polarizing surface is rotated and the polarizing direction changes in a direction perpendicular to the drawing. On the other hand, out of the polarized light transmitted through the linear polarizing element 911A, the polarized light transmitted through the white pattern portion of the half-wavelength plate 912 is such that the polarizing direction remains unchanged from the direction parallel to the drawing. Then, the linear polarizing element 911B transmits the polarized light transmitted through the white pattern portion and hardly transmits the polarized light transmitted through the polarizing portion. Accordingly, an amount of polarized light transmitted through the polarizing portion is reduced as compared to the polarized light transmitted through the white pattern portion. Therefore, a shaded pattern substantially similar to that in a case of using the optical filter BW

is generated on the light-receiving surface 901A of the
imaging element 901.
[0391]
Furthermore, as illustrated in A of Fig. 46, an optical interference mask may be used as an optical filter 902LF. The light emitted from the point light source PA and the light emitted from the point light source PB of the subject surface 31 are emitted to the light-receiving surface 901A of the imaging element 901 via the optical filter 902LF. As illustrated in an enlarged view in a lower portion in A of Fig. 46, for example, a light incident surface of the optical filter 902LF includes irregularities of about a wavelength. Furthermore, in the optical filter 902LF, transmission of light having a specific wavelength emitted in the vertical direction is the maximum. When a change in incident angle (inclination with respect to the vertical direction) of the light beams having the specific wavelength emitted from the point light sources PA and PB of the subject surface 31 on the optical filter 902LF increases, an optical path length changes. Here, when the optical path length is an odd multiple of the half wavelength, the light beams weaken each other, and when this is an even multiple of the half wavelength, the light beams strengthen each other. That is, intensities of the transmitted light beams having the specific wavelength emitted from the point light sources PA and PB and transmitted through the optical filter 902LF are modulated according to the incident angle with respect to the optical filter 902LF to be incident on the light-receiving surface 901A of the imaging element 901 as

illustrated in B of Fig. 46. Therefore, the detection signal output from each pixel output unit of the imaging element 901 is a signal obtained by combining the modulated light intensities of the point light sources for each pixel output unit. [0392]
This method is disclosed in detail, for example, in Patent Document 1 described above. [0393]
Note that, in the methods of Patent Document 1 and Non-Patent Document 1, it is not possible to independently set the incident angle directivity in a pixel 121a unit without affecting adjacent pixels as the imaging element 121 using the pixel 121a in Fig. 4 or the pixel 121a in Fig. 5 described above. Therefore, for example, when the pattern of the optical filter 902BW or the pattern of a diffraction grating of the optical filter 902LF is different, the incident angle directivities of at least a plurality of adjacent pixels of the imaging element 901 are different from each other. Furthermore, the pixels 121a located at close positions have incident angle directivities close to each other. [0394]

Furthermore, the present disclosure is also applicable to the imaging device and imaging element that image light of a wavelength other than visible light such as infrared light. In this case, the restored image is not the image in which the user may visually recognize the subject but the image in which the user cannot visually recognize the subject. Also in this case, by

using the present technology, the image quality of the restored image is improved for an image processing device and the like capable of recognizing the subject. Note that, since it is difficult for a normal imaging lens to transmit far-infrared light, the present technology is effective in a case of imaging the far-infrared light, for example. Therefore, the restored image may be a far-infrared light image, and may be a visible light image or a non-visible light image in addition to the far-infrared light image. [0395]
Moreover, for example, by applying machine learning such as deep learning, it is also possible to perform image recognition and the like using the detection image before restoration without using the restored image after the restoration. In this case also, accuracy of image recognition using the detection image before the restoration is improved by using the present technology. In other words, the image quality of the detection image before the restoration is improved. [0396]
<<5. Other >>
The above-described series of processes may be executed by hardware or may be executed by software. In a case where a series of processes is executed by the software, a program which forms the software is installed on a computer. Here, the computer includes a computer (for example, the control unit 123 and the like) incorporated in dedicated hardware, for example. [0397]
The program executed by the computer may be

recorded in a recording medium (for example, the recording medium 130 and the like) as a package medium and the like to be provided, for example. Furthermore, the program may be provided by means of a wired or wireless transmission medium such as a local area network, the Internet, and digital broadcasting. [0398]
Note that, the program executed by the computer may be the program of which processes are performed in chronological order in the order described in this specification or may be the program of which processes are performed in parallel or at required timing such as when a call is issued. [0399]
Furthermore, the embodiment of the present technology is not limited to the above-described embodiments and various modifications may be made without departing from the gist of the present technology. [0400]
For example, the present technology may be configured as cloud computing in which a function is shared by a plurality of devices through the network to process together. [0401]
Moreover, each step described in the above-described flowchart may be executed by one device or executed by a plurality of devices in a shared manner. [0402]
Furthermore, in a case where a plurality of processes is included in one step, a plurality of processes included in one step may be executed by one

device or by a plurality of devices in a shared manner. [0403]
Note that, the present disclosure may also have the following configuration. [0404]
(1)
A signal processing device including:
a restoration unit that restores one restored image by using a plurality of detection signal sets obtained by an imaging element in a plurality of states in which at least one of a position or orientation with respect to a subject is different, the imaging element that includes a plurality of pixel output units that receives incident light from the subject incident without an intervention of an imaging lens or a pinhole and each outputs one detection signal indicating an output pixel value modulated by an incident angle of the incident light, and outputs a detection signal set including a plurality of detection signals output from the plurality of pixel output units.
(2)
The signal processing device according to (1) described above,
in which the plurality of detection signal sets is obtained by a plurality of imaging elements.
(3)
The signal processing device according to (2) described above,
in which the restoration unit restores the one restored image by using the plurality of detection signal sets and a coefficient set group selected on the basis of

a distance between each of the plurality of imaging elements and the subject.
(4)
The signal processing device according to (3) described above,
in which, in a case where the distance between each of the imaging elements and the subject is shorter than a predetermined distance, the restoration unit restores the one restored image by using a coefficient set group selected on the basis of both the distance and a position of each of the imaging elements with respect to the subject, and the plurality of detection signal sets.
(5)
The signal processing device according to (3) described above,
in which the restoration unit restores the one restored image by using a coefficient set group selected on the basis of the distance, the position, and orientation of each of the imaging elements with respect to the subject, and the plurality of detection signal sets.
(6)
The signal processing device according to any one of (3) to (5) described above,
in which the restoration unit restores the one restored image by using one simultaneous equation using the plurality of detection signal sets and the coefficient set group.
(7)
The signal processing device according to any one of (2) to (6) described above, further including:

at least a part of the imaging elements out of the plurality of imaging elements.
(8)
The signal processing device according to (7) described above,
in which the restoration unit restores the one restored image by using a detection signal set obtained by the at least a part of the imaging elements included in the signal processing device, and a detection signal set obtained by an imaging element different from the imaging element included in the signal processing device.
(9)
The signal processing device according to (7) or (8) described above,
in which at least a part of the pixel output units of the at least a part of the imaging elements has incident angle directivity indicating directivity to the incident angle of the incident light from the subject.
(10)
The signal processing device according to (9) described above,
in which incident angle directivities of two or more imaging elements out of the plurality of imaging elements are different from each other.
(11)
The signal processing device according to (9) or (10) described above,
in which incident angle directivities of two or more imaging elements out of the plurality of imaging elements are same.
(12)

The signal processing device according to (1) described above,
in which the plurality of detection signal sets is obtained in each of the states by one imaging element.
(13)
The signal processing device according to (12) described above, further including:
the imaging element.
(14)
The signal processing device according to (13) described above,
in which at least a part of the pixel output units of the imaging element has incident angle directivity indicating directivity to the incident angle of the incident light.
(15)
The signal processing device according to (13) or (14) described above,
in which the plurality of pixel output units of the imaging element has a configuration capable of independently setting incident angle directivity indicating directivity to the incident angle of the incident light.
(16)
An imaging device including:
one or more imaging elements that includes a plurality of pixel output units that receives incident light from a subject incident without an intervention of an imaging lens or a pinhole and each outputs one detection signal indicating an output pixel value modulated by an incident angle of the incident light, and

outputs a detection signal set including a plurality of detection signals output from the plurality of pixel output units; and
an association unit that associates the detection signal set with metadata used for restoring a restored image using the detection signal set.
(17)
The imaging device according to (16) described above,
in which the metadata includes at least one of a distance between the imaging element and the subject, a position of the imaging element with respect to the subject, or orientation of the imaging element with respect to the subject.
(18)
The imaging device according to (16) or (17) described above,
in which the association unit associates a plurality of detection signal sets obtained by a plurality of imaging elements including the one or more imaging elements and the metadata used for restoring the restored image using the plurality of detection signal sets.
(19)
The imaging device according to any one of (16) to
(18) described above,
in which the plurality of imaging elements includes an imaging element provided in another imaging device. (20) The imaging device according to any one of (16) to
(19) described above,

in which at least a part of the pixel output units of the plurality of imaging elements has incident angle directivity indicating directivity to the incident angle of the incident light from the subject.
(21)
The imaging device according to any one of (16) to (20), further including:
a communication unit that transmits the detection signal set.
(22)
The imaging device according to (21) described above,
in which the communication unit transmits the metadata associated with the detection signal set.
(23)
The imaging device according to any one of (16) to (22) described above,
in which the plurality of pixel output units has a configuration capable of independently setting incident angle directivity indicating directivity to the incident angle of the incident light from the subject. [0405]
Note that, the effects described in this specification are illustrative only and are not limitative; there may also be another effect.
REFERENCE SIGNS LIST
[0406]
101 Imaging device
111 Signal processing control unit
121 Imaging element

121a, 121a' Pixel
121aA Pixel for restoration
121aB Pixel for exposure
121A Light-receiving surface
121b Light-shielding film
121c On-chip lens
121e, 121f Photodiode
122 Restoration unit
123 Control unit

125 Detection unit
126 Association unit 301 Imaging system

311 Imaging device group
312 Signal processing device 321-1 to 321-n Imaging device 401 Signal processing control unit 411 Control unit
413 Association unit
501 Restoration unit
502 Control unit 504 Association unit 601 Imaging system
611 Imaging device group
621 Imaging device
651 Imaging system
661 Imaging device group
671 Imaging device
701 Imaging device
711 Imaging unit
801b, 801s Pixel output unit
901 Imaging element

901A Light-receiving surface 902, 902BW, 902F Optical filter

CLAIMS
1. A signal processing device comprising:
a restoration unit that restores one restored image by using a plurality of detection signal sets obtained by an imaging element in a plurality of states in which at least one of a position or orientation with respect to a subject is different, the imaging element that includes a plurality of pixel output units that receives incident light from the subject incident without an intervention of an imaging lens or a pinhole and each outputs one detection signal indicating an output pixel value modulated by an incident angle of the incident light, and outputs a detection signal set including a plurality of detection signals output from the plurality of pixel output units.
2. The signal processing device according to claim 1,
wherein the plurality of detection signal sets is
obtained by a plurality of imaging elements.
3. The signal processing device according to claim 2,
wherein the restoration unit restores the one
restored image by using the plurality of detection signal sets and a coefficient set group selected on a basis of a distance between each of the plurality of imaging elements and the subject.
4. The signal processing device according to claim 3,
wherein, in a case where the distance between each
of the imaging elements and the subject is shorter than a

predetermined distance, the restoration unit restores the one restored image by using a coefficient set group selected on a basis of both the distance and a position of each of the imaging elements with respect to the subject, and the plurality of detection signal sets.
5. The signal processing device according to claim 3,
wherein the restoration unit restores the one
restored image by using a coefficient set group selected on a basis of the distance, the position, and orientation of each of the imaging elements with respect to the subject, and the plurality of detection signal sets.
6. The signal processing device according to claim 3,
wherein the restoration unit restores the one
restored image by using one simultaneous equation using the plurality of detection signal sets and the coefficient set group.
7. The signal processing device according to claim 2,
further comprising:
at least a part of the imaging elements out of the plurality of imaging elements.
8. The signal processing device according to claim 7,
wherein the restoration unit restores the one
restored image by using a detection signal set obtained by the at least a part of the imaging elements included in the signal processing device, and a detection signal set obtained by an imaging element different from the imaging element included in the signal processing device.

9. The signal processing device according to claim 7,
wherein at least a part of the pixel output units
of the at least a part of the imaging elements has incident angle directivity indicating directivity to the incident angle of the incident light from the subject.
10. The signal processing device according to claim 9,
wherein incident angle directivities of two or more
imaging elements out of the plurality of imaging elements are different from each other.
11. The signal processing device according to claim 9,
wherein incident angle directivities of two or more
imaging elements out of the plurality of imaging elements are same.
12. The signal processing device according to claim 1,
wherein the plurality of detection signal sets is
obtained in each of the states by one imaging element.
13. The signal processing device according to claim 12,
further comprising:
the imaging element.
14. The signal processing device according to claim 13,
wherein at least a part of the pixel output units
of the imaging element has incident angle directivity indicating directivity to the incident angle of the incident light.

15. The signal processing device according to claim 13,
wherein the plurality of pixel output units of the
imaging element has a configuration capable of independently setting incident angle directivity indicating directivity to the incident angle of the incident light.
16. An imaging device comprising:
one or more imaging elements that includes a plurality of pixel output units that receives incident light from a subject incident without an intervention of an imaging lens or a pinhole and each outputs one detection signal indicating an output pixel value modulated by an incident angle of the incident light, and outputs a detection signal set including a plurality of detection signals output from the plurality of pixel output units; and
an association unit that associates the detection signal set with metadata used for restoring a restored image using the detection signal set.
17. The imaging device according to claim 16,
wherein the metadata includes at least one of a
distance between the imaging element and the subject, a position of the imaging element with respect to the subject, orientation of the imaging element with respect to the subject, or a coefficient set group.
18. The imaging device according to claim 16,
wherein the association unit associates a plurality
of detection signal sets obtained by a plurality of

imaging elements including the one or more imaging elements and the metadata used for restoring the restored image using the plurality of detection signal sets.
19. The imaging device according to claim 18,
wherein the plurality of imaging elements includes
an imaging element provided in another imaging device.
20. The imaging device according to claim 18,
wherein at least a part of the pixel output units
of the plurality of imaging elements has incident angle directivity indicating directivity to the incident angle of the incident light from the subject.
21. The imaging device according to claim 16, further
comprising:
a communication unit that transmits the detection signal set.
22. The imaging device according to claim 21,
wherein the communication unit transmits the
metadata associated with the detection signal set.
23. The imaging device according to claim 16,
wherein the plurality of pixel output units has a
configuration capable of independently setting incident angle directivity indicating directivity to the incident angle of the incident light from the subject.