Title of the invention: Image processing device, image processing method, and non-transitory computer-readable medium storing image processing program
Technical field
[0001]
The present invention relates to an image processing device, an image processing method, and a non-transitory computer-readable medium storing an image processing program.
Background technology
[0002]
In recent years, in surveillance systems, etc., technology has been used to detect and search for the state of a person's posture, behavior, etc. from surveillance camera images. As a related technology, for example, Patent Documents 1 and 2 are known. Patent Literature 1 discloses a technique of estimating the posture of a person from a captured image of the person and searching for an image including a posture similar to the estimated posture. Patent Literature 2 discloses a technique of detecting a state of a person from an image of the person and estimating the height of the person based on the detected state. In addition, Non-Patent Document 1 is known as a technique related to human skeleton estimation.
Prior art literature
Patent documents
[0003]
Patent Document 1: Japanese Patent Application Laid-Open No. 2019-091138
Patent Document 2: International Publication No. 2019/064375
Non-patent literature
[0004]
Non-Patent Document 1: Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, P. 7291-7299
SUMMARY OF THE INVENTION
Problems to be solved by the invention
[0005]
In the related technology such as the above-mentioned Patent Document 1, in order to search for similar postures, a feature amount indicating the characteristics of a person's posture is used. May be less robust to lookup from perspective. For this reason, the related technology has a problem of low robustness of state recognition processing such as retrieval and classification of a person's state.
[0006]
In view of such problems, the present disclosure provides an image processing device, an image processing method, and a non-transitory computer-readable medium storing an image processing program capable of improving the robustness of human state recognition processing. intended to
Means to solve problems
[0007]
An image processing apparatus according to the present disclosure includes skeleton detection means for detecting a two-dimensional skeleton structure of a person based on an acquired two-dimensional image; The apparatus includes an estimating means for estimating an upright height of a person, and a normalizing means for normalizing the detected two-dimensional skeletal structure based on the estimated upright height.
[0008]
An image processing method according to the present disclosure detects a 2D skeletal structure of a person based on an acquired 2D image, and based on the detected 2D skeletal structure, when the person is standing upright in a 2D image space. and normalize the detected 2D skeletal structure based on the estimated standing height.
[0009]
A non-transitory computer-readable medium storing an image processing program according to the present disclosure detects a two-dimensional skeletal structure of a person based on an acquired two-dimensional image, and based on the detected two-dimensional skeletal structure for causing a computer to execute a process of estimating an upright height of the person in a two-dimensional image space and normalizing the detected two-dimensional skeletal structure based on the estimated upright height; A non-transitory computer-readable medium storing an image processing program.
Effect of the invention
[0010]
According to the present disclosure, it is possible to provide a non-temporary computer-readable medium storing an image processing device, an image processing method, and an image processing program capable of improving the robustness of human state recognition processing.
Brief description of the drawing
[0011]
1 is a configuration diagram showing an outline of an image processing device according to an embodiment; FIG.
2 is a configuration diagram showing the configuration of an image processing apparatus according to Embodiment 1; FIG.
3 is a configuration diagram showing another configuration of the image processing apparatus according to Embodiment 1; FIG.
4 is a flow chart showing an image processing method according to Embodiment 1. FIG.
5 is a flow chart showing a specific example 1 of a height pixel number calculation method according to Embodiment 1. FIG.
6 is a flowchart showing a specific example 2 of the height pixel number calculation method according to the first embodiment; FIG.
7 is a flow chart showing a specific example 2 of the height pixel number calculation method according to the embodiment 1. FIG.
8 is a flowchart showing a normalization method according to Embodiment 1; FIG.
9 is a diagram showing an example of detection of a skeletal structure according to Embodiment 1. FIG.
10 is a diagram showing a human body model according to Embodiment 1. FIG.
11 is a diagram showing a detection example of a skeletal structure according to Embodiment 1. FIG.
12 is a diagram showing an example of detection of a skeletal structure according to Embodiment 1. FIG.
13 is a diagram showing a detection example of a skeletal structure according to Embodiment 1. FIG.
14 is a diagram showing a human body model according to Embodiment 1. FIG.
15 is a diagram showing a detection example of a skeletal structure according to Embodiment 1. FIG.
16 is a histogram for explaining the height pixel number calculation method according to Embodiment 1. FIG.
17 is a diagram showing a detection example of a skeletal structure according to Embodiment 1. FIG.
18 is a diagram showing a three-dimensional human body model according to Embodiment 1. FIG.
19 is a diagram for explaining a height pixel number calculation method according to Embodiment 1. FIG.
20 is a diagram for explaining a height pixel number calculation method according to Embodiment 1. FIG.
21 is a diagram for explaining a height pixel number calculation method according to Embodiment 1. FIG.
22 is a diagram for explaining a normalization method according to Embodiment 1; FIG.
23 is a diagram for explaining a normalization method according to Embodiment 1; FIG.
24 is a diagram for explaining a normalization method according to Embodiment 1; FIG.
25 is a diagram for explaining the effect of the image processing method according to Embodiment 1; FIG.
26 is a diagram for explaining the effect of the image processing method according to Embodiment 1; FIG.
27 is a diagram for explaining the effect of the image processing method according to Embodiment 1; FIG.
28 is a diagram for explaining the effect of the image processing method according to Embodiment 1; FIG.
29 is a configuration diagram showing an outline of hardware of a computer according to the embodiment; FIG.
Mode for carrying out the invention
[0012]
Embodiments will be described below with reference to the drawings. In each drawing, the same elements are denoted by the same reference numerals, and redundant description will be omitted as necessary.
[0013]
(Examination leading to the embodiment)
In recent years, image recognition technology that utilizes machine learning such as deep learning has been applied to various systems. For example, it is being applied to surveillance systems that monitor images from surveillance cameras. By utilizing machine learning in monitoring systems, it is becoming possible to grasp the state of a person's posture, behavior, etc. from images to some extent.
[0014]
However, with such related technologies, advance preparation by machine learning is a premise, so there are cases where it is not always possible to grasp the state of the person desired by the user on demand. That is, in the related technology, it is necessary to learn a large amount of images in which the state of a person is defined in advance (such as an image of a person sitting down or an image of a person raising their hand). Therefore, if a person's state that cannot be defined in advance is desired to be monitored, it is difficult for machine learning to handle it.
[0015]
In addition, related technologies do not consider searches from various perspectives. For example, if the posture is the same, the user may desire to determine that the posture is the same regardless of the orientation, or to suppress the effect of the angle of view as much as possible. There is also a method of using deep learning to convert posture information into features that are robust to the angle of view and the direction of the person, but this method requires a large amount of learning data as described above, and unknown (learning data There is a problem that it is weak in posture. In addition, for example, when a part of a person's body is hidden, retrieval cannot be performed, and flexible retrieval such as partial matching cannot be supported.
[0016]
Therefore, the inventors studied a method of using skeleton estimation technology such as Non-Patent Document 1 in order to flexibly recognize a person's state without advance preparation such as machine learning of the person's state. Related skeleton estimation techniques, such as OpenPose disclosed in Non-Patent Document 1, estimate a person's skeleton by learning various patterns of correct-correct image data. In the following embodiments, by utilizing such skeleton estimation technology, state recognition processing with high robustness against the effects of the orientation of a person, the angle of view, and the like is made possible.
[0017]
The skeletal structure estimated by skeletal estimation techniques such as OpenPose consists of "keypoints", which are characteristic points such as joints, and "bones (bone links)", which indicate links between keypoints. . Therefore, in the following embodiments, the skeletal structure will be described using the terms "keypoint" and "bone". "Bone" corresponds to the "bones" of a person.
[0018]
(Overview of Embodiment)
FIG. 1 shows an overview of an image processing device 10 according to an embodiment. As shown in FIG. 1, the image processing device 10 includes a skeleton detection unit 11, an estimation unit 12, and a normalization unit 13. The skeleton detection unit 11 detects a two-dimensional skeleton structure of a person based on a two-dimensional image acquired from a camera or the like. Based on the two-dimensional skeleton structure detected by the skeleton detection unit 11, the estimation unit 12 estimates the height of the person in the two-dimensional image space when the person stands upright. The normalization unit 13 normalizes the two-dimensional skeleton structure detected by the skeleton detection unit 11 based on the standing height estimated by the estimation unit 12 .
[0019]
Thus, in the embodiment, the 2D skeletal structure of a person is detected from a 2D image, and the 2D skeletal structure is normalized based on the height of the person when standing upright estimated from the 2D skeletal structure. As a result, in state recognition processing such as retrieval using this normalized normalized value (feature amount), it is possible to improve robustness against the orientation of a person.
[0020]
(Embodiment 1)
Embodiment 1 will be described below with reference to the drawings. FIG. 2 shows the configuration of the image processing apparatus 100 according to this embodiment. The image processing apparatus 100 constitutes an image processing system 1 together with a camera 200 and a database (DB) 110 . An image processing system 1 including an image processing apparatus 100 is a system for normalizing a human skeleton structure estimated from an image. Also, using the normalized skeletal structure feature amount, it is possible to classify, search, etc. the state of a person.
[0021]
The camera 200 is an imaging unit such as a surveillance camera that generates a two-dimensional image. The camera 200 is installed at a predetermined location and captures an image of a person or the like in an imaging area from the installation location. The camera 200 is directly connected or connected via a network or the like so as to be able to output captured images (video) to the image processing apparatus 100 . Note that the camera 200 may be provided inside the image processing apparatus 100 .
[0022]
The database 110 is a database that stores information (data) necessary for processing of the image processing apparatus 100, processing results, and the like. The database 110 stores images acquired by the image acquisition unit 101, detection results of the skeletal structure detection unit 102, data for machine learning, feature amounts normalized by the normalization unit 104, and the like. The database 110 is directly connected to the image processing apparatus 100 so that data can be input/output as needed, or connected via a network or the like. Note that the database 110 may be provided inside the image processing apparatus 100 as a nonvolatile memory such as a flash memory, a hard disk device, or the like.
[0023]
As shown in FIG. 2, the image processing device 100 includes an image acquisition unit 101, a skeletal structure detection unit 102, a height calculation unit 10 3. A normalization unit 104 is provided. The configuration of each unit (block) is an example, and may be configured by other units as long as the method (operation) described later is possible. Further, the image processing apparatus 100 is realized by a computer device such as a personal computer or a server that executes a program, for example. good. Note that an input unit for inputting an operation from the user and a display unit for displaying processing results and the like may be provided as necessary.
[0024]
The image acquisition unit 101 acquires a two-dimensional image including a person captured by the camera 200 . The image acquisition unit 101 acquires, for example, an image including a person (video including a plurality of images) captured by the camera 200 during a predetermined monitoring period. It should be noted that an image including a person prepared in advance may be acquired from the database 110 or the like instead of being acquired from the camera 200 .
[0025]
The skeletal structure detection unit 102 detects the 2D skeletal structure of the person in the image based on the acquired 2D image. The skeletal structure detection unit 102 detects skeletal structures of all persons recognized in the acquired image. The skeletal structure detection unit 102 detects the skeletal structure of a person based on recognized features such as the joints of the person, using a skeletal structure estimation technique using machine learning. The skeleton structure detection unit 102 uses, for example, a skeleton estimation technique such as OpenPose described in Non-Patent Document 1.
[0026]
A height calculation unit (height estimation unit) 103 calculates (estimates) the height of a person in a two-dimensional image when the person stands upright (height in pixels) based on the detected two-dimensional skeletal structure. It can also be said that the number of height pixels is the height of the person in the two-dimensional image (the length of the whole body of the person in the two-dimensional image space). The height calculation unit 103 obtains the number of height pixels (the number of pixels) from the length of each bone of the detected skeletal structure (the length in the two-dimensional image space).
[0027]
In the following examples, specific examples 1 to 3 are used as the method for obtaining the height pixel count. Any one of the methods of Examples 1 to 3 may be used, or a plurality of arbitrarily selected methods may be used in combination. In specific example 1, the number of height pixels is obtained by totaling the length of the bones from the head to the feet among the bones of the skeletal structure. If the skeletal structure detection unit 102 (skeletal structure estimation technology) does not output the top of the head and the feet, it can be corrected by multiplying by a constant as necessary. In specific example 2, the number of height pixels is calculated using a human body model that indicates the relationship between the length of each bone and the length of the whole body (height in a two-dimensional image space). In specific example 3, the number of height pixels is calculated by fitting a three-dimensional human body model to a two-dimensional skeletal structure.
[0028]
The normalization unit 104 normalizes the skeletal structure (skeletal information) of the person based on the calculated number of pixels of the person's height. In this example, the normalization unit 104 normalizes the height on the image of each key point (feature point) included in the skeletal structure by the number of height pixels. The normalization unit 104 stores the normalized skeletal structure feature amount (normalized value) in the database 110 . For example, the height direction (vertical direction or vertical direction) is the vertical direction (Y-axis direction) in the two-dimensional coordinate (XY coordinate) space of the image. In this case, the height of the keypoint can be obtained from the Y coordinate value (the number of pixels) of the keypoint.
Alternatively, the height direction may be the direction of the vertical projection axis (vertical projection direction) obtained by projecting the direction of the vertical axis perpendicular to the ground (reference plane) in the three-dimensional coordinate space of the real world onto the two-dimensional coordinate space. In this case, the height of the keypoint is obtained by calculating the vertical projection axis by projecting the axis perpendicular to the ground in the real world onto the two-dimensional coordinate space based on the camera parameters, and calculating the value along this vertical projection axis (the number of pixels ) can be obtained from Note that the camera parameters are imaging parameters of an image. For example, the camera parameters are the attitude, position, imaging angle, focal length, etc. of the camera 200 . With the camera 200, an object whose length and position are known in advance can be imaged, and camera parameters can be obtained from the image. Distortion occurs at both ends of the captured image, and the vertical direction of the real world may not match the vertical direction of the image. On the other hand, by using the parameters of the camera that captured the image, it is possible to know how much the vertical direction in the real world is tilted in the image. Therefore, by normalizing the values ​​of the keypoints along the vertical projection axis projected into the image based on the camera parameters by the height, it is possible to convert the keypoints into features considering the deviation between the real world and the image. can. The left-right direction (horizontal direction) is the left-right direction (X-axis direction) in the two-dimensional coordinate (XY coordinate) space of the image, or the direction parallel to the ground in the three-dimensional coordinate space of the real world. is projected onto the two-dimensional coordinate space.
[0029]
Note that the image processing apparatus 100 may further include either or both of the classification unit 105 and the search unit 106, as shown in FIG. Both or one of the classification unit 105 and the retrieval unit 106 is a recognition unit that performs recognition processing of the person's state. The classification unit 105 classifies (clusters) a plurality of skeletal structures stored in the database 110 based on the similarity of the feature amounts of the skeletal structures. It can be said that the classification unit 105 classifies the states of a plurality of persons based on the feature amount of the skeletal structure as the process of recognizing the states of the persons. Further, the search unit 106 searches a plurality of skeleton structures stored in the database 110 for a skeleton structure having a high degree of similarity to the feature quantity of the search query (query state). It can be said that the search unit 106 searches for a person's state corresponding to a search condition (query state) from among a plurality of persons' states based on the feature amount of the skeletal structure as recognition processing of the person's state.
[0030]
The similarity is the distance between the features of the skeleton structure. The classification unit 105 and the search unit 106 may classify and search based on the similarity of the feature amount of the entire skeletal structure, or may classify and search based on the similarity of the feature amount of a part of the skeletal structure. Also, the posture of a person may be classified and retrieved based on the feature amount of the skeletal structure of the person in each image, or the pose of the person may be classified and retrieved based on the feature amount of the skeletal structure of the person in a plurality of temporally consecutive images. actions may be classified and retrieved. In other words, the classification unit 105 and the search unit 106 can classify and search the state of the person, including the posture and behavior of the person, based on the feature amount of the skeletal structure.
[0031]
4 to 8 show the operation of the image processing apparatus 100 according to this embodiment. FIG. 4 shows the flow from image acquisition to output of normalization information in the image processing apparatus 100, and FIGS. , and FIG. 8 shows the flow of the normalization process (S104) in FIG.
[0032]
As shown in FIG. 4, the image processing device 100 acquires an image from the camera 200 (S101). The image acquisition unit 101 acquires an image of a person in order to perform normalization processing from the skeletal structure, and stores the acquired image in the database 110 . The image acquisition unit 101 acquires, for example, a plurality of images captured during a predetermined monitoring period, and performs subsequent processing on all persons included in the plurality of images.
[0033]
Subsequently, the image processing apparatus 100 detects the skeletal structure of the person based on the acquired image of the person (S102). FIG. 9 shows an example of skeletal structure detection. As shown in FIG. 9, an image acquired from a monitoring camera or the like includes a plurality of persons, and the skeletal structure of each person included in the image is detected.
[0034]
FIG. 10 shows the skeletal structure of the human body model 300 detected at this time, and FIGS. 11 to 13 show detection examples of the skeletal structure. A skeleton structure detection unit 102 detects the skeleton structure of a human body model (two-dimensional skeleton model) 300 as shown in FIG. 10 from a two-dimensional image using a skeleton estimation technique such as OpenPose. The human body model 300 is a two-dimensional model composed of key points such as human joints and bones connecting the key points. In this example, as shown in FIG. 10, the height of the skeletal structure of an upright person in the image is the number of height pixels (h), and the height of each keypoint of the skeletal structure in the state of the person in the image is the keypoint. Let the height be (yi).
[0035]
For example, the skeletal structure detection unit 102 extracts feature points that can be keypoints from the image, refers to information obtained by machine learning the image of the keypoints, and detects each keypoint of the person. In the example of FIG. 10, the key points of the person are head A1, neck A2, right shoulder A31, left shoulder A32, right elbow A41, left elbow A42, right hand A51, left hand A52, right hip A61, left hip A62, right knee A71. , left knee A72, right foot A81, and left foot A82. Further, as human bones connecting these key points, bone B1 connecting head A1 and neck A2, bone B21 and bone B22 connecting neck A2 and right shoulder A31 and left shoulder A32, right shoulder A31 and left shoulder A32 and right shoulder A31 and right shoulder A32, respectively. Bone B31 and B32 connecting elbow A41 and left elbow A42 respectively, bone B41 and bone B42 connecting right elbow A41 and left elbow A42 with right hand A51 and left hand A52 respectively, neck A2 connecting right hip A61 and left hip A62 respectively Bone B51 and B52, Bone B61 and B62 connecting right hip A61 and left hip A62 and right knee A71 and left knee A72, respectively, bone B71 and bone connecting right knee A71 and left knee A72, right leg A81 and left leg A82, respectively B72 is detected.
[0036]
FIG. 11 is an example of detecting a person standing upright. In FIG. 11, an upright person is imaged from the front, and bones B1, B51 and B52, B61 and B62, and B71 and B72 viewed from the front are detected without overlapping each other. The bones B61 and B71 are slightly more bent than the left leg bones B62 and B72.
[0037]
Fig. 12 is an example of detecting a person who is crouching. In FIG. 12, a crouching person is imaged from the right side, bone B1, bone B51 and bone B52, bone B61 and bone B62, bone B71 and bone B72 seen from the right side are detected, and bone B61 of the right leg is detected. And the bone B71 and the bones B62 and B72 of the left leg are greatly bent and overlapped.
[0038]
FIG. 13 is an example of detecting a sleeping person. In FIG. 13, a person lying down is imaged obliquely from the front left, bones B1, B51 and B52, bones B61 and B62, bones B71 and B72 are detected from the oblique front left, and bones B71 and B72 are detected. The bones B61 and B71 of the left leg and the bones B62 and B72 of the left leg are bent and overlapped.
[0039]
Subsequently, as shown in FIG. 4, the image processing apparatus 100 performs height pixel number calculation processing based on the detected skeletal structure (S103). Specific examples 1 to 3 of the height pixel number calculation process will be described below.
[0040]

In Specific Example 1, the length of the bone from the head to the foot is used to determine the number of pixels in the height. In Specific Example 1, as shown in FIG. 5, the height calculator 103 acquires the length of each bone (S111), and totals the acquired lengths of each bone (S112).
[0041]
The height calculation unit 103 acquires the length of the bones on the two-dimensional image from the head to the feet of the person, and obtains the number of pixels of the height. 10, bone B1 (length L1), bone B51 (length L21), bone B61 (length L31) and bone B71 (length L41), or , bone B1 (length L1), bone B52 (length L22), bone B62 (length L32), and bone B72 (length L42). The length of each bone can be obtained from the coordinates of each keypoint in the two-dimensional image. The height pixel number (h) is calculated by multiplying L1+L21+L31+L41 or L1+L22+L32+L42 by a correction constant. If both values ​​can be calculated, e.g. long The value of the one is taken as the number of height pixels. That is, each bone has the longest length in the image when it is imaged from the front, and it is displayed short when it is tilted in the depth direction with respect to the camera. Therefore, it is more likely that the longer bone is imaged from the front, which is considered to be closer to the true value. Therefore, it is preferable to select the longer value.
[0042]
In the example of FIG. 11, bone B1, bone B51 and bone B52, bone B61 and bone B62, bone B71 and bone B72 are detected without overlapping. The total of these bones, L1 + L21 + L31 + L41, and L1 + L22 + L32 + L42, is obtained, and for example, the value obtained by multiplying L1 + L22 + L32 + L42 on the left foot side where the detected bone length is long by a correction constant is taken as the height pixel number.
[0043]
In the example of FIG. 12, bone B1, bone B51 and bone B52, bone B61 and bone B62, bone B71 and bone B72 are detected, respectively, and the right foot bone B61 and bone B71 and the left foot bone B62 and bone B72 overlap. .. The total of these bones, L1 + L21 + L31 + L41 and L1 + L22 + L32 + L42, is obtained, and for example, the value obtained by multiplying L1 + L21 + L31 + L41 on the right foot side where the detected bone length is long by a correction constant is taken as the height pixel number.
[0044]
In the example of FIG. 13, bone B1, bone B51 and bone B52, bone B61 and bone B62, bone B71 and bone B72 are detected, respectively, and the right foot bone B61 and bone B71 and the left foot bone B62 and bone B72 overlap. .. The total of these bones, L1 + L21 + L31 + L41, and L1 + L22 + L32 + L42, is obtained, and for example, the value obtained by multiplying L1 + L22 + L32 + L42 on the left foot side where the detected bone length is long by a correction constant is taken as the height pixel number.
[0045]
In Specific Example 1, the height can be calculated by summing the lengths of the bones from the head to the feet, so the number of height pixels can be calculated by a simple method. In addition, since it is only necessary to detect the skeleton from the head to the feet by skeleton estimation technology using machine learning, the number of height pixels can be accurately calculated even when the entire person is not always shown in the image, such as when crouching down. Can be estimated.
[0046]

In Specific Example 2, the number of height pixels is obtained using a two-dimensional skeleton model showing the relationship between the length of the bone included in the two-dimensional skeleton structure and the length of the whole body of the person in the two-dimensional image space.
[0047]
FIG. 14 is a human body model (two-dimensional skeleton model) 301 showing the relationship between the length of each bone in the two-dimensional image space and the length of the whole body in the two-dimensional image space used in the second embodiment. As shown in FIG. 14, the relationship between the length of each bone of an average person and the length of the whole body (the ratio of the length of each bone to the length of the whole body) is associated with each bone of the human body model 301. For example, the length of the head bone B1 is the length of the whole body x 0.2 (20%), the length of the bone B41 of the right hand is the length of the whole body x 0.15 (15%), and the length of the right foot. The length of the bone B71 is the length of the whole body × 0.25 (25%). By storing the information of the human body model 301 in the database 110, the average whole body length can be obtained from the length of each bone. In addition to the human body model of an average person, a human body model may be prepared for each attribute of the person such as age, gender, and nationality. As a result, the length (height) of the whole body can be appropriately obtained according to the attributes of the person.
[0048]
In Specific Example 2, as shown in FIG. 6, the height calculation unit 103 acquires the length of each bone (S121). The height calculation unit 103 acquires the lengths (lengths in the two-dimensional image space) of all the bones in the detected skeleton structure. FIG. 15 is an example in which a person in a crouched state is imaged from diagonally right behind and the skeletal structure is detected. In this example, since the face and left side of the person are not shown, the bones of the head and the bones of the left arm and the left hand cannot be detected. Therefore, the lengths of the detected bones B21, B22, B31, B41, B51, B52, B61, B62, B71, and B72 are acquired.
[0049]
Subsequently, as shown in FIG. 6, the height calculation unit 103 calculates the number of height pixels from the length of each bone based on the human body model (S122). The height calculation unit 103 refers to the human body model 301 showing the relationship between each bone and the length of the whole body as shown in FIG. 14, and obtains the number of height pixels from the length of each bone. For example, since the length of the bone B41 on the right hand is the length of the whole body × 0.15, the number of height pixels based on the bone B41 is obtained by the length of the bone B41 / 0.15. Further, since the length of the bone B71 of the right foot is the length of the whole body × 0.25, the number of height pixels based on the bone B71 is obtained from the length of the bone B71 / 0.25.
[0050]
The human body model referred to at this time is, for example, a human body model of an average person, but a human body model may be selected according to the attributes of the person such as age, gender, and nationality. For example, when a person's face is shown in the captured image, the attribute of the person is identified based on the face, and the human body model corresponding to the identified attribute is referred to. It is possible to recognize a person's attributes from the facial features of the image by referring to the information obtained by machine learning the face for each attribute. Further, when the attribute of the person cannot be identified from the image, the human body model of the average person may be used.
[0051]
Further, the number of height pixels calculated from the length of the bone may be corrected by the camera parameter. For example, when the camera is taken at a high position and looking down at a person, the horizontal length of the shoulder-width bones, etc. is not affected by the depression angle of the camera in the two-dimensional skeletal structure, but the vertical length of the neck-waist bones, etc. The length decreases as the depression angle of the camera increases. Then, the number of height pixels calculated from the horizontal length of the shoulder-width bones and the like tends to be larger than the actual number. Therefore, by utilizing the camera parameters, it is possible to know the angle at which the person is looking down at the camera, and the information on the depression angle can be used to correct the two-dimensional skeleton structure as if it was taken from the front. As a result, the number of height pixels can be calculated more accurately.
[0052]
Subsequently, the height calculation unit 103 calculates the optimum value of the number of height pixels as shown in FIG. 6 (S123). The height calculation unit 103 calculates the optimum value of the number of height pixels from the number of height pixels obtained for each bone. For example, as shown in FIG. 16, a histogram of the number of height pixels obtained for each bone is generated, and a large number of height pixels is selected from the histogram. That is, the number of height pixels longer than the others is selected from the plurality of height pixels obtained based on the plurality of bones. For example, the top 30% is set as a valid value, and in FIG. 16, the number of height pixels by bones B71, B61, and B51 is selected. The average number of selected height pixels may be obtained as the optimum value, or the largest number of height pixels may be used as the optimum value. Since the height is calculated from the length of the bone in the two-dimensional image, the length of the bone is imaged from the front when the bone is not formed from the front, that is, when the bone is tilted in the depth direction when viewed from the camera. It will be shorter than the case. Then, a value having a large number of height pixels is more likely to be imaged from the front than a value having a small number of height pixels, and is a more plausible value. Therefore, a larger value is set as the optimum value.
[0053]
In Specific Example 2, the number of height pixels is calculated based on the detected bones of the skeleton structure using a human body model showing the relationship between the bones in the two-dimensional image space and the length of the whole body, so that all the skeletons from the head to the feet are obtained. Even if is not obtained, the number of height pixels can be obtained from some bones. In particular, the number of height pixels can be estimated accurately by adopting a larger value among the values ​​obtained from a plurality of bones.
[0054]

In Specific Example 3, the two-dimensional skeletal structure is fitted to the three-dimensional human body model (three-dimensional skeletal model), and the skeletal vector of the whole body is obtained using the number of height pixels of the fitted three-dimensional human body model.
[0055]
In Specific Example 3, as shown in FIG. 7, the height calculation unit 103 first calculates the camera parameters based on the image captured by the camera 200 (S131). The height calculation unit 103 extracts an object whose length is known in advance from a plurality of images captured by the camera 200, and obtains a camera parameter from the size (number of pixels) of the extracted object. The camera parameters may be obtained in advance, and the obtained camera parameters may be acquired as needed.
[0056]
Subsequently, the height calculation unit 103 adjusts the arrangement and height of the three-dimensional human body model (S132). The height calculation unit 103 prepares a three-dimensional human body model for calculating the number of height pixels for the detected two-dimensional skeleton structure, and arranges the three-dimensional human body model in the same two-dimensional image based on the camera parameters. Specifically, the "relative positional relationship between the camera and the person in the real world" is specified from the camera parameters and the two-dimensional skeleton structure. For example, assuming that the position of the camera is the coordinates (0, 0, 0), the coordinates (x, y, z) of the position where the person is standing (or sitting) are specified. Then, by assuming an image when the three-dimensional human body model is placed at the same position (x, y, z) as the specified person and captured, the two-dimensional skeleton structure and the three-dimensional human body model are superimposed.
[0057]
FIG. 17 is an example in which a crouching person is imaged diagonally from the front left and the two-dimensional skeleton structure 401 is detected. The two-dimensional skeleton structure 401 has two-dimensional coordinate information. It is preferable that all bones are detected, but some bones may not be detected. For this two-dimensional skeleton structure 401, a three-dimensional human body model 402 as shown in FIG. 18 is prepared. The three-dimensional human body model (three-dimensional skeleton model) 402 is a model of a skeleton having three-dimensional coordinate information and having the same shape as the two-dimensional skeleton structure 401. Then, as shown in FIG. 19, the prepared three-dimensional human body model 402 is arranged and superimposed on the detected two-dimensional skeleton structure 401. In addition, the height of the three-dimensional human body model 402 is adjusted so as to match the two-dimensional skeleton structure 401.
[0058]
The three-dimensional human body model 402 prepared at this time may be a model in a state close to the posture of the two-dimensional skeleton structure 401 as shown in FIG. 19, or may be a model in an upright state. For example, a three-dimensional human body model 402 of the estimated posture may be generated by using a technique of estimating the posture of the three-dimensional space from the two-dimensional image using machine learning. By learning the information of the joints in the two-dimensional image and the joints in the three-dimensional space, the three-dimensional posture can be estimated from the two-dimensional image.
[0059]
Subsequently, the height calculation unit 103 fits the three-dimensional human body model into the two-dimensional skeletal structure as shown in FIG. 7 (S133). As shown in FIG. 20, the height calculation unit 103 superimposes the three-dimensional human body model 402 on the two-dimensional skeletal structure 401 so that the postures of the three-dimensional human body model 402 and the two-dimensional skeletal structure 401 match. The dimensional human body model 402 is transformed. That is, the height, body orientation, and joint angle of the three-dimensional human body model 402 are adjusted and optimized so that there is no difference from the two-dimensional skeletal structure 401. For example, the joints of the three-dimensional human body model 402 are rotated within the movable range of the person, the entire three-dimensional human body model 402 is rotated, and the overall size is adjusted. The fitting of the three-dimensional human body model and the two-dimensional skeleton structure is performed in the two-dimensional space (two-dimensional coordinates). That is, a three-dimensional human body model is mapped in a two-dimensional space, and the three-dimensional human body model is transformed into a two-dimensional skeleton structure in consideration of how the deformed three-dimensional human body model changes in the two-dimensional space (image). Optimize.
[0060]
Subsequently, the height calculation unit 103 calculates the number of height pixels of the fitted three-dimensional human body model as shown in FIG. 7 (S134). As shown in FIG. 21, the height calculation unit 103 obtains the number of height pixels of the three-dimensional human body model 402 in that state when the difference between the three-dimensional human body model 402 and the two-dimensional skeleton structure 401 disappears and the postures match. With the optimized 3D human body model 402 upright, the camera The length of the whole body in the two-dimensional space is calculated based on the parameter. For example, the number of height pixels is calculated from the length (number of pixels) of the bones from the head to the foot when the three-dimensional human body model 402 is upright. Similar to Specific Example 1, the lengths of the bones from the head to the foot of the three-dimensional human body model 402 may be totaled.
[0061]
In Specific Example 3, when a three-dimensional human body model is fitted to a two-dimensional skeleton structure based on camera parameters and the number of height pixels is obtained based on the three-dimensional human body model, all bones are not shown in the front. That is, since all the bones are projected diagonally, the number of height pixels can be estimated accurately even when the error is large.
[0062]

As shown in FIG. 4, the image processing apparatus 100 performs a normalization process (S104) following the height pixel number calculation process, and stores the normalized value obtained as a result of the normalized process in the database 110 (S105). ). In the normalization process, as shown in FIG. 8, the normalization unit 104 calculates the key point height (S141). The normalization unit 104 calculates the key point height (number of pixels) of all the key points included in the detected skeletal structure. The key point height is the length in the height direction from the lowest end of the skeletal structure (for example, the key point of any foot) to the key point. Here, as an example, the height of the key point is obtained from the Y coordinate of the key point in the image. As described above, the key point height may be obtained from the length in the direction along the vertical projection axis based on the camera parameters. For example, in the example of FIG. 10, the height (y i) of the key point A2 of the neck is a value obtained by subtracting the Y coordinate of the key point A81 of the right foot or the key point A82 of the left foot from the Y coordinate of the key point A2.
[0063]
Subsequently, the normalization unit 104 specifies a reference point for normalization (S142). The reference point is a reference point for expressing the relative height of the key point. The reference point may be preset or may be selectable by the user. The reference point is preferably the center or higher than the center of the skeletal structure (upper and lower in the vertical direction of the image), and for example, the coordinates of the key point of the neck are used as the reference point. The coordinates of the head and other key points, not limited to the neck, may be used as the reference point. Further, the reference point is not limited to the key point, and any coordinate (for example, the center coordinate of the skeleton structure) may be used as a reference point.
[0064]
Subsequently, the normalization unit 104 normalizes the key point height (y i) by the number of height pixels (S143). Normalize each key point using the key point height, reference point, and height pixel count of each key point. Specifically, the normalization unit 104 normalizes the relative height of the key point with respect to the reference point according to the number of height pixels. Here, as an example focusing only on the height direction, only the Y coordinate is extracted, and the reference point is used as the key point of the neck for normalization. Specifically, the feature amount (normalized value) is obtained by using the following equation (1) with the Y coordinate of the reference point (key point of the neck) as (y c). When using the vertical projection axis based on the camera parameters, (y i) and (y c) are converted into values ​​in the direction along the vertical projection axis.
[Number 1]

[0065]
For example, when there are 18 key points, the coordinates (x 0, y 0), (x 1, y 1), ... (X 17, y 17) of the 18 points of each key point are converted into the above equation (1). ) Is used to convert to 18-dimensional features as follows.
[Number 2]

[0066]
FIG. 22 shows an example of the feature amount of each key point obtained by the normalization unit 104. In this example, since the key point A2 of the neck is used as the reference point, the feature amount of the key point A2 is 0.0, and the feature amount of the key point A31 on the right shoulder and the key point A32 on the left shoulder at the same height as the neck are also. It is 0.0. The feature amount of the key point A1 of the head higher than the neck is -0.2. The feature amount of the right hand key point A51 and the left hand key point A52 lower than the neck is 0.4, and the feature amount of the right foot key point A81 and the left foot key point A82 is 0.9. When the person raises his / her left hand from this state, the left hand becomes higher than the reference point as shown in FIG. 23, so that the feature amount of the left hand key point A52 is −0.4. On the other hand, since the normalization is performed using only the coordinates of the Y-axis, the feature amount does not change even if the width of the skeleton structure changes as compared with FIG. 22 as shown in FIG. That is, the feature amount (normalized value) of the present embodiment shows the characteristics of the skeletal structure (key point) in the height direction (Y direction), and affects the change of the skeletal structure in the lateral direction (X direction). Do not receive.
[0067]

As described above, in the present embodiment, the skeletal structure of a person is detected from the two-dimensional image, and the number of height pixels (height when standing upright on the two-dimensional image space) obtained from the detected skeletal structure is used. Normalize each key point (feature point) of the skeletal structure. By using this normalized feature amount, it is possible to improve the robustness when classification, search, etc. are performed.
[0068]
That is, since the feature amount of the present embodiment is not affected by the lateral change of the person as described above, it is highly robust to the change of the direction of the person and the body shape of the person. For example, as in the skeletal structures 501 to 503 of FIG. 25, the feature amount does not change significantly even when the orientation and body shape of the person are different. Therefore, it can be determined that the skeletal structures 501 to 503 have the same posture at the time of classification or search.
[0069]
FIG. 26 shows an example of the search result, and shows the search results Q11, Q21, and Q31 of the search queries Q10, Q20, and Q30, respectively. When a sitting posture is searched as in the search query Q10, it is determined that the sitting postures facing various directions are similar as in the search result Q11. Similarly, if you search for a posture in which you stand up and extend your right hand horizontally as in search query Q20, it is determined that the postures in which you stand up and extend your right hand in various directions are similar as in search result Q21. .. Further, when the sleeping posture is searched as in the search query Q30, it is determined that the sleeping postures facing various directions are similar as in the search result Q31.
[0070]
Further, since the feature amount of this embodiment is a value obtained by normalizing each key point, it is highly robust to an image in which a part of the body is hidden. For example, as in the skeletal structures 511 and 512 of FIG. 27, even if the key point of the left foot cannot be detected due to the hiding of the left foot, the feature quantities of the other key points that have been detected can be used. Therefore, in the skeletal structures 511 and 512, it can be determined that they have the same posture at the time of classification or search. That is, it is possible to perform classification and search using the features of some key points instead of all the key points. In the examples of the skeletal structures 521 and 522 in FIG. 28, although the directions of both feet are different, the feature quantities of the key points of the upper body (A1, A2, A31, A32, A41, A42, A51, A52) are used as the search query. Therefore, it can be judged that the posture is the same.
[0071]
Further, in the present embodiment, since it can be realized by detecting the skeleton structure of the person using a skeleton estimation technique such as OpenPose, it is not necessary to prepare learning data for learning the posture of the person. Further, by normalizing the key points of the skeletal structure and storing them in the database, it is possible to classify and search the posture of a person, so that it is possible to classify and search even an unknown posture. In addition, by normalizing the key points of the skeletal structure, it is possible to obtain clear and easy-to-understand features, so unlike a black box type algorithm such as machine learning, the user is highly convinced of the processing result.
[0072]
Note that each configuration in the above-described embodiment is composed of hardware and / or software, and may be composed of one hardware or software, or may be composed of a plurality of hardware or software. The functions (processing) of the image processing devices 10 and 100 may be realized by a computer 20 having a processor 21 such as a CPU (Central Processing Unit) and a memory 22 which is a storage device, as shown in FIG. 29. For example, a program (image processing program) for performing the method in the embodiment may be stored in the memory 22, and each function may be realized by executing the program stored in the memory 22 on the processor 21.
[0073]
These programs can be stored and supplied to a computer using various types of non-transitory computer readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-temporary computer-readable media include magnetic recording media (eg flexible disks, magnetic tapes, hard disk drives), opto-magnetic recording media (eg optomagnetic disks), CD-ROMs (Read Only Memory), CD-Rs. It includes a CD-R / W and a semiconductor memory (for example, a mask ROM, a PROM (Programmable ROM), an EPROM (Erasable PROM), a flash ROM, and a RAM (random access memory)). The program may also be supplied to the computer by various types of transient computer readable media. Examples of temporary computer-readable media include electrical, optical, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.
[0074]
Further, the present disclosure is not limited to the above-described embodiment, and can be appropriately changed without departing from the spirit. For example, although the skeletal structure of a person is detected in the above, animals other than the person having the skeletal structure (mammals, reptiles, birds, amphibians, fish, etc.) may be detected.
[0075]
Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the structure and details of the present disclosure within the scope of the present disclosure.
[0076]
Part or all of the above embodiments may be described as in the following appendix, but are not limited to the following.
(Appendix 1)
A skeleton detection means that detects the two-dimensional skeleton structure of a person based on the acquired two-dimensional image,
An estimation means for estimating the standing height of the person in the two-dimensional image space based on the detected two-dimensional skeleton structure,
With the normalization means for normalizing the detected two-dimensional skeletal structure based on the estimated standing height.
Image processing device equipped with.
(Appendix 2)
The normalization means normalizes the height of the feature points included in the two-dimensional skeleton structure by the height when standing upright.
The image processing device described in Appendix 1.
(Appendix 3)
The height of the feature point is the height in the Y-axis direction in the XY coordinates representing the two-dimensional image space.
The image processing device described in Appendix 2.
(Appendix 4)
The height of the feature point is the height of the vertical projection direction in which the vertical direction with respect to the reference plane in the three-dimensional space in the real world is projected onto the two-dimensional image space based on the imaging parameters of the two-dimensional image.
The image processing device described in Appendix 2.
(Appendix 5)
The normalization means normalizes the relative height of the feature point with respect to the reference point in the two-dimensional skeleton structure by the height when standing upright.
The image processing device according to any one of Appendix 2 to 4.
(Appendix 6)
The reference point is a point above the center in the two-dimensional skeleton structure in the two-dimensional image space.
The image processing device according to Appendix 5.
(Appendix 7)
The reference point is a characteristic point of the neck or head in the two-dimensional skeletal structure.
The image processing device according to Appendix 6.
(Appendix 8)
The estimation means is the secondary The height of the person in the standing position is estimated based on the length of the bone in the two-dimensional image space included in the original skeletal structure.
The image processing device according to any one of Appendix 1 to 7.
(Appendix 9)
The estimating means estimates the standing height of the person based on the total length of the bones from the foot to the head included in the two-dimensional skeletal structure.
The image processing device according to Appendix 8.
(Appendix 10)
The estimating means estimates the height of the person in an upright position based on a two-dimensional skeleton model showing the relationship between the length of the bone and the length of the whole body of the person in the two-dimensional image space.
The image processing device according to Appendix 8.
(Appendix 11)
The estimation means estimates the height of the person in an upright position based on a three-dimensional skeleton model fitted to the two-dimensional skeleton structure based on the imaging parameters of the two-dimensional image.
The image processing device according to any one of Appendix 1 to 7.
(Appendix 12)
The skeleton detecting means detects the two-dimensional skeleton structure of a plurality of people based on the two-dimensional image, and
The normalization means normalizes the two-dimensional skeletal structure of the plurality of persons.
Further provided with a recognition means for recognizing the states of the plurality of persons based on the normalized values ​​of the plurality of normalized two-dimensional skeleton structures.
The image processing device according to any one of Appendix 1 to 11.
(Appendix 13)
The recognition means classifies the states of the plurality of persons as the recognition process.
The image processing device according to Appendix 12.
(Appendix 14)
The recognition means classifies the states of the plurality of persons based on the normalized values ​​of all or part of the two-dimensional skeletal structure.
The image processing device according to Appendix 13.
(Appendix 15)
The skeleton detecting means detects the two-dimensional skeleton structure of the plurality of persons from a plurality of two-dimensional images continuous in time series.
The recognition means classifies the states of the plurality of persons based on the normalized values ​​of the two-dimensional skeleton structures detected from the plurality of two-dimensional images.
The image processing device according to Appendix 13 or 14.
(Appendix 16)
The recognition means classifies the states of the plurality of persons based on changes in the normalized values ​​of the two-dimensional skeleton structures in the plurality of two-dimensional images.
The image processing device according to Appendix 15.
(Appendix 17)
The recognition means searches the query state from the states of the plurality of persons as the recognition process.
The image processing device according to Appendix 12.
(Appendix 18)
The recognition means searches for the query state based on the normalized values ​​of all or part of the two-dimensional skeleton structure.
The image processing device according to Appendix 17.
(Appendix 19)
The skeleton detecting means detects the two-dimensional skeleton structure of the plurality of persons from a plurality of two-dimensional images continuous in time series.
The recognition means searches for the query state based on the normalized value of the two-dimensional skeleton structure detected from the plurality of two-dimensional images.
The image processing device according to Appendix 17 or 18.
(Appendix 20)
The recognition means searches for the query state based on the change in the normalized value of the two-dimensional skeleton structure in the plurality of two-dimensional images.
The image processing device according to Appendix 19.
(Appendix 21)
Detects the 2D skeleton structure of a person based on the acquired 2D image,
Based on the detected 2D skeleton structure, the height of the person in the 2D image space when standing upright is estimated.
Normalize the detected two-dimensional skeletal structure based on the estimated upright height.
Image processing method.
(Appendix 22)
In the normalization, the height of the feature points included in the two-dimensional skeleton structure is normalized by the height when standing upright.
The image processing method described in Appendix 21.
(Appendix 23)
Detects the 2D skeleton structure of a person based on the acquired 2D image,
Based on the detected 2D skeleton structure, the height of the person in the 2D image space when standing upright is estimated.
Normalize the detected two-dimensional skeletal structure based on the estimated upright height.
An image processing program for letting a computer execute the processing.
(Appendix 24)
In the normalization, the height of the feature points included in the two-dimensional skeleton structure is normalized by the height when standing upright.
The image processing program described in Appendix 23.
Code explanation
[0077]
1 Image processing system
10 Image processing device
11 Skeleton detection unit
12 Estimate department
13 Normalization department
20 Computer
21 processor
22 Memory
100 Image processing device
101 Image acquisition department
102 Skeleton structure detector
103 Height calculation department
104 Normalization department
105 Classification department
106 Search department
110 database
200 camera
300, 301 human body model
401 Two-dimensional skeletal structure
402 3D human body model
The scope of the claims
[Claim 1]
A skeleton detection means that detects the two-dimensional skeleton structure of a person based on the acquired two-dimensional image,
An estimation means for estimating the standing height of the person in the two-dimensional image space based on the detected two-dimensional skeleton structure,
With the normalization means for normalizing the detected two-dimensional skeletal structure based on the estimated standing height.
Image processing device equipped with.
[Claim 2]
The normalization means normalizes the height of the feature points included in the two-dimensional skeleton structure by the height when standing upright.
The image processing device according to claim 1.
[Claim 3]
The height of the feature point is the height in the Y-axis direction in the XY coordinates representing the two-dimensional image space.
The image processing device according to claim 2.
[Claim 4]
The height of the feature point is the height of the vertical projection direction in which the vertical direction with respect to the reference plane in the three-dimensional space in the real world is projected onto the two-dimensional image space based on the imaging parameters of the two-dimensional image.
The image processing device according to claim 2.
[Claim 5]
The normalization means normalizes the relative height of the feature point with respect to the reference point in the two-dimensional skeleton structure by the height when standing upright.
The image processing apparatus according to any one of claims 2 to 4.
[Claim 6]
The reference point is a point above the center in the two-dimensional skeleton structure in the two-dimensional image space.
The image processing device according to claim 5.
[Claim 7]
The reference point is a characteristic point of the neck or head in the two-dimensional skeletal structure.
The image processing device according to claim 6.
[Claim 8]
The estimating means estimates the height of the person in an upright position based on the length of the bone in the two-dimensional image space included in the two-dimensional skeletal structure.
The image processing apparatus according to any one of claims 1 to 7.
[Claim 9]
The estimating means estimates the standing height of the person based on the total length of the bones from the foot to the head included in the two-dimensional skeletal structure.
The image processing device according to claim 8.
[Claim 10]
The estimating means estimates the height of the person in an upright position based on a two-dimensional skeleton model showing the relationship between the length of the bone and the length of the whole body of the person in the two-dimensional image space.
The image processing device according to claim 8.
[Claim 11]
The estimation means estimates the height of the person in an upright position based on a three-dimensional skeleton model fitted to the two-dimensional skeleton structure based on the imaging parameters of the two-dimensional image.
The image processing apparatus according to any one of claims 1 to 7.
[Claim 12]
The skeleton detecting means detects the two-dimensional skeleton structure of a plurality of people based on the two-dimensional image, and
The normalization means normalizes the two-dimensional skeletal structure of the plurality of persons.
Further provided with a recognition means for recognizing the states of the plurality of persons based on the normalized values ​​of the plurality of normalized two-dimensional skeleton structures.
The image processing apparatus according to any one of claims 1 to 11.
[Claim 13]
The recognition means classifies the states of the plurality of persons as the recognition process.
The image processing device according to claim 12.
[Claim 14]
The recognition means classifies the states of the plurality of persons based on the normalized values ​​of all or part of the two-dimensional skeletal structure.
The image processing device according to claim 13.
[Claim 15]
The skeleton detecting means detects the two-dimensional skeleton structure of the plurality of persons from a plurality of two-dimensional images continuous in time series.
The recognition means classifies the states of the plurality of persons based on the normalized values ​​of the two-dimensional skeleton structures detected from the plurality of two-dimensional images.
The image processing apparatus according to claim 13 or 14.
[Claim 16]
The recognition means classifies the states of the plurality of persons based on changes in the normalized values ​​of the two-dimensional skeleton structures in the plurality of two-dimensional images.
The image processing apparatus according to claim 15.
[Claim 17]
The recognition means searches the query state from the states of the plurality of persons as the recognition process.
The image processing device according to claim 12.
[Claim 18]
The recognition means searches for the query state based on the normalized values ​​of all or part of the two-dimensional skeleton structure.
The image processing device according to claim 17.
[Claim 19]
The skeleton detecting means detects the two-dimensional skeleton structure of the plurality of persons from a plurality of two-dimensional images continuous in time series.
The recognition means searches for the query state based on the normalized value of the two-dimensional skeleton structure detected from the plurality of two-dimensional images.
The image processing apparatus according to claim 17 or 18.
[Claim 20]
The recognition means searches for the query state based on the change in the normalized value of the two-dimensional skeleton structure in the plurality of two-dimensional images.
The image processing apparatus according to claim 19.
[Claim 21]
Detects the 2D skeleton structure of a person based on the acquired 2D image,
Based on the detected 2D skeleton structure, the height of the person in the 2D image space when standing upright is estimated.
Normalize the detected two-dimensional skeletal structure based on the estimated upright height.
Image processing method.
[Claim 22]
In the normalization, the height of the feature points included in the two-dimensional skeleton structure is normalized by the height when standing upright.
The image processing method according to claim 21.
[Claim 23]
Detects the 2D skeleton structure of a person based on the acquired 2D image,
Based on the detected 2D skeleton structure, the height of the person in the 2D image space when standing upright is estimated.
Normalize the detected two-dimensional skeletal structure based on the estimated upright height.
A non-temporary computer-readable medium that stores an image processing program for causing a computer to perform processing.
[Claim 24]
In the normalization, the height of the feature points included in the two-dimensional skeleton structure is normalized by the height when standing upright.
The non-transitory computer-readable medium according to claim 23.