Title of the invention: A non-temporary computer-readable medium containing a height estimation device, a height estimation method, and a program.
Technical field
[0001]
The present invention relates to a non-temporary computer-readable medium in which a height estimation device, a height estimation method, and a program are stored.
Background technology
[0002]
In recent years, a technique of capturing an animal such as a person with a camera and recognizing the attributes of the person or the like from the captured image has been used. For example, Patent Documents 1 to 3 are known as techniques related to the estimation of height, which is an attribute of a person or the like. Patent Document 1 describes a technique for estimating the height of a person from the length of the long side and the lengths of the long side and the short side of the person area in the image. Patent Document 2 describes a technique for estimating the height of a person based on a distance image. Patent Document 3 describes a technique for estimating height using an imaging result by an X-ray CT apparatus. In addition, Non-Patent Document 1 is known as a technique related to estimating the skeleton of a person.
Prior art literature
Patent documents
[0003]
Patent Document 1: International Publication No. 2017/209089
Patent Document 2: Japanese Patent Application Laid-Open No. 2012-120647
Patent Document 3: Japanese Patent Application Laid-Open No. 2012-231816
Non-patent literature
[0004]
Non-Patent Document 1: Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, "Realtime Multi-Person 2D Pose Optimization using Part Affinity Fields", The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, P. 7291-7299
Outline of the invention
Problems to be solved by the invention
[0005]
As described above, in Patent Document 1, since the height is estimated based on the size of the person area in the image, the height estimation accuracy may decrease depending on the posture of the person and the orientation of the person with respect to the camera. .. Further, in Patent Document 2, it is essential to acquire a distance image, and in Patent Document 3, a special contrast imaging is performed by an X-ray CT apparatus. For this reason, there is a problem that it is difficult to accurately estimate the height from a two-dimensional image of an animal such as a person in a related technique.
[0006]
In view of such problems, it is an object of the present disclosure to provide a non-temporary computer-readable medium in which a height estimation device, a height estimation method, and a program capable of improving the height estimation accuracy can be provided. ..
Means to solve the problem
[0007]
The height estimation device according to the present disclosure includes an acquisition means for acquiring a two-dimensional image of an animal, a detection means for detecting the two-dimensional skeletal structure of the animal based on the acquired two-dimensional image, and the detection. It is provided with an estimation means for estimating the height of the animal in a three-dimensional real world based on the two-dimensional skeletal structure and the imaging parameters of the two-dimensional image.
[0008]
The height estimation method according to the present disclosure acquires a two-dimensional image of an animal, detects the two-dimensional skeletal structure of the animal based on the acquired two-dimensional image, and the detected two-dimensional skeletal structure and the detected two-dimensional skeletal structure. The height of the animal in the three-dimensional real world is estimated based on the imaging parameters of the two-dimensional image.
[0009]
The non-temporary computer-readable medium in which the program according to the present disclosure is stored acquires a two-dimensional image of an animal, detects the two-dimensional skeletal structure of the animal based on the acquired two-dimensional image, and determines the two-dimensional skeletal structure of the animal. A non-temporary storage program for causing a computer to perform a process of estimating the height of the animal in the three-dimensional real world based on the detected two-dimensional skeletal structure and the imaging parameters of the two-dimensional image. It is a computer-readable medium.
Effect of the invention
[0010]
According to the present disclosure, it is possible to provide a non-temporary computer-readable medium in which a height estimation device, a height estimation method, and a program capable of improving the height estimation accuracy can be provided.
A brief description of the drawing
[0011]
[Fig. 1] Fig. 1 is a flowchart showing a related monitoring method.
FIG. 2 is a configuration diagram showing an outline of a height estimation device according to an embodiment.
FIG. 3 is a configuration diagram showing a configuration of a height estimation device according to the first embodiment.
FIG. 4 is a flowchart showing a height estimation method according to the first embodiment.
FIG. 5 is a flowchart showing a method of calculating the number of height pixels according to the first embodiment.
FIG. 6 is a diagram showing a human body model according to the first embodiment.
FIG. 7 is a diagram showing a detection example of a skeletal structure according to the first embodiment.
FIG. 8 is a diagram showing a detection example of a skeletal structure according to the first embodiment.
FIG. 9 is a diagram showing a detection example of a skeletal structure according to the first embodiment.
FIG. 10 is a diagram showing a human body model according to the second embodiment.
FIG. 11 is a flowchart showing a method of calculating the number of height pixels according to the second embodiment.
FIG. 12 is a diagram showing a detection example of a skeletal structure according to a second embodiment.
FIG. 13 is a histogram for explaining a method for calculating the number of height pixels according to the second embodiment.
FIG. 14 is a flowchart showing a height estimation method according to the third embodiment.
FIG. 15 is a diagram showing a detection example of a skeletal structure according to a third embodiment.
FIG. 16 is a diagram showing a three-dimensional human body model according to the third embodiment.
FIG. 17 is a diagram for explaining a height estimation method according to the third embodiment.
FIG. 18 is a diagram for explaining a height estimation method according to the third embodiment.
FIG. 19 is a diagram for explaining a height estimation method according to the third embodiment.
FIG. 20 is a configuration diagram showing an outline of computer hardware according to an embodiment.
Embodiment for carrying out the invention
[0012]
Hereinafter, embodiments will be described with reference to the drawings. In each drawing, the same elements are designated by the same reference numerals, and duplicate explanations are omitted as necessary.
[0013]
(Examination leading to the embodiment)
In recent years, image recognition technology utilizing machine learning has been applied to various systems. As an example, consider a surveillance system that monitors images from a surveillance camera.
[0014]
FIG. 1 shows a monitoring method in a related monitoring system. As shown in FIG. 1, the surveillance system acquires an image from the surveillance camera (S101), detects a person from the acquired image (S102), and performs behavior recognition and attribute recognition (S103) of the person. For example, it recognizes the behavior and movement lines of a person as the behavior of the person, and recognizes the age, gender, height, etc. of the person as the attributes of the person. Further, in the monitoring system, data is analyzed from the behaviors and attributes of the recognized person (S104), and actions such as coping are performed based on the analysis results (S105). For example, an alert is displayed from the recognized action, or a person with an attribute such as the recognized height is monitored.
[0015]
As shown in this example, there is an increasing demand for easily obtaining attribute information such as the age, gender, and height of a person from images and videos of a surveillance camera. Among the attributes, height is useful information for identifying an individual and distinguishing between an adult and a child. For example, attribute information is used for investigation as a characteristic of a criminal (30s, male, 170 cm, etc.), is used for marketing as information for a visitor, and is used for a lost child search as a characteristic of a lost child.
[0016]
When the inventors examined a method of recognizing the height of a person from an image, they found a problem that the height could not always be recognized (estimated) accurately with related technology. For example, when the whole body of a person is shown in the image, it is possible to estimate the height to some extent. However, the person in the image is not always upright, or the crown and feet are not always visible. In particular, in the case of a lost child, there is a high possibility that he / she is crouching. In such cases, it is difficult to estimate the height.
[0017]
Therefore, the inventors examined a method of using skeletal estimation technology using machine learning to estimate the height of a person. For example, in a related skeleton estimation technique such as OpenPose disclosed in Non-Patent Document 1, the skeleton of a person is estimated by learning the image data in which various patterns are correctly answered. In the following embodiments, it is possible to accurately estimate the height of a person by utilizing such a skeletal estimation technique.
The skeletal structure estimated by a skeletal estimation technique such as OpenPose is composed of "key points" which are characteristic points of joints and the like and "bones (bone links)" which indicate links between key points. .. Therefore, in the following embodiments, the skeletal structure will be described using the terms "key point" and "bone", but unless otherwise specified, the "key point" corresponds to the "joint" of a person and ". "Bone" corresponds to the "bone" of a person.
[0018]
(Outline of embodiment)
FIG. 2 shows an outline of the height estimation device 10 according to the embodiment. As shown in FIG. 2, the height estimation device 10 includes an acquisition unit 11, a detection unit 12, and an estimation unit 13.
[0019]
The acquisition unit 11 acquires a two-dimensional image of an animal such as a person. The detection unit 12 detects the two-dimensional skeletal structure of an animal based on the two-dimensional image acquired by the acquisition unit 11. The estimation unit 13 estimates the height of an animal in the three-dimensional real world based on the two-dimensional skeletal structure detected by the detection unit 12 and the imaging parameters of the two-dimensional image.
[0020]
As described above, in the embodiment, the two-dimensional skeletal structure of an animal such as a person is detected from the two-dimensional image, and the height of the animal in the real world is estimated based on the two-dimensional skeletal structure. Instead, the height of the animal can be estimated accurately.
[0021]
(Embodiment 1)
Hereinafter, the first embodiment will be described with reference to the drawings. FIG. 3 shows the configuration of the height estimation device 100 according to the present embodiment. The height estimation device 100 constitutes the height estimation system 1 together with the camera 200. For example, the height estimation device 100 and the height estimation system 1 are applied to the monitoring method in the monitoring system as shown in FIG. 1, estimate the height as an attribute of a person, and monitor the person with that attribute. The camera 200 may be provided inside the height estimation device 100.
[0022]
As shown in FIG. 3, the height estimation device 100 includes an image acquisition unit 101, a skeleton structure detection unit 102, a height pixel number calculation unit 103, a camera parameter calculation unit 104, a height estimation unit 105, and a storage unit 106. The configuration of each part (block) is an example, and may be composed of other parts as long as the method (operation) described later is possible. The height pixel number calculation unit 103 and the height estimation unit 105 may be used as an estimation unit for estimating the height of a person. Further, the height estimation device 100 is realized by, for example, a computer device such as a personal computer or a server that executes a program, but may be realized by one device or by a plurality of devices on a network. good.
[0023]
The storage unit 106 stores information (data) necessary for the operation (processing) of the height estimation device 100. For example, the storage unit 106 is a non-volatile memory such as a flash memory, a hard disk device, or the like. The storage unit 106 stores an image acquired by the image acquisition unit 101, an image processed by the skeletal structure detection unit 102, data for machine learning, and the like. The storage unit 106 may be an external storage device or an external storage device on the network. That is, the height estimation device 100 may acquire necessary images, machine learning data, and the like from an external storage device.
[0024]
The image acquisition unit 101 acquires a two-dimensional image captured by the camera 200 from the camera 200 connected communicably. The camera 200 is an imaging unit such as a surveillance camera that captures a person, and the image acquisition unit 101 acquires an image of a person from the camera 200.
[0025]
The skeletal structure detection unit 102 detects the two-dimensional skeletal structure of a person in the image based on the acquired two-dimensional image. The skeletal structure detection unit 102 uses a skeletal estimation technique using machine learning to recognize a person's skeleton based on features such as the recognized person's joints and the like.Detect the structure. The skeletal structure detection unit 102 uses, for example, a skeletal estimation technique such as OpenPose of Non-Patent Document 1.
[0026]
The height pixel number calculation unit 103 calculates the height of a person in a two-dimensional image when standing upright (referred to as the number of height pixels) based on the detected two-dimensional skeletal structure. It can also be said that the number of height pixels is the height of a person in a two-dimensional image (the length of the whole body of the person in the two-dimensional image space). The height pixel number calculation unit 103 obtains the height pixel number (number of pixels) from the length (length on the two-dimensional image space) of each bone of the detected skeleton structure. In the present embodiment, the number of height pixels is obtained by totaling the lengths of the bones from the head to the foot among the bones of the skeletal structure. If the skeletal structure detection unit 102 (skeletal estimation technique) does not output the crown and feet, it can be corrected by multiplying by a constant if necessary.
[0027]
The camera parameter calculation unit 104 calculates the camera parameters, which are the imaging conditions of the camera 200, based on the image captured by the camera 200. The camera parameter is an image pickup parameter of an image, and is a parameter for converting a length in a two-dimensional image into a three-dimensional real-world real-world length. For example, the camera parameters are the posture, position, imaging angle, focal distance, and the like of the camera 200. The camera 200 can capture an object whose length is known in advance, and obtain camera parameters from the image.
[0028]
The height estimation unit 105 estimates the height of a person in the three-dimensional real world based on the calculated camera parameters and the number of height pixels in the two-dimensional image. The height estimation unit 105 obtains the relationship between the length of the pixels in the image and the length in the real world from the camera parameters, and converts the number of height pixels into the height in the real world.
[0029]
4 and 5 show the operation of the height estimation device 100 according to the present embodiment. FIG. 4 shows the flow from image acquisition to height estimation in the height estimation device 100, and FIG. 5 shows the flow of the height pixel number calculation process (S203) of FIG.
[0030]
As shown in FIG. 4, the height estimation device 100 acquires an image from the camera 200 (S201). The image acquisition unit 101 acquires an image of an image of a person for detecting a skeletal structure, and acquires an image of an object of a predetermined length for calculation of camera parameters.
[0031]
Subsequently, the height estimation device 100 detects the skeletal structure of the person based on the acquired image of the person (S202). FIG. 6 shows the skeletal structure of the human body model 300 detected at this time, and FIGS. 7 to 9 show an example of detecting the skeletal structure. The skeletal structure detection unit 102 detects the skeletal structure of the human body model (two-dimensional skeletal model) 300 as shown in FIG. 6 from a two-dimensional image by using a skeletal estimation technique such as OpenPose. The human body model 300 is a two-dimensional model composed of key points such as human joints and bones connecting the key points.
[0032]
The skeletal structure detection unit 102, for example, extracts feature points that can be key points from an image, and detects each key point of a person by referring to information obtained by machine-learning the image of the key points. In the example of FIG. 6, as key points of a person, head A1, neck A2, right shoulder A31, left shoulder A32, right elbow A41, left elbow A42, right hand A51, left hand A52, right waist A61, left waist A62, right knee A71. , Left knee A72, right foot A81, left foot A82 are detected. Further, as the bones of the person connecting these key points, the bone B1 connecting the head A1 and the neck A2, the bones B21 and B22 connecting the neck A2 and the right shoulder A31 and the left shoulder A32, respectively, the right shoulder A31 and the left shoulder A32 and the right Bone B31 and B32 connecting elbow A41 and left elbow A42, respectively, bone B41 and bone B42 connecting right elbow A41 and left elbow A42 with right hand A51 and left hand A52, respectively, connecting neck A2 and right waist A61 and left waist A62, respectively. Bones B51 and B52, right waist A61 and left waist A62, right knee A71 and left knee A72, respectively, bone B61 and bone B62, right knee A71 and left knee A72, right foot A81 and left foot A82, respectively. B72 is detected.
[0033]
FIG. 7 is an example of detecting an upright person. In FIG. 7, an upright person is imaged from the front, and bones B1, bone B51 and bone B52, bones B61 and bone B62, bones B71 and bone B72 viewed from the front are detected without overlapping, and the right foot is detected. Bone B61 and Bone B71 are slightly bent more than Bone B62 and Bone B72 of the left foot. FIG. 8 is an example of detecting a person in a crouched state. In FIG. 8, a crouching person is imaged from the right side, and bone B1, bone B51 and bone B52, bone B61 and bone B62, bone B71 and bone B72, respectively, viewed from the right side are detected, and bone B61 of the right foot is detected. And bone B71 and bone B62 and bone B72 of the left foot are greatly bent and overlapped. FIG. 9 is an example of detecting a person who is sleeping. In FIG. 9, a sleeping person is imaged from diagonally left front, and bone B1, bone B51 and bone B52, bone B61 and bone B62, bone B71 and bone B72 viewed from diagonally left front are detected, respectively, and the right foot. Bone B61 and B71 and bone B62 and bone B72 of the left foot are bent and overlapped.
[0034]
Subsequently, the height estimation device 100 performs a height pixel number calculation process based on the detected skeleton structure (S203). In the height pixel number calculation process, as shown in FIG. 5, the height pixel number calculation unit 103 acquires the length of each bone (S211) and totals the lengths of the acquired bones (S212). The height pixel number calculation unit 103 acquires the length of a bone on a two-dimensional image of a foot from the head of a person, and obtains the number of height pixels. That is, from the image in which the skeletal structure is detected, among the bones of FIG. 6, bone B1 (length L1), bone B51 (length L21), bone B61 (length L31) and bone B71 (length L41), or , Bone B1 (length L1), bone B52 (length L22), bone B62 (length L32) and bone B72 (length L42) are acquired. The length of each bone can be obtained from the coordinates of each key point in the two-dimensional image. The sum of these is calculated as the number of height pixels by multiplying L1 + L21 + L31 + L41 or L1 + L22 + L32 + L42 by a correction constant. When both values ​​can be calculated, for example, the longer value is taken as the number of height pixels. That is, each bone has the longest length in the image when it is imaged from the front, and it is displayed short when it is tilted in the depth direction with respect to the camera. Therefore, it is more likely that the longer bone is imaged from the front, which is considered to be closer to the true value. Therefore, it is preferable to select the longer value.
[0035]
In the example of FIG. 7, bone B1, bone B51 and bone B52, bone B61 and bone B62, bone B71 and bone B72 are detected without overlapping. The total of these bones, L1 + L21 + L31 + L41 and L1 + L22 + L32 + L42, is obtained, and for example, the value obtained by multiplying L1 + L22 + L32 + L42 on the left foot side where the detected bone length is long by a correction constant is taken as the number of height pixels.
[0036]
In the example of FIG. 8, bone B1, bone B51 and bone B52, bone B61 and bone B62, bone B71 and bone B72 are detected, respectively, and bone B61 and bone B71 of the right foot and bone B62 and bone B72 of the left foot overlap each other. .. The total of these bones, L1 + L21 + L31 + L41 and L1 + L22 + L32 + L42, is obtained, and for example, the value obtained by multiplying L1 + L21 + L31 + L41 on the right foot side where the detected bone length is long by a correction constant is taken as the number of height pixels.
[0037]
In the example of FIG. 9, bone B1, bone B51 and bone B52, bone B61 and bone B62, bone B71 and bone B72 are detected, respectively, and bone B61 and bone B71 of the right foot and bone B62 and bone B72 of the left foot overlap each other. .. The total of these bones, L1 + L21 + L31 + L41 and L1 + L22 + L32 + L42, is obtained, and for example, the value obtained by multiplying L1 + L22 + L32 + L42 on the left foot side where the detected bone length is long by a correction constant is taken as the number of height pixels.
[0038]
On the other hand, as shown in FIG. 4, the height estimation device 100 calculates the camera parameters based on the image captured by the camera 200 (S205). The camera parameter calculation unit 104 extracts an object whose length is known in advance from a plurality of images captured by the camera 200, and obtains a camera parameter from the size (number of pixels) of the extracted object. The camera parameters may be obtained in advance, and the obtained camera parameters may be acquired as needed.
[0039]
Subsequently, the height estimation device 100 estimates the height of a person based on the number of height pixels and camera parameters (S204). The height estimation unit 105 obtains a three-dimensional real-world length for one pixel in a region where a person is present in a two-dimensional image, that is, an actual length in pixel units, by a camera parameter. In particular, since the length of the real world for one pixel in the image changes depending on the location in the image, the "length of the real world per pixel in the region where a person exists" in the image is obtained. The number of height pixels is converted into height from the obtained actual length in pixel units. For example, in FIG. 8, when the total length of bone B1, bone B51, bone B61, and bone B71 is L1 + L21 + L31 + L41 = 100 pixels, and 1 pixel in the area where a person is present = 1.7 cm, the height is 170 cm. Become.
[0040]
As described above, in the present embodiment, the skeletal structure of a person is detected from the two-dimensional image, and the number of height pixels is obtained by totaling the lengths of the bones on the two-dimensional image of the detected skeletal structure. In addition, the real-world height of the person is estimated in consideration of the camera parameters. Since the height can be calculated by summing the lengths of the bones from the head to the legs, the height can be estimated by a simple method. In addition, since it is only necessary to detect the skeleton from the head to the foot by skeletal estimation technology using machine learning, the height can be estimated accurately even when the entire person is not always shown in the image, such as when crouching down. be able to.
[0041]
(Embodiment 2)
Next, the second embodiment will be described. In the present embodiment, in the height pixel number calculation process of the first embodiment, the number of height pixels is used by using a human body model showing the relationship between the length of each bone and the length of the whole body (height on the two-dimensional image space). Is calculated. The process is the same as that of the first embodiment except for the height pixel number calculation process.
[0042]
FIG. 10 is a human body model (two-dimensional skeleton model) 301 showing the relationship between the length of each bone in the two-dimensional image space and the length of the whole body in the two-dimensional image space used in the present embodiment. As shown in FIG. 10, the relationship between the length of each bone of an average person and the length of the whole body (the ratio of the length of each bone to the length of the whole body) is associated with each bone of the human body model 301. For example, the length of the bone B1 of the head is the length of the whole body x 0.2 (20%), the length of the bone B41 of the right hand is the length of the whole body x 0.15 (15%), and the length of the right foot. The length of bone B71 is the length of the whole body × 0.25 (25%). By storing the information of the human body model 301 in the storage unit 106, the average length of the whole body (number of pixels) can be obtained from the length of each bone. In addition to the human body model of an average person, a human body model may be prepared for each attribute of the person such as age, gender, and nationality. As a result, the length (height) of the whole body can be appropriately obtained according to the attributes of the person.
[0043]
FIG. 11 is a height pixel number calculation process according to the present embodiment, and shows the flow of the height pixel number calculation process (S203) of FIG. 4 in the first embodiment. In the height pixel number calculation process of the present embodiment, as shown in FIG. 11, the height pixel number calculation unit 103 is the length of each bone.(S301). The height pixel number calculation unit 103 acquires the lengths (lengths in the two-dimensional image space) of all the bones in the skeletal structure detected as in the first embodiment. FIG. 12 is an example in which a person in a crouched state is imaged from diagonally right behind and the skeletal structure is detected. In this example, since the face and left side of the person are not shown, the bones of the head and the bones of the left arm and the left hand cannot be detected. Therefore, the lengths of the detected bones B21, B22, B31, B41, B51, B52, B61, B62, B71, and B72 are acquired.
[0044]
Subsequently, the height pixel number calculation unit 103 calculates the height pixel number from the length of each bone based on the human body model (S302). The height pixel number calculation unit 103 refers to the human body model 301 showing the relationship between each bone and the length of the whole body as shown in FIG. 10, and obtains the number of height pixels from the length of each bone. For example, since the length of the bone B41 on the right hand is the length of the whole body × 0.15, the number of height pixels based on the bone B41 is obtained by the length of the bone B41 / 0.15. Further, since the length of the bone B71 of the right foot is the length of the whole body × 0.25, the number of height pixels based on the bone B71 is obtained from the length of the bone B71 / 0.25.
[0045]
The human body model referred to at this time is, for example, a human body model of an average person, but a human body model may be selected according to the attributes of the person such as age, gender, and nationality. For example, when a person's face is shown in the captured image, the attribute of the person is identified based on the face, and the human body model corresponding to the identified attribute is referred to. It is possible to recognize the attributes of a person from the features of the face in the image by referring to the information obtained by machine-learning the face for each attribute. Further, when the attribute of the person cannot be identified from the image, the human body model of the average person may be used.
[0046]
Subsequently, the height pixel number calculation unit 103 calculates the optimum value of the height pixel number (S303). The height pixel number calculation unit 103 calculates the optimum value of the height pixel number from the height pixel number obtained for each bone. For example, as shown in FIG. 13, a histogram of the number of height pixels obtained for each bone is generated, and a large number of height pixels is selected from the histogram. That is, the number of height pixels longer than the others is selected from the plurality of height pixels obtained based on the plurality of bones. For example, the top 30% is set as a valid value, and in FIG. 13, the number of height pixels by bones B71, B61, and B51 is selected. The average number of selected height pixels may be obtained as the optimum value, or the largest number of height pixels may be used as the optimum value. Since the height is calculated from the length of the bone in the two-dimensional image, if the bone cannot be imaged from the front, that is, if the bone is tilted in the depth direction when viewed from the camera, the length of the bone is imaged from the front. It will be shorter than if you did. Then, a value having a large number of height pixels is more likely to be imaged from the front than a value having a small number of height pixels, and is a more plausible value. Therefore, a larger value is set as the optimum value.
[0047]
As described above, in the present embodiment, the number of height pixels is based on the detected bones of the skeletal structure using a human body model showing the relationship between the bones on the two-dimensional image space and the length of the whole body. Estimate the real-world height of a person by asking for. This makes it possible to estimate the height from some bones even if not all the skeletons from the head to the feet can be obtained. In particular, by adopting a larger value among the heights (number of height pixels) obtained from a plurality of bones, the height can be estimated accurately.
[0048]
(Embodiment 3)
Next, the third embodiment will be described. In the present embodiment, instead of the height pixel number calculation process and the height estimation process of the first embodiment, the height in the real world is estimated by fitting (fitting) the three-dimensional human body model to the two-dimensional skeleton structure. Others are the same as those in the first embodiment.
[0049]
FIG. 14 shows the flow of the height estimation process according to the present embodiment. In the height estimation process of the present embodiment, as shown in FIG. 14, first, as in FIG. 4 of the first embodiment, the height estimation device 100 acquires a two-dimensional image from the camera 200 (S201) and obtains an image. The two-dimensional skeletal structure of the person inside is detected (S202), and the camera parameters are calculated (S205). Subsequently, the height estimation unit 105 of the height estimation device 100 adjusts the arrangement and height of the three-dimensional human body model (S401). The height estimation unit 105 prepares a three-dimensional human body model for height calculation for the two-dimensional skeletal structure detected as in the first embodiment, and arranges the three-dimensional human body model in the same two-dimensional image based on the camera parameters. Specifically, the "relative positional relationship between the camera and the person in the real world" is specified from the camera parameters and the two-dimensional skeleton structure. For example, assuming that the position of the camera is the coordinate (0,0,0), the coordinate (x, y, z) of the position where the person is standing (or sitting) is specified. Then, by assuming an image in which a three-dimensional human body model is placed at the same position (x, y, z) as the specified person and captured, the two-dimensional skeletal structure and the three-dimensional human body model are superimposed.
[0050]
FIG. 15 is an example in which a crouching person is imaged diagonally from the front left and the two-dimensional skeletal structure 401 is detected. The two-dimensional skeleton structure 401 has two-dimensional coordinate information. It is preferable that all bones are detected, but some bones may not be detected. For this two-dimensional skeleton structure 401, a three-dimensional human body model 402 as shown in FIG. 16 is prepared. The three-dimensional human body model (three-dimensional skeleton model) 402 has three-dimensional coordinate information and is a model of a skeleton having the same shape as the two-dimensional skeleton structure 401. Then, as shown in FIG. 17, the prepared three-dimensional human body model 402 is arranged and superimposed on the detected two-dimensional skeleton structure 401. In addition, the height of the three-dimensional human body model 402 is adjusted so as to match the two-dimensional skeleton structure 401.
[0051]
The three-dimensional human body model 402 prepared at this time may be a model in a state close to the posture of the two-dimensional skeleton structure 401 as shown in FIG. 17, or may be a model in an upright state. For example, a technique of estimating a posture in a three-dimensional space from a two-dimensional image using machine learning may be used to generate a three-dimensional human body model 402 of the estimated posture. By learning the information of the joints in the two-dimensional image and the joints in the three-dimensional space, the three-dimensional posture can be estimated from the two-dimensional image.
[0052]
Subsequently, the height estimation unit 105 fits the three-dimensional human body model into the two-dimensional skeletal structure (S402). As shown in FIG. 18, the height estimation unit 105 superimposes the three-dimensional human body model 402 on the two-dimensional skeletal structure 401 so that the postures of the three-dimensional human body model 402 and the two-dimensional skeletal structure 401 match. Transform the dimensional human body model 402. That is, the height, body orientation, and joint angle of the three-dimensional human body model 402 are adjusted and optimized so that there is no difference from the two-dimensional skeletal structure 401. For example, the joints of the three-dimensional human body model 402 are rotated within the range of movement of the person, the entire three-dimensional human body model 402 is rotated, and the overall size is adjusted. The fitting of the three-dimensional human body model and the two-dimensional skeleton structure is performed in the two-dimensional space (two-dimensional coordinates). That is, a three-dimensional human body model is mapped in a two-dimensional space, and the three-dimensional human body model is converted into a two-dimensional skeleton structure in consideration of how the deformed three-dimensional human body model changes in the two-dimensional space (image). Optimize.
[0053]
Subsequently, the height estimation unit 105 calculates the height of the fitted three-dimensional human body model (S403). As shown in FIG. 19, the height estimation unit 105 obtains the height of the three-dimensional human body model 402 in that state when the difference between the three-dimensional human body model 402 and the two-dimensional skeletal structure 401 disappears and the postures match. Since the height of the three-dimensional human body model when the optimization is completed becomes the height in the real world (for example, the height in cm units) as it is, in this embodiment, the height is as in the first and second embodiments. There is no need to calculate the number of pixels. For example, the height is calculated from the length of the bones from the head to the legs when the three-dimensional human body model 402 is upright. Similar to the first embodiment, the lengths of the bones from the head to the foot of the three-dimensional human body model 402 may be totaled.
[0054]
As described above, in the present embodiment, the three-dimensional human body model is fitted to the two-dimensional skeletal structure based on the camera parameters, and the height of the person in the real world is estimated based on the three-dimensional human body model. Specifically, the height of the fitted three-dimensional human body model is used as the estimated height. As a result, the height can be estimated accurately even when all the bones are not shown in the front, that is, even when all the bones are shown diagonally and the error is large. If the methods of the first to third embodiments are applicable, the height may be determined by using all the methods (or a method in which any of them is combined). In that case, the one closer to the average height of the person may be set as the optimum value.
[0055]
Note that each configuration in the above-described embodiment is configured by hardware and / or software, and may be composed of one hardware or software, or may be composed of a plurality of hardware or software. The functions (processing) of the height estimation devices 10 and 100 may be realized by a computer 20 having a processor 21 such as a CPU (Central Processing Unit) and a memory 22 which is a storage device, as shown in FIG. For example, a program (height estimation program) for performing the method in the embodiment may be stored in the memory 22, and each function may be realized by executing the program stored in the memory 22 on the processor 21.
[0056]
These programs are stored using various types of non-transitory computer readable medium and can be supplied to the computer. Non-temporary computer-readable media include various types of tangible storage media. Examples of non-temporary computer-readable media include magnetic recording media (eg, flexible discs, magnetic tapes, hard disk drives), optomagnetic recording media (eg, optomagnetic discs), CD-ROMs (Read Only Memory), CD-Rs, etc. Includes CD-R / W, semiconductor memory (eg, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (random access memory)). The program may also be supplied to the computer by various types of transient computer readable medium. Examples of temporary computer readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.
[0057]
Further, the present disclosure is not limited to the above-described embodiment, and can be appropriately changed without departing from the spirit. For example, although the height of a person is estimated above, the height of an animal other than a person having a skeletal structure (mammalia, reptile, bird, amphibian, fish, etc.) may be estimated.
[0058]
Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. The structure and details of the present disclosure may be modified in various ways within the scope of the present disclosure as may be understood by those skilled in the art.
[0059]
A part or all of the above embodiment may be described as in the following appendix, but is not limited to the following.
(Appendix 1)
Acquisition means to acquire a two-dimensional image of an animal,
A detection means for detecting the two-dimensional skeletal structure of the animal based on the acquired two-dimensional image,
Based on the detected 2D skeletal structure and the imaging parameters of the 2D image, the body of the animal in the 3D real world Estimating means to estimate the length and
A height estimation device equipped with.
(Appendix 2)
The estimation means estimates the height based on the length of the bone in the two-dimensional image space included in the two-dimensional skeletal structure.
The height estimation device described in Appendix 1.
(Appendix 3)
The estimation means estimates the height based on the total length of the bones from the foot to the head included in the two-dimensional skeletal structure.
The height estimation device described in Appendix 2.
(Appendix 4)
The estimation means estimates the height based on a two-dimensional skeletal model showing the relationship between the length of the bone and the length of the whole body of the animal in the two-dimensional image space.
The height estimation device described in Appendix 2.
(Appendix 5)
The estimation means estimates the height based on the two-dimensional skeletal model corresponding to the attributes of the animal.
The height estimation device described in Appendix 4.
(Appendix 6)
The estimation means estimates the height based on the height that is longer than the others among the plurality of heights obtained based on the plurality of bones in the two-dimensional skeletal structure.
The height estimation device described in Appendix 4 or 5.
(Appendix 7)
The estimation means estimates the height based on a three-dimensional skeleton model fitted to the two-dimensional skeletal structure based on the imaging parameters.
The height estimation device described in Appendix 1.
(Appendix 8)
The estimation means uses the height of the fitted three-dimensional skeleton model as the estimated height.
The height estimation device described in Appendix 7.
(Appendix 9)
Acquire a two-dimensional image of an animal,
Detecting the skeletal structure of the 2D animal based on the acquired 2D image,
The height of the animal in the three-dimensional real world is estimated based on the detected two-dimensional skeletal structure and the imaging parameters of the two-dimensional image.
Height estimation method.
(Appendix 10)
In the height estimation, the height is estimated based on the length of the bone in the two-dimensional image space included in the two-dimensional skeletal structure.
The height estimation method described in Appendix 9.
(Appendix 11)
Acquire a two-dimensional image of an animal,
Detecting the 2D skeletal structure of the animal based on the acquired 2D image,
The height of the animal in the three-dimensional real world is estimated based on the detected two-dimensional skeletal structure and the imaging parameters of the two-dimensional image.
A height estimation program for letting a computer execute the process.
(Appendix 12)
In the height estimation, the height is estimated based on the length of the bone in the two-dimensional image space included in the two-dimensional skeletal structure.
The height estimation program described in Appendix 11.
(Appendix 13)
Equipped with a camera and height estimation device
The height estimation device is
Acquisition means for acquiring a two-dimensional image of an animal taken from the camera,
A detection means for detecting the two-dimensional skeletal structure of the animal based on the acquired two-dimensional image,
An estimation means for estimating the height of the animal in the three-dimensional real world based on the detected two-dimensional skeletal structure and the imaging parameters of the two-dimensional image.
A height estimation system equipped with.
(Appendix 14)
The estimation means estimates the height based on the length of the bone in the two-dimensional image space included in the two-dimensional skeletal structure.
The height estimation system described in Appendix 13.
Description of the sign
[0060]
1 Height estimation system
10 Height estimation device
11 Acquisition department
12 Detection unit
13 Estimator
20 Computer
21 processor
22 Memory
100 Height estimation device
101 Image acquisition department
102 Skeletal structure detector
103 Height pixel number calculation unit
104 Camera parameter calculation unit
105 Height estimation department
106 Storage unit
200 camera
300, 301 human body model
401 Two-dimensional skeleton structure
402 3D human body model
The scope of the claims
[Claim 1]
Acquisition means to acquire a two-dimensional image of an animal,
A detection means for detecting the two-dimensional skeletal structure of the animal based on the acquired two-dimensional image,
An estimation means for estimating the height of the animal in the three-dimensional real world based on the detected two-dimensional skeletal structure and the imaging parameters of the two-dimensional image.
A height estimation device equipped with.
[Claim 2]
The estimation means estimates the height based on the length of the bone in the two-dimensional image space included in the two-dimensional skeletal structure.
The height estimation device according to claim 1.
[Claim 3]
The estimation means estimates the height based on the total length of the bones from the foot to the head included in the two-dimensional skeletal structure.
The height estimation device according to claim 2.
[Claim 4]
The estimation means estimates the height based on a two-dimensional skeletal model showing the relationship between the length of the bone and the length of the whole body of the animal in the two-dimensional image space.
The height estimation device according to claim 2.
[Claim 5]
The estimation means estimates the height based on the two-dimensional skeletal model corresponding to the attributes of the animal.
The height estimation device according to claim 4.
[Claim 6]
The estimation means estimates the height based on the height that is longer than the others among the plurality of heights obtained based on the plurality of bones in the two-dimensional skeletal structure.
The height estimation device according to claim 4 or 5.
[Claim 7]
The estimation means estimates the height based on a three-dimensional skeleton model fitted to the two-dimensional skeletal structure based on the imaging parameters.
The height estimation device according to claim 1.
[Claim 8]
The estimation means uses the height of the fitted three-dimensional skeleton model as the estimated height.
The height estimation device according to claim 7.
[Claim 9]
Acquire a two-dimensional image of an animal,
Detecting the 2D skeletal structure of the animal based on the acquired 2D image,
The height of the animal in the three-dimensional real world is estimated based on the detected two-dimensional skeletal structure and the imaging parameters of the two-dimensional image.
Height estimation method.
[Claim 10]
Acquire a two-dimensional image of an animal,
Detecting the 2D skeletal structure of the animal based on the acquired 2D image,
The height of the animal in the three-dimensional real world is estimated based on the detected two-dimensional skeletal structure and the imaging parameters of the two-dimensional image.
A non-temporary computer-readable medium containing a program for causing a computer to execute processing.