Title of invention: Human body detection device, human body detection method, and recording medium
Technical field
[0001]
The present invention relates to technology for detecting a human body from an image.
Background technology
[0002]
In recent years, many object detection technologies using neural networks using deep learning have been proposed. Object detection involves estimating what an object in an image or video is, and at the same time estimating the position and size of the object by obtaining a circumscribed rectangle of the object called a "bounding box". Therefore, the object detector outputs the position coordinates of the bounding box of the object, the category of the object, and the confidence indicating the probability of being in that category.
[0003]
An example of an object detection device is described in Non-Patent Document 1. In this document, a classifier that outputs the bounding box position of the object candidate position from within the image and the reliability representing the object-likeness, and the bounding box position from the object candidate position obtained above, the object category, and the reliability of the category are output. A discriminator is provided.
[0004]
Human body detection is one of the most important tasks in the field of object detection. Human body detection from moving images can be applied to various applications such as automatic driving, security monitoring, and biometric authentication. Especially in the real world, it is possible to assume the problem of shielding such as overlapping of people in a crowded environment and partial hiding of the torso by obstacles. Patent Document 1 describes a method of calculating the distance between a face region and a human body region detected from an image, and deleting the human body region as inappropriate when the face region and the human body region are in a situation where the face region and the human body region are actually impossible. is doing.
prior art documents
patent literature
[0005]
Patent document 1: JP 2018-088049 A
Non-patent literature
[0006]
Non-Patent Document 1: Ren, Shaoquing, et al. "Faster r-cnn: Towards real-time object detection with region proposed networks". Advances in neural information processing systems, 2015.
SUMMARY OF THE INVENTION
Problems to be Solved by the Invention
[0007]
The method of Non-Patent Document 1 has the problem that it is not possible to detect the human body with high accuracy in scenes where the object to be detected is shielded. In this case, there are two cases of shielding. One is overlap between different categories. For example, a pedestrian may overlap an obstacle such as a fence or a car, and a part of the body may not be visible. In this case, the visible region of the body becomes small, lack of information occurs, and it becomes difficult to estimate the bounding box position of the whole body.
[0008]
The other is the overlap between the same categories. For example, in situations where people are crowded, such as public facilities and event venues, people overlap, and occlusion makes it difficult to estimate the bounding box position of the whole body. In object detection, when a plurality of bounding box estimation results are obtained for the same object in an image, these are integrated into one by a method called NMS (Non Maximum Suppression). Therefore, even if the bounding box position can be estimated, the NMS process will actually determine that different objects are the same object, and the bounding box with the lower reliability will be rejected. Therefore, an object that has been detected once is discarded by NMS processing and becomes undetected.
[0009]
One object of the present invention is to provide a human body detection device that is robust against shielding.
Means to solve problems
[0010]
In one aspect of the present invention, the human body detection device
a partial rectangle estimation unit that estimates a specific part of the human body from the image and outputs the coordinates and reliability of the partial rectangle containing the part;
a human body candidate region estimating unit that estimates and outputs a human body candidate region, which is a region in which a human body corresponding to the partial rectangle is predicted to exist, based on the coordinates of the partial rectangle;
a human body rectangle estimation unit for estimating a human body rectangle including the human body based on the human body candidate region and outputting coordinates and reliability of the human body rectangle.
[0011]
In another aspect of the present invention, the human body detection method is
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
　Based on the coordinates of the partial rectangle, estimate and output a human body candidate area, which is an area in which a human body corresponding to the part is predicted to exist,
A human body rectangle containing a human body is estimated based on the human body candidate region, and the coordinates and reliability of the human body rectangle are output.
[0012]
In another aspect of the present invention, the recording medium is
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
　Based on the coordinates of the partial rectangle, estimate and output a human body candidate area, which is an area in which a human body corresponding to the part is predicted to exist,
A program is recorded that causes a computer to execute a process of estimating a human body rectangle containing a human body based on the human body candidate region and outputting the coordinates and reliability of the human body rectangle.
[0013]
In another aspect of the present invention, the human body detection device
a first partial rectangle estimation unit that estimates a specific part of the human body from the image and outputs the coordinates and reliability of a first partial rectangle containing the part;
a first human body candidate region estimation unit for estimating and outputting a first human body candidate region, which is a region in which a human body corresponding to the first partial rectangle is predicted to exist, based on the coordinates of the first partial rectangle; ,
a first human body rectangle estimation unit for estimating a first human body rectangle including a human body based on the first human body candidate region and outputting the coordinates and reliability of the human body rectangle;
a second human body candidate region estimation unit that estimates and outputs a second human body candidate region from the image;
Second partial rectangle estimation for estimating a specific part corresponding to the human body based on the second human body candidate region and outputting coordinates and reliability of a second partial rectangle containing the part Department and
a second human body rectangle estimation unit for estimating a second human body rectangle including the human body based on the second human body candidate region and outputting the coordinates and reliability of the human body rectangle;
For a plurality of pairs of partial rectangles and human body rectangles that correspond to each other, the coordinates and reliability of the partial rectangles and the coordinates and reliability of the human body rectangles are obtained, and the overlapping ratio of the partial rectangles and the ratio of the human body rectangles to each other a human body integration unit that integrates overlapping human body rectangles based on the overlapping ratio.
[0014]
In another aspect of the present invention, the human body detection method is
　Estimate a specific part of the human body from the image, output the coordinates and reliability of the first partial rectangle containing the part,
　Based on the coordinates of the first partial rectangle, estimate and output a first human body candidate area, which is an area in which a human body corresponding to the part is predicted to exist,
Estimate a first human body rectangle containing the human body based on the first human body candidate region, output the coordinates and reliability of the human body rectangle,
Estimate and output the second human body candidate region from the image,
Based on the second human body candidate region, estimate a specific part corresponding to the human body, output the coordinates and reliability of a second partial rectangle containing the part,
Estimate a second human body rectangle containing the human body based on the second human body candidate region, output the coordinates and reliability of the human body rectangle,
For a plurality of pairs of partial rectangles and human body rectangles that correspond to each other, the coordinates and reliability of the partial rectangles and the coordinates and reliability of the human body rectangles are obtained, and the overlapping ratio of the partial rectangles and the ratio of the human body rectangles to each other Merge the overlapping body rectangles based on the overlapping ratio.
[0015]
In another aspect of the present invention, the recording medium is
　Estimate a specific part of the human body from the image, output the coordinates and reliability of the first partial rectangle containing the part,
　Based on the coordinates of the first partial rectangle, estimate and output a first human body candidate area, which is an area in which a human body corresponding to the part is predicted to exist,
Estimate a first human body rectangle containing the human body based on the first human body candidate region, output the coordinates and reliability of the human body rectangle,
Estimate and output the second human body candidate region from the image,
Based on the second human body candidate region, estimate a specific part corresponding to the human body, output the coordinates and reliability of a second partial rectangle containing the part,
Estimate a second human body rectangle containing the human body based on the second human body candidate region, output the coordinates and reliability of the human body rectangle,
For a plurality of pairs of partial rectangles and human body rectangles that correspond to each other, the coordinates and reliability of the partial rectangles and the coordinates and reliability of the human body rectangles are obtained, and the overlapping ratio of the partial rectangles and the ratio of the human body rectangles to each other A program is recorded that causes a computer to integrate overlapping human body rectangles based on the overlapping ratio.
[0016]
In another aspect of the present invention, the human body detection device
a partial rectangle estimation unit that estimates a specific part of the human body from the image and outputs the coordinates and reliability of the partial rectangle containing the part;
a human body center estimation unit that estimates the center position of the human body corresponding to the part estimated by the partial rectangle estimation unit;
a human body region estimation unit that estimates and outputs a human body region based on the coordinates of the partial rectangle and the center position of the human body;
a human body rectangle estimation unit that estimates a human body rectangle including the human body from the image and outputs the coordinates and reliability of the human body rectangle;
an integration candidate determination unit that determines a human body rectangle with a lower reliability as an integration candidate human body rectangle from among a plurality of human body rectangles in which the ratio of overlap between the human body rectangles is greater than a third threshold;
and a human body rectangle integration unit that rejects human body rectangles other than the human body rectangles whose overlapping ratio with the human body region is larger than a fourth threshold among the human body rectangles of the integration candidates.
[0017]
In another aspect of the present invention, the human body detection method is
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
Estimate the center position of the human body corresponding to the estimated part,
Estimate and output the human body region based on the coordinates of the partial rectangle and the center position of the human body,
Estimate a human body rectangle containing the human body from the image, output the coordinates and reliability of the human body rectangle,
Among a plurality of human body rectangles whose overlapping ratio between the human body rectangles is larger than the third threshold, the human body rectangle with the lower reliability is determined as the integration candidate human body rectangle,
Among the human body rectangles of the integration candidates, human body rectangles other than the human body rectangles whose overlapping ratio with the human body region is greater than a fourth threshold are rejected.
[0018]
In another aspect of the present invention, the recording medium is
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
Estimate the center position of the human body corresponding to the estimated part,
Estimate and output the human body region based on the coordinates of the partial rectangle and the center position of the human body,
Estimate a human body rectangle containing the human body from the image, output the coordinates and reliability of the human body rectangle,
Among a plurality of human body rectangles whose overlapping ratio between the human body rectangles is larger than the third threshold, the human body rectangle with the lower reliability is determined as the integration candidate human body rectangle,
A program for causing a computer to execute a process of rejecting human body rectangles other than the human body rectangles whose overlapping ratio with the human body region is greater than a fourth threshold among the integration candidate human body rectangles is recorded.
[0019]
In another aspect of the present invention, the human body detection device
a partial rectangle estimation unit that estimates a specific part of the human body from the image and outputs the coordinates and reliability of the partial rectangle containing the part;
a human body rectangle estimation unit that estimates a human body rectangle including the human body from the image and outputs the coordinates and reliability of the human body rectangle;
a threshold determination unit that determines a fifth threshold based on the number of partial rectangles;
a threshold determination unit that obtains a sixth threshold based on the human body region estimated from the head rectangle;
a threshold determination unit that determines a seventh threshold between the fifth threshold and the sixth threshold using the fifth threshold and the sixth threshold;
a human body rectangle integration unit that excludes a human body rectangle with a lower degree of reliability from among a plurality of human body rectangles whose overlapping ratio is greater than a seventh threshold.
[0020]
In another aspect of the present invention, the human body detection method is
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
　From the image, a human body rectangle including the human body is Estimate, output the coordinates and reliability of the human body rectangle,
A fifth threshold is obtained based on the number of partial rectangles,
A sixth threshold is obtained based on the human body region estimated from the head rectangle,
using the fifth threshold and the sixth threshold to determine a seventh threshold between the fifth threshold and the sixth threshold,
Among the plurality of human body rectangles whose overlapping ratio is greater than the seventh threshold, the human body rectangle with the lower reliability is excluded.
[0021]
In another aspect of the present invention, the recording medium is
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
Estimate a human body rectangle containing the human body from the image, output the coordinates and reliability of the human body rectangle,
A fifth threshold is obtained based on the number of partial rectangles,
A sixth threshold is obtained based on the human body region estimated from the head rectangle,
using the fifth threshold and the sixth threshold to determine a seventh threshold between the fifth threshold and the sixth threshold,
Record a program that causes a computer to execute a process of excluding a human body rectangle with a lower degree of reliability from among a plurality of human body rectangles whose overlapping ratio is greater than the seventh threshold.
Effect of the invention
[0022]
According to the present invention, it is possible to provide a human body detection device that is robust against shielding.
Brief description of the drawing
[0023]
1 shows a hardware configuration of a human body detection device according to an embodiment; FIG.
2 shows the functional configuration of the human body detection device according to the first embodiment; FIG.
3 shows examples of image feature quantities and anchor boxes. FIG.
4 shows an example of a method of determining a human body candidate region; FIG.
5 shows an example of a human body candidate region; FIG.
6 is a flowchart of human body detection processing according to the first embodiment; FIG.
7 shows a functional configuration of a human body detection device according to a second embodiment; FIG.
8 is a flowchart of human body detection processing according to the second embodiment; FIG.
9 is a diagram illustrating a normal NMS and an NMS of an embodiment; FIG.
10 shows a functional configuration of a human body detection device according to a third embodiment; FIG.
11 is a flowchart of human body detection processing according to the third embodiment; FIG.
12 shows a flowchart of human body rectangle integration processing. FIG.
13 shows a functional configuration of a human body detection device according to a fourth embodiment; FIG.
14 is a flowchart of human body detection processing according to a fourth embodiment; FIG.
15 shows the functional configuration of a human body detection device according to a fifth embodiment; FIG.
16 is a flowchart of human body detection processing according to the fifth embodiment; FIG.
17 shows the functional configuration of a human body detection device according to a sixth embodiment; FIG.
18 is a flowchart of human body detection processing according to the sixth embodiment; FIG.
MODE FOR CARRYING OUT THE INVENTION
[0024]
Preferred embodiments of the present invention will be described below with reference to the drawings.
[Basic principle]
One of the challenges that shielding poses in object detection is that it is difficult to directly estimate the entire object that is shielded from the image. Therefore, among the locations dependent on the object to be detected, locations that are less likely to be shielded are estimated first, and the object to be detected is estimated only in the vicinity of those locations. For example, in the case of the human body, the head corresponds to that part. Surveillance cameras and in-vehicle cameras in the real world are often installed at high positions, and the head tends to be relatively difficult to be shielded (the head relative to the human body is called a "subordinate category"). Therefore, in the present embodiment, the subcategory is first estimated, and then the candidate area of ​​the entire object is estimated in the surrounding area. After estimating the candidate area of ​​the entire object, the target object is detected by performing processing only on that area.
[0025]
The second issue is integrated processing by NMS. In a crowded environment, there is a large overlap between people, and rectangles estimated for different people are merged into one by NMS. In order to solve this problem, in the present embodiment, rectangle position information of the subcategory is used in the integration process. Specifically, in addition to the overlapping of the entire objects, the overlapping of subcategories is taken into account in performing the integration process. This prevents actually different persons from being integrated into one person by the integration process.
[0026]
[Hardware configuration]
FIG. 1 is a block diagram showing the hardware configuration of the human body detection device according to the embodiment. As illustrated, the human body detection device 10 includes an input device 12 , a processor 13 , a memory 14 , a recording medium 15 , a database (DB) 16 and a display section 17 .
[0027]
The input device 12 inputs image data used for learning and inference of the human body detection device 10 . The image data may be moving images or still images. As the input device 12, for example, a digital camera, a smart phone with a camera, an in-vehicle camera, etc. can be used.
[0028]
The processor 13 is a computer such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and controls the entire human body detection device 10 by executing a program prepared in advance. Specifically, the processor 13 executes human body detection processing, which will be described later.
[0029]
The memory 14 is composed of ROM (Read Only Memory), RAM (Random Access Memory), and the like. The memory 14 stores various programs executed by the processor 13 . The memory 14 is also used as a working memory while the processor 13 is executing various processes.
[0030]
The recording medium 15 is a non-volatile, non-temporary recording medium such as a disk-shaped recording medium or a semiconductor memory, and is detachably attached to the human body detection device 10 . The recording medium 15 records various programs executed by the processor 13 . When the human body detection device 10 executes various processes, the programs recorded in the recording medium 15 are loaded into the memory 14 and executed by the processor 13 .
[0031]
The database 16 stores image data input from external devices including the input device 12 . Specifically, image data used for learning of the human body detection device 10 is stored. The display unit 17 is, for example, a liquid crystal display device or a projector, and displays the detection result by the human body detection device 10 . In addition to the above, the human body detection device 10 may include input devices such as a keyboard and a mouse for the user to give instructions and inputs.
[0032]
[First embodiment]
Next, a first embodiment of the present invention will be described. The first embodiment first estimates the head of a person and detects the human body based on it.
(Functional configuration)
FIG. 2 is a block diagram showing the functional configuration of the human body detection device 100 according to the first embodiment. The human body detection apparatus 100 includes an image storage unit 101 , a head rectangle estimation unit 102 , a human body candidate region estimation unit 103 , and a human body rectangle estimation unit 104 .
[0033]
The image storage unit 101 stores images to be subjected to image processing, which are input from the input device 12 . This image may be a color image or a grayscale image. Moreover, the image size does not matter.
[0034]
The head rectangle estimation unit 102 receives the image from the image storage unit 101, calculates the image feature amount, and calculates the coordinates of the bounding box of the head (hereinafter referred to as “rectangle”) (hereinafter referred to as “head rectangle coordinates”). ) and the reliability indicating head-likeness (hereinafter referred to as “head reliability”) are output. The head rectangle estimation in this embodiment is not limited to a specific head rectangle estimation process, and for example, a matching method using a sliding window or a method using machine learning such as deep learning can be used.
[0035]
An example using machine learning is described below. The head rectangle estimating unit 102 first extracts an image feature amount using a neural network. Examples of neural networks include VGG and ResNet. An image input to the neural network is reduced by multiple convolution processes, and image features are generated in the process. The image feature amount is three-dimensional information. FIG. 3 shows examples of image feature values ​​and anchor boxes. Let h×w×c be the size of the image feature amount generated when an image having an image size of H×W is input. Here, "h" and "w" are the vertical and horizontal sizes of the image feature amount, and "c" is the number of classes. A head rectangle estimation unit 102 estimates the position of the head rectangle using the extracted image feature amount and the anchor box. Anchor boxes are a group of rectangles with a predefined aspect ratio and number, and are infinitely scattered in the image. Actually, a fixed number of anchor boxes are arranged at each point in the spatial direction of the image feature amount. The spatial size of the image feature amount of size h×w×c is h×w. Assuming that four anchor boxes are arranged at each point, a total of h×w×4 anchor boxes are arranged.
[0036]
Next, the head rectangle estimation unit 102 uses the image features of each anchor box to output head rectangle coordinates and head reliability. The image feature of each anchor box is the information amount of 1×1×c in which the anchor box is arranged. Convolution processing is performed again on the information amount of 1×1×c to estimate the head rectangular coordinates and the head reliability. The estimation here is made possible by the neural network learning rectangular position regression and category classification. Specifically, the position and category of the head are given as correct data, the error between the correct data and the estimation result is calculated by a loss function, and the neural network is corrected so that the error becomes small. This process is repeated, and learning ends when the specified number of times is reached. The head rectangle estimation unit 102 estimates the head rectangle coordinates and the head reliability from the image using the learned neural network thus obtained, and supplies them to the human body candidate region estimation unit 103 .
[0037]
The human body candidate region estimation unit 103 receives the head rectangle coordinates and head reliability from the head rectangle estimation unit 102, and estimates the human body candidate region. A “human body candidate region” is a region in an image in which a human body is considered to exist. In general, the human body of the same person often exists in the image for the detected head. In addition, given the physical characteristics of humans, it is possible to estimate the region of the human body to some extent based on the position of the head. For example, prior knowledge is obtained that the body often exists below the neck with respect to the head. FIG. 4 shows an example of a method for determining a human body candidate region. In this example, a mechanical model is created based on prior knowledge that there is a body under the head and that the human body is often a vertically long rectangle, and the human body candidate region is calculated from the head position. Specifically, in this example, the width of the human body is 3 times the width of the head, and the height of the human body is 2.5 times the width of the human body. Assume that the head rectangle coordinates given from the head rectangle estimation unit 102 include the coordinates (x, y) of the upper left point of the head rectangle and the width and height (w H, h H) of the head rectangle. Then, the human body candidate region estimation unit 103 determines the coordinates of the upper left point of the human body candidate region as (x−w H, y) and the width and height as (3w H, 3w H/0.4). Then, as shown in FIG. 5, the human body candidate region estimation unit 103 extracts the feature quantity corresponding to the human body candidate region from the feature quantity of the entire image, and obtains the feature quantity of the human body candidate region.
[0038]
The human body rectangle estimation unit 104 receives the human body candidate region from the human body candidate region estimation unit 103, and calculates the coordinates of the human body rectangle (hereinafter referred to as “human body rectangle coordinates”) and the degree of reliability representing human body likeness (hereinafter referred to as “human body ) is output. The input of the human body rectangle estimating unit 104 is information obtained based on the human body candidate region. A cut piece is also acceptable. The human body rectangle estimation process in this embodiment is not limited to a specific human body position estimation process. There are other methods. specifically In the method using machine learning, the human body rectangle estimation unit 104 inputs the feature amount extracted by the human body candidate region estimation unit 103 to a CNN (Convolutional Neural Network), and outputs the human body rectangle coordinates and the human body reliability. In the same way as the head rectangle estimation unit 102, the neural network learns the regression of the human body candidate region and the category classification problem, and estimates the human body rectangle coordinates and the human body reliability using the trained neural network. done.
[0039]
(Human body detection processing)
Next, human body detection processing according to the first embodiment will be described. FIG. 6 is a flowchart of human body detection processing according to the first embodiment. This processing is realized by the processor 13 shown in FIG. 1 executing a program prepared in advance. First, the head rectangle estimation unit 102 receives an input image from the image storage unit 101, and estimates a head rectangle of a person existing in the image (step S11). The head rectangle coordinates and the head reliability estimated by the head rectangle estimation unit 102 are sent to the human body candidate region estimation unit 103 . Next, the human body candidate region estimation unit 103 estimates a human body candidate region using the head rectangle coordinates obtained from the head rectangle estimation unit 102 (step S12). Then, the human body rectangle estimation unit 104 estimates the human body rectangle using the human body candidate region obtained from the human body candidate region estimation unit 103, and outputs the human body rectangle coordinates and the human body reliability (step S13).
[0040]
(effect)
In the conventional technology, candidate areas of the human body to be detected are estimated and used to estimate the position of the human body rectangle. However, in crowded situations, shielding by objects and people is likely to occur. It is difficult to estimate the human body candidate region in a state where the human body region is lost due to shielding. In this regard, in the first embodiment, instead of estimating the human body candidate region directly, the head, which is a place where occlusion is unlikely to occur, is detected first, and the human body candidate region is estimated therefrom. , and can reduce undetected body rectangles.
[0041]
[Second embodiment]
Next, a second embodiment will be described. In the first embodiment, the head rectangle is estimated from the image, and the human body candidate region is estimated based on the position of the head rectangle. In contrast, in the second embodiment, the center position of the human body is estimated in addition to the head rectangle, and the human body candidate region is estimated based on the position of the head rectangle and the center position of the human body.
[0042]
(Functional configuration)
FIG. 7 is a block diagram showing the functional configuration of the human body detection device 200 according to the second embodiment. The human body detection apparatus 200 includes an image storage unit 201 , a head rectangle/body center estimation unit 202 , a human body candidate region estimation unit 203 , and a human body rectangle estimation unit 204 . The hardware configuration of the human body detection device 200 of the second embodiment is the same as that shown in FIG.
[0043]
A head rectangle/body center estimating unit 202 receives an image from the image storage unit 201, calculates image feature amounts, estimates head rectangle coordinates and head reliability, and calculates the center position of the human body to which the head belongs. to estimate Here, the head rectangle estimation process can use the same method as in the first embodiment. In addition, estimation of the center position of the human body is made possible by providing the center position of the human body as correct data and allowing the neural network to learn the regression problem. More specifically, the neural network is trained by providing information in which the same person's head rectangle and body rectangle are paired as correct data. Then, the image from the image storage unit 201 is input to the trained neural network to estimate the center position of the human body.
[0044]
The human body candidate region estimating unit 203 receives the head rectangle coordinates, the head reliability, and the center position of the human body to which the head belongs from the head rectangle/body center estimating unit 202, and determines whether the human body of the person having the head exists. Estimate the human body candidate region that is likely to occur. Specifically, assuming that the head rectangle coordinates given from the head rectangle/body center estimating unit 202 include the width and height (w H, h H) of the head rectangle, the human body candidate region estimating unit 203 A rectangle whose width and height are (3w H, 3w H/0.4) and whose center is the center position of the human body received from the head rectangle/body center estimating unit 202 is estimated as the human body candidate region. Then, the human body candidate region estimation unit 203 extracts the feature quantity corresponding to the human body candidate region from the feature quantity of the entire image, thereby outputting the feature quantity of the human body candidate region.
[0045]
The human body rectangle estimation unit 204 is basically the same as the human body rectangle estimation unit 104 of the first embodiment. Print degrees.
[0046]
(Human body detection processing)
Next, human body detection processing according to the second embodiment will be described. FIG. 8 is a flowchart of human body detection processing according to the second embodiment. This processing is realized by the processor 13 shown in FIG. 1 executing a program prepared in advance. First, the head rectangle/body center estimating unit 202 receives an input image from the image storage unit 201, and estimates the human head rectangle and the center position of the human body in the image (step S21). The head rectangle coordinates and head reliability estimated by the head rectangle/body center estimation unit 202 and the center position of the human body are sent to the human body candidate region estimation unit 203 . Next, the human body candidate region estimating unit 203 estimates a human body candidate region using the head rectangle coordinates and the center position of the human body obtained from the head rectangle/body center estimating unit 202 (step S22). Then, the human body rectangle estimation unit 204 estimates the human body rectangle using the human body candidate region obtained from the human body candidate region estimation unit 203 (step S23). Thus, the human body rectangle coordinates and the human body reliability estimated by the human body rectangle estimation unit 204 are output.
[0047]
(effect)
In the first embodiment, since the human body candidate region candidates are set mechanically, there is a possibility that deviation from the actual human body position will occur. In this regard, in the second embodiment, since the central position of the human body is determined by learning, it is robust against the displacement of the human body.
[0048]
[Third embodiment]
Next, a third embodiment will be described. The third embodiment performs processing for integrating a plurality of human body rectangles detected from an image. Many object detection methods using deep learning use a reference rectangle called an “anchor box” when learning and estimating the rectangular position of an object. Anchor boxes have various sizes and aspect ratios and are innumerably scattered in the image. When estimating the position of an object, the neural network obtains an image feature amount near the area of ​​each anchor box, and estimates the bounding box position of the object and the object category. When learning the bounding box position and object category of an object, calculate the difference between the estimated bounding box position and object category and the correct bounding box position and object category of the teacher data that overlaps the anchor box the most, Iteratively adjust the neural network to make them smaller.
[0049]
An object detector using anchor boxes estimates the same number of objects as the number of set anchor boxes. That is, the position coordinates of the bounding box of the object, the category of the object, and the reliability of the category are output in the same number as the number of anchor boxes. Therefore, a plurality of bounding box estimation results may be obtained for the same object in the image, and these must be integrated into one. At that time, NMS is used as an integration technique.
[0050]
FIG. 9(A) schematically shows an example of a normal NMS. The NMS calculates the ratio of overlap between the two bounding boxes representing the human body region, and only when the ratio exceeds a set threshold (first threshold), the bounding box with the lower reliability of the two. discard the box. In the example of FIG. 9A, two bounding boxes with confidence levels of 0.7 and 0.9 are detected. The object detector calculates the overlap ratio of the two bounding boxes and compares it with a predetermined threshold. If the overlap ratio is greater than the threshold, the two bounding boxes are considered to be the detection result of the same object. The one with the higher (reliability of 0.9) is left. By performing this process for all combinations of bounding boxes of the same category, one bounding box is finally determined for one object. In the following description, the threshold for the human body region or "human body threshold" is an example of the first threshold. Also, the threshold for the head or "head threshold" is an example of the second threshold.
[0051]
However, for example, in situations where people are crowded, such as public facilities and event venues, people overlap, and occlusion makes it difficult to estimate the bounding box position of the whole body. Therefore, the NMS process determines that different objects are actually the same object, and the less reliable bounding box is rejected. As a result, an object that has been detected once may be rejected by NMS processing and become undetected.
[0052]
Therefore, in the third embodiment, in addition to the overlapping ratio of the bounding box indicating the human body area, the overlapping ratio of the bounding box indicating the head area is also taken into account to perform the integration process. FIG. 9B schematically shows integration processing according to the second embodiment. Suppose that two bounding boxes representing human body regions and two bounding boxes representing their heads are obtained as shown in the figure. In this case, the human body detection apparatus calculates the ratio of overlap between bounding boxes indicating the human body region (hereinafter referred to as “human body weight ratio”) and compares it with a predetermined threshold value for the human body region. In addition to this, the human body detection apparatus calculates an overlapping ratio between bounding boxes indicating the head area (hereinafter referred to as “head overlapping ratio”), and uses this as a predetermined head threshold. compare. If the human body weight ratio is greater than the threshold for the human body region (first threshold) and the head overlap ratio is greater than the threshold for the head (second threshold), the two detected persons are the same person. possible, the human body detection apparatus rejects the bounding box with the lower confidence of the human body region. On the other hand, even if the human body weight ratio is greater than the threshold for the human body region, if the head overlap ratio is smaller than the threshold for the head region, the human body regions simply overlap due to occlusion, and the two detected persons are different. , the human body detector does not merge these two bounding boxes. For example, in a scene where people overlap in front and behind, and the person in the back is sticking out from the shoulder of the person in front, the overlap between the human body areas is large, but the overlap of the head area is small, so integration is not performed. . By doing so, even when the overlapping of human body parts is large due to congestion or the like, different people can be detected correctly.
[0053]
(Functional configuration)
FIG. 10 is a block diagram showing the functional configuration of the human body detection device 300 according to the third embodiment. The human body detection apparatus 300 includes an image storage unit 301 , a head rectangle/body center estimation unit 302 , a human body candidate region estimation unit 303 , a human body rectangle estimation unit 304 , and a human body rectangle integration unit 305 . The hardware configuration of the human body detection device 300 of the third embodiment is the same as that shown in FIG.
[0054]
In the third embodiment, the image storage unit 301, the head rectangle/body center estimation unit 302, the human body candidate region estimation unit 303, and the human body rectangle estimation unit 304 are the same as the image storage unit 201, the head rectangle/body center estimation unit 302, and the head rectangle/body center estimation unit 304 in the second embodiment. Since it is basically the same as the center estimating section 202, the human body candidate region estimating section 203, and the human body rectangle estimating section 204, the description thereof will be omitted.
[0055]
The human body rectangle integration unit 305 performs the integration process described above. Specifically, the human body rectangle integration unit 305 acquires the head rectangle coordinates and the head reliability from the head rectangle/body center estimation unit 302 , and acquires the human body rectangle coordinates and the human body reliability from the human body rectangle estimation unit 304 . . Then, as described with reference to FIG. 9B, the human body rectangle integration unit 305 determines that the overlapping ratio of both the human body rectangle and the head rectangle is high. Reject the body rectangle with the lower confidence only if both are greater than or equal to the threshold. Then, the human body rectangle integration unit 305 outputs the human body rectangle coordinates and the human body reliability after the integration process. As a result, even when many parts of the human body are hidden on the image due to shielding, different people can be detected correctly.
[0056]
(Human body detection processing)
Next, human body detection processing according to the third embodiment will be described. FIG. 11 is a flowchart of human body detection processing according to the third embodiment. This processing is realized by the processor 13 shown in FIG. 1 executing a program prepared in advance. Steps S31 to S33 in the third embodiment are the same as steps S21 to S23 in the second embodiment, so description thereof will be omitted.
[0057]
When the human body rectangle is estimated in step S33, the human body rectangle integration unit 305 receives the head rectangle and the human body rectangle from the head rectangle/body center estimation unit 302 and the human body rectangle estimation unit 304, respectively. Human body rectangle integration processing is performed so that the estimation result of . is not obtained (step S34). The head rectangle estimated by the head rectangle/body center estimating unit 302 and the human body rectangle estimated by the human body rectangle estimating unit 504 belong to the same person and form a one-to-one pair.
[0058]
FIG. 12 shows a flowchart of human body rectangle integration processing. The human body rectangle integration unit 305 receives a pair of the human body rectangle and the head rectangle (hereinafter also referred to as a “head/human pair”) estimated from the head rectangle/body center estimation unit 302 and the human body rectangle estimation unit 504 . Multiple given. The human body rectangle integration unit 305 first adds them to the unprocessed list (step S301). Next, the human body rectangle integration unit 305 selects a pair with the highest human body reliability from the unprocessed list, puts it in the processed list, and deletes it from the unprocessed list (step S302).
[0059]
Next, the human body rectangle integration unit 305 selects one pair in the unprocessed list (step S303), and divides the pair with the highest human body reliability from the pair selected in step S303. The overlapping ratios of the part rectangles and the human body rectangles are calculated (step S304). IoU (Intersection over Union) is used as an index for evaluating the overlapping ratio. IoU is given by the following formula. Assuming that the two rectangles are box 1 and box 2, respectively, the higher the IoU value, the greater the overlap between the two rectangles. Note that the numerator of the formula below indicates the area of ​​the overlapping portion of the two rectangles, and the denominator indicates the area of ​​the combined area of ​​the two rectangles.
[0060]
[Number 1]

[0061]
Next, the human body rectangle integration unit 305 determines whether or not the overlapping ratio of the human body rectangles is greater than a predetermined threshold value for human body rectangles (step S305). If the overlapping ratio of the human body rectangles is greater than a predetermined first threshold for human body rectangles (denoted as “human body threshold” in FIG. 12) (step S305: Yes), the human body rectangle integration unit 305 It is determined whether or not the overlap ratio is greater than a predetermined head threshold (step S306). If the overlapping ratio of the head rectangles is greater than a predetermined second threshold for the head (described as “head threshold” in FIG. 12) (step S306: Yes), the head/human body pair is the same person. , the human body rectangle integration unit 305 deletes the pair with the lower head reliability from the unprocessed list (step S307). The reason why the pair to be deleted is determined based on the reliability of the head is that in an environment where a part of the human body is shielded, the reliability of the head is considered to be more reliable than the reliability of the human body. be. If the overlapping ratio of the human body rectangle is less than or equal to the threshold for the human body rectangle (step S305: No) and if the overlapping ratio of the head rectangle is less than or equal to the threshold for the head (step S306: No), the process proceeds to step Returning to S303, another pair in the unprocessed list is processed.
[0062]
Next, the human body rectangle integration unit 305 determines whether or not all pairs in the unprocessed list have been processed (step S308). If all pairs have not been processed (step S308: No), the process returns to step S303, and steps S303 to S307 are performed for another pair in the unprocessed list. On the other hand, if all pairs in the unprocessed list have been processed (step S308: Yes), the process returns to step S301. When the integration process is performed for all the pairs in the unprocessed list (step S301: No), the human body rectangle integration unit 305 outputs the human body rectangle coordinates and human body reliability after integration (step S309). Then the process ends.
[0063]
(effect)
In the prior art, only the information of the human body rectangles was used to evaluate the overlapping of the human body rectangles, so in a crowded environment where people overlap, the rectangles of people with strong overlaps are merged. In this regard, in the third embodiment, the integration process is performed in consideration of not only the degree of overlapping of the rectangles of the human body but also the rectangles of the head, so that it is possible to prevent the same person from being integrated.
[0064]
[Fourth embodiment]
Next, a fourth embodiment will be described. The fourth embodiment integrates a plurality of human body rectangles detected from an image in the same manner as in the third embodiment. However, in the third embodiment, the human body candidate region is estimated based on the head rectangle estimated from the image, so if the head cannot be detected from the image, the human body cannot be detected either. . Therefore, in the fourth embodiment, a human body candidate region directly estimated from an image and a human body candidate region obtained based on the estimation result of the head rectangle are used in combination. This makes it possible to detect a larger number of human body candidate regions than in the third embodiment.
[0065]
(Functional configuration)
FIG. 13 is a block diagram showing the functional configuration of the human body detection device 400 according to the fourth embodiment. The human body detection apparatus 400 includes an image storage unit 401, a head rectangle/body center estimation unit 402, a human body candidate region estimation unit 403, a human body rectangle estimation unit 404, a human body candidate region estimation unit 405, and a head rectangle estimation unit. A human body rectangle estimation unit 407 and a human body rectangle integration unit 408 are provided. The hardware configuration of the human body detection device 400 of the fourth embodiment is the same as that shown in FIG.
[0066]
The image storage unit 401 stores images to be subjected to image processing in this embodiment. A head rectangle/body center estimating unit 402 receives an image from the image storage unit 401, calculates the image feature amount, and calculates the head rectangle coordinates, the head reliability, and the center position of the human body to which the head belongs. Output to candidate area estimation section 403 . The head rectangle/body center estimation unit 402 also outputs the estimated head rectangle coordinates and head reliability to the human body rectangle integration unit 408 .
[0067]
Based on the head rectangle coordinates received from the head rectangle/body center estimation unit 402, the head reliability level, and the center position of the human body, the human body candidate region estimation unit 403 determines whether the human body of the person having the head exists. A human body candidate region that is likely to occur is output. Human body rectangle estimation section 404 estimates a human body rectangle based on the human body candidate area output from human body candidate area estimation section 403 , and outputs the human body rectangle coordinates and the human body reliability to human body rectangle integration section 408 . In this manner, the human body rectangle integration unit 408 receives the head rectangle coordinates and the head reliability from the head rectangle/body center estimation unit 402 and the human body rectangle coordinates and the human body rectangle reliability from the human body rectangle estimation unit 404 . be. That is, the human body rectangle integrating unit 408 first estimates the head from the image, and inputs the head/human body pair estimated based on the head.
[0068]
On the other hand, the human body candidate region estimating unit 405 receives the image from the image storage unit 801, calculates the image feature amount, and estimates the human body candidate region and the center position of the head belonging to the human body. Then, human body candidate region estimation section 405 inputs the human body candidate region and the center position of the head to head rectangle estimation section 406 and also inputs the human body candidate region to human body rectangle estimation section 407 .
[0069]
The head rectangle estimation unit 406 estimates head rectangle coordinates and head reliability based on the human body candidate region and the center position of the head belonging to the human body, and outputs them to the human body rectangle integration unit 408 . Since the human body candidate region directly estimated from the image by the human body candidate region estimation unit 405 does not have a paired head rectangle, it is necessary to estimate the head rectangle from the human body candidate region. Therefore, the head rectangle estimation unit 406 generates a head rectangle from the center coordinates of the head obtained from the human body candidate region estimation unit 405 . Any model may be used to generate the head rectangle. For example, a square having a width one-third the width of the human body candidate region may be used as the head rectangle.
[0070]
The human body rectangle estimation unit 407 receives the human body candidate region from the human body candidate region estimation unit 405 , estimates the human body rectangle coordinates and the human body reliability, and outputs them to the human body rectangle integration unit 408 . Thus, the human body rectangle integration unit 408 receives the head rectangle coordinates and the head reliability from the head rectangle estimation unit 406 and the human body rectangle coordinates and the human rectangle reliability from the human body rectangle estimation unit 407 . That is, the human body rectangle integration unit 408 estimates the human body directly from the image, and inputs the head/human body pair obtained based on the human body.
[0071]
As described above, the human body rectangle integration unit 408 uses the head/human body pair obtained by first estimating the head from the image and the head/human body pair obtained by estimating the human body from the image, Integrating processing similar to that of the third embodiment is performed, and the human body rectangular coordinates and the human body reliability are output.
[0072]
(Human body detection processing)
Next, human body detection processing according to the fourth embodiment will be described. FIG. 14 is a flowchart of human body detection processing according to the fourth embodiment. This processing is realized by the processor 13 shown in FIG. 1 executing a program prepared in advance. First, the head rectangle/body center estimation unit 402 estimates the head rectangle from the image stored in the image storage unit 401, and outputs the head rectangle coordinates, the head reliability, and the center position of the human body (step S41). ). The human body candidate region estimation unit 403 estimates a human body candidate region from the head rectangular coordinates and the center position of the human body (step S42). The human body rectangle estimation unit 404 estimates the human body rectangle coordinates and the human body reliability from the human body candidate region (step S43).
[0073]
Also, the human body candidate region estimation unit 405 estimates the human body candidate region and the center position of the head from the image stored in the image storage unit 401 (step S44). The head rectangle estimation unit 406 estimates a head rectangle from the human body candidate region and the center position of the head (step S45). The human body rectangle estimation unit 407 estimates a human body rectangle from the human body candidate region (step S46). The order of steps S41 to S43 and steps S44 to S46 may be reversed, or both may be performed in parallel.
[0074]
Then, the human body rectangle integration unit 408 generates a pair of the head rectangle obtained in step S41 and the human body rectangle obtained in step S43, and the head rectangle obtained in step S45 and the human body rectangle obtained in step S46. Human body rectangle integration processing is performed for the pair (step S47). Note that the integration process itself is the same as in the third embodiment.
[0075]
(effect)
In the third embodiment, the human body candidate region is estimated from the head rectangle, so if the head cannot be detected, the human body cannot be detected either. In this respect, in the fourth embodiment, since the human body candidate region directly estimated from the image and the human body candidate region obtained from the estimation result of the head rectangle are used in combination, the human body may not be detected as compared with the third embodiment. sex can be reduced.
[0076]
[Fifth embodiment]
Next, a fifth embodiment will be described. In the third and fourth embodiments, in order to perform the human body rectangle integration process, it is necessary to prepare correct data pairs of the same person's human body and head during the learning process. In contrast, the fifth embodiment facilitates preparation of learning data by independently estimating the human body and the head.
[0077]
(Functional configuration)
FIG. 15 is a block diagram showing the functional configuration of a human body detection device 500 according to the fifth embodiment. The human body detection apparatus 500 includes an image storage unit 501, a head rectangle/body center estimation unit 502, a human body region estimation unit 503, a human body rectangle estimation unit 504, and a human body rectangle integration unit.505; The hardware configuration of the human body detection device 500 of the fifth embodiment is the same as that shown in FIG.
[0078]
The image storage unit 501 stores images to be subjected to image processing in this embodiment. A head rectangle/body center estimating unit 502 receives an image from the image storage unit 501, calculates the image feature amount, and calculates the head rectangle coordinates, the head reliability, and the center position of the human body to which the head belongs. Output to region estimation section 503 . Based on the head rectangle coordinates received from the head rectangle/body center estimation unit 502, the head reliability level, and the center position of the human body, the human body region estimation unit 503 determines whether the human body of the person having the head exists. A human body region that is assumed to be a human body region is estimated as a rectangle, and output to the human body rectangle integration unit 505 . On the other hand, the human body rectangle estimation unit 504 receives the image from the image storage unit 501 , calculates the image feature amount, estimates the human body rectangle coordinates and the human body reliability, and outputs them to the human body rectangle integration unit 505 .
[0079]
The human body rectangle integration unit 505 performs integration processing using the human body region output by the human body region estimation unit 503 and the human body rectangle and human body reliability output by the human body rectangle estimation unit 504 . Specifically, the human body rectangle integration unit 505 first performs integration processing on the human body rectangles input from the human body rectangle estimation unit 504 using the normal NMS shown in FIG. For human body rectangles larger than the threshold (third threshold), those with lower reliability are determined as integration candidates. Next, the human body rectangle integration unit 505 does not reject the human body rectangles having a large overlap ratio with the human body region generated by the human body region estimation unit 503 among the human body rectangles determined as integration candidates, and calculates the overlap ratio with the human body region. Reject the body rectangles with small . Then, the human body rectangle integration unit 505 outputs the human body rectangle coordinates and the human body reliability for the remaining human body rectangles. In the following description of this embodiment, the threshold is an example of the third threshold. The third threshold may be different from the first and second thresholds in the third embodiment.
[0080]
(Human body detection processing)
Next, human body detection processing according to the fifth embodiment will be described. FIG. 16 is a flowchart of human body detection processing according to the fifth embodiment. This processing is realized by the processor 13 shown in FIG. 1 executing a program prepared in advance. First, the head rectangle/body center estimation unit 502 estimates the head rectangle from the image stored in the image storage unit 501, and outputs the head rectangle coordinates, the head reliability, and the center position of the human body (step S51). ). The human body region estimation unit 503 estimates the human body region from the head rectangular coordinates and the center position of the human body (step S52). The human body rectangle estimation unit 504 estimates the human body rectangle from the image stored in the image storage unit 501, and estimates the human body rectangle coordinates and the human body reliability (step S53). Then, the human body rectangle integration unit 505 uses the human body region input from the human body region estimation unit 503 and the human body rectangle and the human body reliability input from the human body rectangle estimation unit 504 to perform integration processing by the method described above. , and output the human body rectangle coordinates and the human body reliability for the integrated human body rectangle (step S54). Then the process ends.
[0081]
(effect)
In the fifth embodiment, a pair of correct data of the same person's human body and head was required in the learning process. In this regard, in the present embodiment, learning data for the human body and the head may be prepared separately, so preparation of the learning data is facilitated.
[Sixth embodiment]
Next, a sixth embodiment will be described. In the above-described third to fifth embodiments, the ratio of overlap between human body rectangles indicated by the IoU value is compared with a threshold in the human body rectangle integration process (NMS). using the value. However, if the threshold is a fixed value, non-detection or erroneous detection may occur depending on the image. Therefore, in the sixth embodiment, the threshold is dynamically determined for each image. Note that the threshold to be compared with the IoU value in the integration process is hereinafter also referred to as the "IoU threshold". Basically, the higher the IoU threshold, the more rectangles are output and the more false positives. Also, the lower the IoU threshold, the fewer the number of human body rectangles to be output and the more undetected. Note that when the IoU threshold is set to "1", the number of human body rectangles excluded by the integration process is "0".
[0082]
Specifically, in the sixth embodiment, information on the head rectangle is used to estimate the number of people in the target image and the degree of human overlap (congestion), and determine the IoU threshold for each image. First, the estimation of the number of humans will be explained. As mentioned earlier, even in a crowded situation with a large degree of overlap of the human body, the degree of overlap of the head is relatively small. Therefore, NMS processing is performed on the head rectangles estimated from the image, and the number of obtained head rectangles is assumed to be the number of persons included in the image. When the human body rectangles are integrated while the IoU threshold value is decreased from "1", and the number of output human body rectangles matches the number of people included in the image, that is, the estimated number of head rectangles. is the first IoU threshold. The first IoU threshold in this embodiment corresponds to the upper limit of the range of suitable IoU thresholds and corresponds to the fifth threshold.
[0083]
Next, we will explain how to estimate the degree of human overlap. The degree of overlapping of people included in the image differs from image to image. Therefore, the human body region is estimated from the head rectangle included in the image, and the IoU value when the degree of overlap between the estimated human body regions is the largest is set as the second IoU threshold. Since the accuracy of the human body region estimated from the head rectangle is relatively high, the second IoU threshold is considered to correspond to the maximum degree of overlap in the image. Therefore, when the IoU value of two human body rectangles is higher than the second IoU value in the integration process, it is considered that the two human body rectangles should be integrated as the same person. From this point of view, the second IoU threshold in this embodiment corresponds to the lower limit of the range of appropriate IoU thresholds and corresponds to the sixth threshold.
[0084]
Then, a value between the first IoU threshold and the second IoU threshold is determined as a third IoU threshold suitable for the image. Note that the third IoU threshold in this embodiment corresponds to the seventh threshold. As described above, in the sixth embodiment, the third IoU threshold is determined for each image, and the human body rectangles are integrated using this threshold.
[0085]
(Functional configuration)
FIG. 17 is a block diagram showing the functional configuration of a human body detection device 600 according to the sixth embodiment. The human body detection apparatus 600 includes an image storage unit 601, a head rectangle estimation unit 602, a human body rectangle estimation unit 603, a threshold determination unit 604, a human body area estimation unit 605, threshold determination units 606 and 607, and a human body rectangle. and an integration unit 608 . The hardware configuration of the human body detection device 600 of the sixth embodiment is the same as that shown in FIG.
[0086]
The image storage unit 601 stores images to be subjected to image processing. The head rectangle estimation unit 602 receives the image from the image storage unit 601 , calculates the image feature amount, and outputs the head rectangle coordinates and the head reliability to the threshold determination unit 604 and the human body area estimation unit 605 . The human body rectangle estimation unit 603 receives the image from the image storage unit 601 , calculates the image feature amount, estimates the human body rectangle coordinates and the human body reliability, and outputs them to the threshold determination unit 604 and the human body rectangle integration unit 608 .
[0087]
The threshold determination unit 604 determines the first IoU threshold using the head rectangle received from the head rectangle estimation unit 602 and the human body rectangle received from the human body rectangle estimation unit 603 . Note that the received head rectangle and human body rectangle do not correspond to the same person. First, the threshold determination unit 604 performs NMS processing on the received head rectangle to determine the head rectangle. This NMS process is the normal NMS process shown in FIG. 9A, and its IoU threshold is a fixed value, generally 0.5 to 0.7. Then, the threshold determining unit 604 assumes that the number of head rectangles obtained by the NMS processing is the number of body rectangles. Next, the threshold determination unit 604 performs NMS processing on the received human body rectangle. At this time, the threshold determination unit 604 decreases the IoU threshold from "1" to "0", and the number of human body rectangles output by the NMS process matches the number of head rectangles obtained previously. The IoU threshold at this time is determined as the first IoU threshold. Threshold determination section 604 then outputs the first IoU threshold to threshold determination section 607 .
[0088]
The human body region estimation unit 605 uses the head rectangle coordinates and the head reliability received from the head rectangle estimation unit 602 to estimate the human body region, in which the human body of the person having the head is likely to exist, as a rectangle. , to the threshold determination unit 606 . As described above, the human body region may be generated using a mechanical generation model or a model using machine learning. The threshold determination unit 606 determines the IoU value of the input human body regions that overlap the most with each other as the second IoU threshold. Threshold determination section 606 then outputs the second IoU threshold to threshold determination section 607 .
[0089]
The threshold determination unit 607 determines the third IoU threshold using the first IoU threshold and the second IoU threshold. Here, the threshold determination unit 607 determines a value within the range of the first IoU threshold and the second IoU threshold as the third IoU threshold. For example, the third IoU threshold may be an intermediate value between the first IoU threshold and the second IoU threshold, or a value close to either of them. The threshold determination unit 607 outputs the determined third IoU threshold to the human body rectangle integration unit 608 .
[0090]
The human body rectangle integration unit 608 uses the third IoU threshold determined by the threshold determination unit 607 to perform integration processing of the human body rectangles output by the human body rectangle estimation unit 603 . Specifically, the human body rectangle integration unit 608 performs integration processing on the human body rectangles output by the human body rectangle estimation unit 603, and excludes the human body rectangles with the overlapping ratio greater than the third IoU threshold, those with lower reliability. Then, the human body rectangle integration unit 608 outputs the human body rectangle coordinates and the human body reliability of the human body rectangles remaining after the integration process.
[0091]
(Human body detection processing)
Next, human body detection processing according to the sixth embodiment will be described. FIG. 18 is a flowchart of human body detection processing according to the fifth embodiment. This processing is realized by the processor 13 shown in FIG. 1 executing a program prepared in advance. First, the head rectangle estimation unit 602 estimates a head rectangle from the image stored in the image storage unit 601, and outputs head rectangle coordinates and head reliability (step S61). The human body rectangle estimation unit 603 estimates the human body rectangle from the image stored in the image storage unit 601, and estimates the human body rectangle coordinates and the human body reliability (step S62). The order of steps S61 and S62 may be reversed, or may be performed in parallel. Next, the threshold determining unit 604 determines a first IoU threshold from the head rectangle and the human body rectangle (step S63). The first IoU threshold illustrated in FIG. 18 is an example of a fifth threshold.
[0092]
Next, the human body area estimation unit 605 estimates the human body area from the head rectangular coordinates (step S64). Next, the threshold determination unit 606 determines a second IoU threshold from the human body region (step S65). The second IoU threshold illustrated in FIG. 18 is an example of the sixth threshold. Next, the threshold determining unit 607 determines a third IoU threshold using the first IoU threshold and the second IoU threshold (step S66). The third IoU threshold illustrated in FIG. 18 is an example of the seventh threshold. Then, the human body rectangle integration unit 608 performs integration processing of the human body rectangles estimated by the human body rectangle estimation unit 603 using the third IoU threshold determined by the threshold determination unit 607, and calculates the human body rectangle coordinates of the integrated human body rectangle. And the human body reliability is output (step S67). Then the process ends.
[0093]
(effect)
In the human body rectangle integration processing in the third to fifth embodiments described above, a fixed value is used as the IoU threshold, which must be manually determined by a person. However, in images captured in a real environment, the degree of crowding of people varies, and it is not preferable to fix the IoU threshold used in the process of integrating the rectangles of the human body. In this regard, in the sixth embodiment, the degree of congestion (the number of people and the degree of overlapping of human bodies) is estimated using the head of each image, and the IoU threshold value that matches the scene is estimated. Since the value is used, it is possible to reduce non-detection and false detection.
[0094]
[Modification]
In the above embodiment, the human body and head are used, but a specific part of the human body other than the head may be used. For example, a foot may be used as a particular part of the human body. The present invention can also be applied to categories having a positional relationship, such as cars and tires, and faces and mouths.
[0095]
Some or all of the above embodiments can also be described as the following additional remarks, but are not limited to the following.
[0096]
(Appendix 1)
a partial rectangle estimation unit that estimates a specific part of the human body from the image and outputs the coordinates and reliability of the partial rectangle containing the part;
a human body candidate region estimating unit that estimates and outputs a human body candidate region, which is a region in which a human body corresponding to the partial rectangle is predicted to exist, based on the coordinates of the partial rectangle;
a human body rectangle estimation unit that estimates a human body rectangle including a human body based on the human body candidate region and outputs the coordinates and reliability of the human body rectangle;
human body detection device.
[0097]
(Appendix 2)
a human body center estimation unit for estimating the center position of the human body corresponding to the part estimated by the partial rectangle estimation unit,
The human body detection device according to Supplementary Note 1, wherein the human body candidate region estimation unit estimates the human body candidate region based on the coordinates of the partial rectangle and the center position of the human body.
[0098]
(Appendix 3)
The human body detection apparatus according to appendix 2, wherein the human body candidate region estimation unit estimates a rectangular region centered on the center position of the human body and having a predetermined aspect ratio as the human body candidate region.
[0099]
(Appendix 4)
For a plurality of pairs of partial rectangles and human body rectangles that correspond to each other, the coordinates and reliability of the partial rectangles and the coordinates and reliability of the human body rectangles are obtained, and the overlapping ratio of the partial rectangles and the ratio of the human body rectangles to each other 4. The human body detection device according to any one of appendices 1 to 3, comprising a human body integration unit that integrates overlapping human body rectangles based on an overlapping ratio.
[0100]
(Appendix 5)
The human body integration unit determines that the reliability of the partial rectangle is low when the overlapping ratio of the human body rectangles is larger than a first threshold and the overlapping ratio of the partial rectangles is larger than a second threshold. The human body detection device according to appendix 4, which excludes one human body rectangle.
[0101]
(Appendix 6)
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
　Based on the coordinates of the partial rectangle, estimate and output a human body candidate area, which is an area in which a human body corresponding to the part is predicted to exist,
A human body detection method for estimating a human body rectangle containing a human body based on the human body candidate region and outputting the coordinates and reliability of the human body rectangle.
[0102]
(Appendix 7)
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
　Based on the coordinates of the partial rectangle, estimate and output a human body candidate area, which is an area in which a human body corresponding to the part is predicted to exist,
A recording medium recording a program for causing a computer to execute a process of estimating a human body rectangle containing a human body based on the human body candidate region and outputting the coordinates and reliability of the human body rectangle.
[0103]
(Appendix 8)
a first partial rectangle estimation unit that estimates a specific part of the human body from the image and outputs the coordinates and reliability of a first partial rectangle containing the part;
a first human body candidate region estimation unit for estimating and outputting a first human body candidate region, which is a region in which a human body corresponding to the first partial rectangle is predicted to exist, based on the coordinates of the first partial rectangle; ,
a first human body rectangle estimation unit for estimating a first human body rectangle including a human body based on the first human body candidate region and outputting the coordinates and reliability of the first human body rectangle;
a second human body candidate region estimation unit that estimates and outputs a second human body candidate region from the image;
Second partial rectangle estimation for estimating a specific part corresponding to the human body based on the second human body candidate region and outputting coordinates and reliability of a second partial rectangle containing the part Department and
a second human body rectangle estimation unit for estimating a second human body rectangle including the human body based on the second human body candidate region and outputting the coordinates and reliability of the second human body rectangle;
For a plurality of pairs of partial rectangles and human body rectangles that correspond to each other, the coordinates and reliability of the partial rectangles and the coordinates and reliability of the human body rectangles are obtained, and the overlapping ratio of the partial rectangles and the ratio of the human body rectangles to each other a human body integration unit that integrates overlapping human body rectangles based on the overlapping ratio;
human body detection device.
[0104]
(Appendix 9)
　Estimate a specific part of the human body from the image, output the coordinates and reliability of the first partial rectangle containing the part,
　Based on the coordinates of the first partial rectangle, estimate and output a first human body candidate area, which is an area in which a human body corresponding to the part is predicted to exist,
Estimate a first human body rectangle containing the human body based on the first human body candidate region, output the coordinates and reliability of the first human body rectangle,
Estimate and output the second human body candidate region from the image,
Based on the second human body candidate region, estimate a specific part corresponding to the human body, output the coordinates and reliability of a second partial rectangle containing the part,
Estimate a second human body rectangle containing the human body based on the second human body candidate region, output the coordinates and reliability of the second human body rectangle,
For a plurality of pairs of partial rectangles and human body rectangles that correspond to each other, the coordinates and reliability of the partial rectangles and the coordinates and reliability of the human body rectangles are obtained, and the overlapping ratio of the partial rectangles and the ratio of the human body rectangles to each other A human body detection method that integrates overlapping human body rectangles based on the overlapping ratio.
[0105]
(Appendix 10)
　Estimate a specific part of the human body from the image, output the coordinates and reliability of the first partial rectangle containing the part,
　Based on the coordinates of the first partial rectangle, estimate and output a first human body candidate area, which is an area in which a human body corresponding to the part is predicted to exist,
Estimate a first human body rectangle containing the human body based on the first human body candidate region, output the coordinates and reliability of the first human body rectangle,
Estimate and output the second human body candidate region from the image,
Based on the second human body candidate region, estimate a specific part corresponding to the human body, output the coordinates and reliability of a second partial rectangle containing the part,
Estimate a second human body rectangle containing the human body based on the second human body candidate region, output the coordinates and reliability of the second human body rectangle,
For a plurality of pairs of partial rectangles and human body rectangles that correspond to each other, the coordinates and reliability of the partial rectangles and the coordinates and reliability of the human body rectangles are obtained, and the overlapping ratio of the partial rectangles and the ratio of the human body rectangles to each other A recording medium recording a program for causing a computer to execute a process of integrating overlapping human body rectangles based on an overlapping ratio.
[0106]
(Appendix 11)
a partial rectangle estimation unit that estimates a specific part of the human body from the image and outputs the coordinates and reliability of the partial rectangle containing the part;
a human body center estimation unit that estimates the center position of the human body corresponding to the part estimated by the partial rectangle estimation unit;
a human body region estimation unit that estimates and outputs a human body region based on the coordinates of the partial rectangle and the center position of the human body;
a human body rectangle estimation unit that estimates a human body rectangle including the human body from the image and outputs the coordinates and reliability of the human body rectangle;
an integration candidate determination unit that determines a human body rectangle with a lower reliability as an integration candidate human body rectangle from among a plurality of human body rectangles in which the ratio of overlap between the human body rectangles is greater than a third threshold;
a human body rectangle integration unit that rejects human body rectangles other than human body rectangles whose overlap ratio with the human body region is greater than a fourth threshold among the human body rectangles of the integration candidates;
human body detection device.
[0107]
(Appendix 12)
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
Estimate the center position of the human body corresponding to the estimated part,
Estimate and output the human body region based on the coordinates of the partial rectangle and the center position of the human body,
Estimate a human body rectangle containing the human body from the image, output the coordinates and reliability of the human body rectangle,
Among a plurality of human body rectangles whose overlapping ratio between the human body rectangles is larger than the third threshold, the human body rectangle with the lower reliability is determined as the integration candidate human body rectangle,
A human body detection method for rejecting human body rectangles other than human body rectangles whose overlapping ratio with the human body region is greater than a fourth threshold, among the human body rectangles of the integration candidates.
[0108]
(Appendix 13)
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
Estimate the center position of the human body corresponding to the estimated part,
Estimate and output the human body region based on the coordinates of the partial rectangle and the center position of the human body,
Estimate a human body rectangle containing the human body from the image, output the coordinates and reliability of the human body rectangle,
Among a plurality of human body rectangles whose overlapping ratio between the human body rectangles is larger than the third threshold, the human body rectangle with the lower reliability is determined as the integration candidate human body rectangle,
A recording medium recording a program for causing a computer to execute a process of rejecting human body rectangles other than the human body rectangles whose overlapping ratio with the human body region is greater than a fourth threshold among the integration candidate human body rectangles.
[0109]
(Appendix 14)
a partial rectangle estimation unit that estimates a specific part of the human body from the image and outputs the coordinates and reliability of the partial rectangle containing the part;
a human body rectangle estimation unit that estimates a human body rectangle including the human body from the image and outputs the coordinates and reliability of the human body rectangle;
a threshold determination unit that determines a fifth threshold based on the number of partial rectangles;
a threshold determination unit that obtains a sixth threshold based on the human body region estimated from the partial rectangle;
a threshold determination unit that determines a seventh threshold between the fifth threshold and the sixth threshold using the fifth threshold and the sixth threshold;
a human body rectangle integration unit that excludes a human body rectangle with a lower reliability from among a plurality of human body rectangles whose overlapping ratio is greater than the seventh threshold;
human body detection device.
[0110]
(Appendix 15)
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
Estimate a human body rectangle containing the human body from the image, output the coordinates and reliability of the human body rectangle,
A fifth threshold is obtained based on the number of partial rectangles,
A sixth threshold is obtained based on the human body region estimated from the partial rectangle,
using the fifth threshold and the sixth threshold to determine a seventh threshold between the fifth threshold and the sixth threshold,
A human body detection method that excludes a human body rectangle with a lower degree of reliability from among a plurality of human body rectangles whose overlapping ratio is greater than the seventh threshold.
[0111]
(Appendix 16)
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
Estimate a human body rectangle containing the human body from the image, output the coordinates and reliability of the human body rectangle,
A fifth threshold is obtained based on the number of partial rectangles,
A sixth threshold is obtained based on the human body region estimated from the partial rectangle,
using the fifth threshold and the sixth threshold to determine a seventh threshold between the fifth threshold and the sixth threshold,
A recording medium recording a program that causes a computer to execute a process of excluding a human body rectangle with a lower reliability from among a plurality of human body rectangles whose overlapping ratio is greater than the seventh threshold.
[0112]
Although the present invention has been described with reference to the embodiments and examples, the present invention is not limited to the above embodiments and examples. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.
Code explanation
[0113]
10, 100, 200, 300, 400, 500 human body detection device
　101, 201, 301, 401, 501　Image storage unit
102, 406 head rectangle estimation unit
　103, 203, 303, 403, 405 Human body candidate region estimation unit
　104, 204, 304, 404, 407, 504 Human body rectangle estimation unit
　202, 302, 402, 502 Head rectangle/body center estimation unit
　305, 408, 505 Human body rectangle integration part
503 Human body area estimation unit
The scope of the claims
[Claim 1]
a partial rectangle estimation unit that estimates a specific part of the human body from the image and outputs the coordinates and reliability of the partial rectangle containing the part;
　Based on the coordinates of the partial rectangle, a human body candidate region estimating unit that estimates and outputs a human body candidate region that is a region in which a human body corresponding to the part is predicted to exist;
a human body rectangle estimation unit that estimates a human body rectangle including a human body based on the human body candidate region and outputs the coordinates and reliability of the human body rectangle;
human body detection device.
[Claim 2]
a human body center estimation unit for estimating the center position of the human body corresponding to the part estimated by the partial rectangle estimation unit,
The human body detection device according to claim 1, wherein the human body candidate region estimation unit estimates the human body candidate region based on the coordinates of the partial rectangle and the center position of the human body.
[Claim 3]
3. The human body detection apparatus according to claim 2, wherein the human body candidate region estimating unit estimates a rectangular region centered at the center position of the human body and having a predetermined aspect ratio as the human body candidate region.
[Claim 4]
For a plurality of pairs of partial rectangles and human body rectangles that correspond to each other, the coordinates and reliability of the partial rectangles and the coordinates and reliability of the human body rectangles are obtained, and the overlapping ratio between the head rectangles and the ratio of the human body rectangles are obtained. 4. The human body detection device according to claim 1, further comprising a human body integration unit that integrates overlapping human body rectangles based on an overlapping ratio.
[Claim 5]
The human body integration unit determines that the reliability of the partial rectangle is low when the overlapping ratio of the human body rectangles is larger than a first threshold and the overlapping ratio of the partial rectangles is larger than a second threshold. 5. The human body detecting device according to claim 4, wherein one human body rectangle is excluded.
[Claim 6]
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
　Based on the coordinates of the partial rectangle, estimate and output a human body candidate area, which is an area in which a human body corresponding to the part is predicted to exist,
A human body detection method for estimating a human body rectangle containing a human body based on the human body candidate region and outputting the coordinates and reliability of the human body rectangle.
[Claim 7]
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
　Based on the coordinates of the partial rectangle, estimate and output a human body candidate area, which is an area in which a human body corresponding to the part is predicted to exist,
A recording medium recording a program for causing a computer to execute a process of estimating a human body rectangle containing a human body based on the human body candidate region and outputting the coordinates and reliability of the human body rectangle.
[Claim 8]
a first partial rectangle estimation unit that estimates a specific part of the human body from the image and outputs the coordinates and reliability of a first partial rectangle containing the part;
a first human body candidate region estimation unit for estimating and outputting a first human body candidate region, which is a region in which a human body corresponding to the first partial rectangle is predicted to exist, based on the coordinates of the first partial rectangle; ,
a first human body rectangle estimation unit for estimating a first human body rectangle including a human body based on the first human body candidate region and outputting the coordinates and reliability of the first human body rectangle;
a second human body candidate region estimation unit that estimates and outputs a second human body candidate region from the image;
Second partial rectangle estimation for estimating a specific part corresponding to the human body based on the second human body candidate region and outputting coordinates and reliability of a second partial rectangle containing the part Department and
a second human body rectangle estimation unit for estimating a second human body rectangle including the human body based on the second human body candidate region and outputting the coordinates and reliability of the second human body rectangle;
For a plurality of pairs of partial rectangles and human body rectangles that correspond to each other, the coordinates and reliability of the partial rectangles and the coordinates and reliability of the human body rectangles are obtained, and the overlapping ratio between the head rectangles and the ratio of the human body rectangles are obtained. a human body integration unit that integrates overlapping human body rectangles based on the overlapping ratio;
human body detection device.
[Claim 9]
　Estimate a specific part of the human body from the image, output the coordinates and reliability of the first partial rectangle containing the part,
　Based on the coordinates of the first partial rectangle, estimate and output a first human body candidate area, which is an area in which a human body corresponding to the part is predicted to exist,
Estimate a first human body rectangle containing the human body based on the first human body candidate region, output the coordinates and reliability of the first human body rectangle,
Estimate and output the second human body candidate region from the image,
Based on the second human body candidate region, estimate a specific part corresponding to the human body, output the coordinates and reliability of a second partial rectangle containing the part,
Estimate a second human body rectangle containing the human body based on the second human body candidate region, output the coordinates and reliability of the second human body rectangle,
For a plurality of pairs of partial rectangles and human body rectangles that correspond to each other, the coordinates and reliability of the partial rectangles and the coordinates and reliability of the human body rectangles are obtained, and the overlapping ratio of the partial rectangles and the ratio of the human body rectangles to each other A human body detection method that integrates overlapping human body rectangles based on the overlapping ratio.
[Claim 10]
　Estimate a specific part of the human body from the image, output the coordinates and reliability of the first partial rectangle containing the part,
　Based on the coordinates of the first partial rectangle, estimate and output a first human body candidate area, which is an area in which a human body corresponding to the part is predicted to exist,
Estimate a first human body rectangle containing the human body based on the first human body candidate region, output the coordinates and reliability of the first human body rectangle,
Estimate and output the second human body candidate region from the image,
Based on the second human body candidate region, estimate a specific part corresponding to the human body, output the coordinates and reliability of the second head rectangle including the part,
Estimate a second human body rectangle containing the human body based on the second human body candidate region, output the coordinates and reliability of the second human body rectangle,
For a plurality of pairs of partial rectangles and human body rectangles that correspond to each other, the coordinates and reliability of the partial rectangles and the coordinates and reliability of the human body rectangles are obtained, and the overlapping ratio of the partial rectangles and the ratio of the human body rectangles to each other A recording medium recording a program for causing a computer to execute a process of integrating overlapping human body rectangles based on an overlapping ratio.
[Claim 11]
a partial rectangle estimation unit that estimates a specific part of the human body from the image and outputs the coordinates and reliability of the partial rectangle containing the part;
a human body center estimation unit that estimates the center position of the human body corresponding to the part estimated by the partial rectangle estimation unit;
a human body region estimation unit that estimates and outputs a human body region based on the coordinates of the partial rectangle and the center position of the human body;
a human body rectangle estimation unit that estimates a human body rectangle including the human body from the image and outputs the coordinates and reliability of the human body rectangle;
an integration candidate determination unit that determines a human body rectangle with a lower reliability as an integration candidate human body rectangle from among a plurality of human body rectangles in which the ratio of overlap between the human body rectangles is greater than a third threshold;
a human body rectangle integration unit that rejects human body rectangles other than human body rectangles whose overlap ratio with the human body region is greater than a fourth threshold among the human body rectangles of the integration candidates;
human body detection device.
[Claim 12]
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
Estimate the center position of the human body corresponding to the estimated part,
Estimate and output the human body region based on the coordinates of the partial rectangle and the center position of the human body,
Estimate a human body rectangle containing the human body from the image, output the coordinates and reliability of the human body rectangle,
Among a plurality of human body rectangles whose overlapping ratio between the human body rectangles is larger than the third threshold, the human body rectangle with the lower reliability is determined as the integration candidate human body rectangle,
A human body detection method for rejecting human body rectangles other than human body rectangles whose overlapping ratio with the human body region is greater than a fourth threshold, among the human body rectangles of the integration candidates.
[Claim 13]
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
Estimate the center position of the human body corresponding to the estimated part,
Estimate and output the human body region based on the coordinates of the partial rectangle and the center position of the human body,
Estimate a human body rectangle containing the human body from the image, output the coordinates and reliability of the human body rectangle,
Among a plurality of human body rectangles whose overlapping ratio between the human body rectangles is larger than the third threshold, the human body rectangle with the lower reliability is determined as the integration candidate human body rectangle,
A recording medium recording a program for causing a computer to execute a process of rejecting human body rectangles other than the human body rectangles whose overlapping ratio with the human body region is greater than a fourth threshold among the integration candidate human body rectangles.
[Claim 14]
a partial rectangle estimation unit that estimates a specific part of the human body from the image and outputs the coordinates and reliability of the partial rectangle containing the part;
a human body rectangle estimation unit that estimates a human body rectangle including the human body from the image and outputs the coordinates and reliability of the human body rectangle;
a threshold determination unit that determines a fifth threshold based on the number of partial rectangles;
a threshold determination unit that obtains a sixth threshold based on the human body region estimated from the partial rectangle;
a threshold determination unit that determines a seventh threshold between the fifth threshold and the sixth threshold using the fifth threshold and the sixth threshold;
a human body rectangle integration unit that excludes a human body rectangle with a lower reliability from among a plurality of human body rectangles whose overlapping ratio is greater than the seventh threshold;
human body detection device.
[Claim 15]
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
Estimate a human body rectangle containing the human body from the image, output the coordinates and reliability of the human body rectangle,
A fifth threshold is obtained based on the number of partial rectangles,
A sixth threshold is obtained based on the human body region estimated from the partial rectangle,
using the fifth threshold and the sixth threshold to determine a seventh threshold between the fifth threshold and the sixth threshold,
A human body detection method that excludes a human body rectangle with a lower degree of reliability from among a plurality of human body rectangles whose overlapping ratio is greater than the seventh threshold.
[Claim 16]
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
Estimate a human body rectangle containing the human body from the image, output the coordinates and reliability of the human body rectangle,
A fifth threshold is obtained based on the number of partial rectangles,
A sixth threshold is obtained based on the human body region estimated from the partial rectangle,
using the fifth threshold and the sixth threshold to determine a seventh threshold between the fifth threshold and the sixth threshold,
A recording medium recording a program that causes a computer to execute a process of excluding a human body rectangle with a lower reliability from among a plurality of human body rectangles whose overlapping ratio is greater than the seventh threshold.
Amended Claims (Article 19 of the Convention)
[ Accepted by the International Bureau on May 10, 2021 ( 10.05.2021 ) ]
[1]
a partial rectangle estimation unit that estimates a specific part of the human body from the image and outputs the coordinates and reliability of the partial rectangle containing the part;
a human body candidate region estimating unit that estimates and outputs a human body candidate region, which is a region in which a human body corresponding to the partial rectangle is predicted to exist, based on the coordinates of the partial rectangle;
a human body rectangle estimation unit that estimates a human body rectangle including a human body based on the human body candidate region and outputs the coordinates and reliability of the human body rectangle;
human body detection device.
[2]
a human body center estimation unit for estimating the center position of the human body corresponding to the part estimated by the partial rectangle estimation unit,
The human body detection device according to claim 1, wherein the human body candidate region estimation unit estimates the human body candidate region based on the coordinates of the partial rectangle and the center position of the human body.
[3]
3. The human body detection apparatus according to claim 2, wherein the human body candidate region estimating unit estimates a rectangular region centered at the center position of the human body and having a predetermined aspect ratio as the human body candidate region.
[Four]
For a plurality of pairs of partial rectangles and human body rectangles that correspond to each other, the coordinates and reliability of the partial rectangles and the coordinates and reliability of the human body rectangles are obtained, and the overlapping ratio between the head rectangles and the ratio of the human body rectangles are obtained. 4. The human body detection device according to claim 1, further comprising a human body integration unit that integrates overlapping human body rectangles based on an overlapping ratio.
[Five]
The human body integration unit determines that the reliability of the partial rectangle is low when the overlapping ratio of the human body rectangles is larger than a first threshold and the overlapping ratio of the partial rectangles is larger than a second threshold. 5. The human body detecting device according to claim 4, wherein one human body rectangle is excluded.
[6]
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
　Based on the coordinates of the partial rectangle, the human Estimate and output the body candidate region,
A human body detection method for estimating a human body rectangle containing a human body based on the human body candidate region and outputting the coordinates and reliability of the human body rectangle.
[7]
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
　Based on the coordinates of the partial rectangle, estimate and output a human body candidate area, which is an area in which a human body corresponding to the part is predicted to exist,
A recording medium recording a program for causing a computer to execute a process of estimating a human body rectangle containing a human body based on the human body candidate region and outputting the coordinates and reliability of the human body rectangle.
[8]
a first partial rectangle estimation unit that estimates a specific part of the human body from the image and outputs the coordinates and reliability of a first partial rectangle containing the part;
a first human body candidate region estimation unit for estimating and outputting a first human body candidate region, which is a region in which a human body corresponding to the first partial rectangle is predicted to exist, based on the coordinates of the first partial rectangle; ,
a first human body rectangle estimation unit for estimating a first human body rectangle including a human body based on the first human body candidate region and outputting the coordinates and reliability of the first human body rectangle;
a second human body candidate region estimation unit that estimates and outputs a second human body candidate region from the image;
Second partial rectangle estimation for estimating a specific part corresponding to the human body based on the second human body candidate region and outputting coordinates and reliability of a second partial rectangle containing the part Department and
a second human body rectangle estimation unit for estimating a second human body rectangle including the human body based on the second human body candidate region and outputting the coordinates and reliability of the second human body rectangle;
For a plurality of pairs of partial rectangles and human body rectangles that correspond to each other, the coordinates and reliability of the partial rectangles and the coordinates and reliability of the human body rectangles are obtained, and the overlapping ratio between the head rectangles and the ratio of the human body rectangles are obtained. a human body integration unit that integrates overlapping human body rectangles based on the overlapping ratio;
human body detection device.
[9]
　Estimate a specific part of the human body from the image, output the coordinates and reliability of the first partial rectangle containing the part,
　Based on the coordinates of the first partial rectangle, estimate and output a first human body candidate area, which is an area in which a human body corresponding to the part is predicted to exist,
Estimate a first human body rectangle containing the human body based on the first human body candidate region, output the coordinates and reliability of the first human body rectangle,
Estimate and output the second human body candidate region from the image,
Based on the second human body candidate region, estimate a specific part corresponding to the human body, output the coordinates and reliability of a second partial rectangle containing the part,
Estimate a second human body rectangle containing the human body based on the second human body candidate region, output the coordinates and reliability of the second human body rectangle,
For a plurality of pairs of partial rectangles and human body rectangles that correspond to each other, the coordinates and reliability of the partial rectangles and the coordinates and reliability of the human body rectangles are obtained, and the overlapping ratio of the partial rectangles and the ratio of the human body rectangles to each other A human body detection method that integrates overlapping human body rectangles based on the overlapping ratio.
[Ten]
　Estimate a specific part of the human body from the image, output the coordinates and reliability of the first partial rectangle containing the part,
　Based on the coordinates of the first partial rectangle, estimate and output a first human body candidate area, which is an area in which a human body corresponding to the part is predicted to exist,
Estimate a first human body rectangle containing the human body based on the first human body candidate region, output the coordinates and reliability of the first human body rectangle,
Estimate and output the second human body candidate region from the image,
Based on the second human body candidate region, estimate a specific part corresponding to the human body, output the coordinates and reliability of the second head rectangle including the part,
Estimate a second human body rectangle containing the human body based on the second human body candidate region, output the coordinates and reliability of the second human body rectangle,
For a plurality of pairs of partial rectangles and human body rectangles that correspond to each other, the coordinates and reliability of the partial rectangles and the coordinates and reliability of the human body rectangles are obtained, and the overlapping ratio of the partial rectangles and the ratio of the human body rectangles to each other A recording medium recording a program for causing a computer to execute a process of integrating overlapping human body rectangles based on an overlapping ratio.
[11]
a partial rectangle estimation unit that estimates a specific part of the human body from the image and outputs the coordinates and reliability of the partial rectangle containing the part;
a human body center estimation unit that estimates the center position of the human body corresponding to the part estimated by the partial rectangle estimation unit;
a human body region estimation unit that estimates and outputs a human body region based on the coordinates of the partial rectangle and the center position of the human body;
a human body rectangle estimation unit that estimates a human body rectangle including the human body from the image and outputs the coordinates and reliability of the human body rectangle;
an integration candidate determination unit that determines a human body rectangle with a lower reliability as an integration candidate human body rectangle from among a plurality of human body rectangles in which the ratio of overlap between the human body rectangles is greater than a third threshold;
a human body rectangle integration unit that rejects human body rectangles other than human body rectangles whose overlap ratio with the human body region is greater than a fourth threshold among the human body rectangles of the integration candidates;
human body detection device.
[12]
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
Estimate the center position of the human body corresponding to the estimated part,
Estimate and output the human body region based on the coordinates of the partial rectangle and the center position of the human body,
Estimate a human body rectangle containing the human body from the image, output the coordinates and reliability of the human body rectangle,
Among a plurality of human body rectangles whose overlapping ratio between the human body rectangles is larger than the third threshold, the human body rectangle with the lower reliability is determined as the integration candidate human body rectangle,
A human body detection method for rejecting human body rectangles other than human body rectangles whose overlapping ratio with the human body region is greater than a fourth threshold, among the human body rectangles of the integration candidates.
[13]
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
Estimate the center position of the human body corresponding to the estimated part,
Estimate and output the human body region based on the coordinates of the partial rectangle and the center position of the human body,
Estimate a human body rectangle containing the human body from the image, output the coordinates and reliability of the human body rectangle,
Among a plurality of human body rectangles whose overlapping ratio between the human body rectangles is larger than the third threshold, the human body rectangle with the lower reliability is determined as the integration candidate human body rectangle,
A recording medium recording a program for causing a computer to execute a process of rejecting human body rectangles other than the human body rectangles whose overlapping ratio with the human body region is greater than a fourth threshold among the integration candidate human body rectangles.
[14]
a partial rectangle estimation unit that estimates a specific part of the human body from the image and outputs the coordinates and reliability of the partial rectangle containing the part;
a human body rectangle estimation unit that estimates a human body rectangle including the human body from the image and outputs the coordinates and reliability of the human body rectangle;
a threshold determination unit that determines a fifth threshold based on the number of partial rectangles;
a threshold determination unit that obtains a sixth threshold based on the human body region estimated from the partial rectangle;
a threshold determination unit that determines a seventh threshold between the fifth threshold and the sixth threshold using the fifth threshold and the sixth threshold;
a human body rectangle integration unit that excludes a human body rectangle with a lower reliability from among a plurality of human body rectangles whose overlapping ratio is greater than the seventh threshold;
human body detection device.
[15]
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
Estimate a human body rectangle containing the human body from the image, output the coordinates and reliability of the human body rectangle,
A fifth threshold is obtained based on the number of partial rectangles,
A sixth threshold is obtained based on the human body region estimated from the partial rectangle,
using the fifth threshold and the sixth threshold to determine a seventh threshold between the fifth threshold and the sixth threshold,
A human body detection method that excludes a human body rectangle with a lower degree of reliability from among a plurality of human body rectangles whose overlapping ratio is greater than the seventh threshold.
[16]
　Estimate a specific part of the human body from the image, output the coordinates and reliability of a partial rectangle containing the part,
Estimate a human body rectangle containing the human body from the image, output the coordinates and reliability of the human body rectangle,
A fifth threshold is obtained based on the number of partial rectangles,
A sixth threshold is obtained based on the human body region estimated from the partial rectangle,
using the fifth threshold and the sixth threshold to determine a seventh threshold between the fifth threshold and the sixth threshold,
A recording medium recording a program that causes a computer to execute a process of excluding a human body rectangle with a lower reliability from among a plurality of human body rectangles whose overlapping ratio is greater than the seventh threshold.
[17]
[Addition] A threshold determination unit that determines a threshold related to the reliability of the detection result of the object based on the IoU value related to the first part of the object and the IoU value related to the second part of the object;
an integration unit that integrates the plurality of object regions when the overlapping ratio between the plurality of object regions exceeds the threshold;
An object detection device equipped with
[18]
[Addition] Based on the IoU value related to the first part of the object and the IoU value related to the second part of the object, determine the threshold related to the reliability of the detection result of the object,
An object detection method that integrates the plurality of object regions when the overlapping ratio between the plurality of object regions exceeds the threshold.
[19]
[Addition] Based on the IoU value related to the first part of the object and the IoU value related to the second part of the object, determine the threshold related to the reliability of the detection result of the object,
A recording medium recording a program that causes a computer to execute a process of integrating the multiple object areas when the overlapping ratio between the multiple object areas exceeds the threshold.