[0001]The present disclosure relates to an image processing apparatus, an image processing method, and a program, and, for example, relates to an image processing apparatus that executes face authentication of a person. [Background Art]
10 [0002]
In entrance/exit management at airports, stadiums, and the like, face authentication is used for security or identity verification. An action of attempting to break the face authentication is performed using a face image of another person (for example, a print of a photograph, or the
15 like). Such an action is an example of spoofing that is an action of pre-
tending to be another person. [0003]
An example of spoofing will be described in detail with reference to Fig. 8. As illustrated in Fig. 8, a person conceals one’s face by raising
20 a print of a photograph, a display, or the like in front of the one’s face. A
face image of another person is displayed on the print or display . The person illegally passes the face authentication by using the face image of another person displayed on the print or display. PTL 1 and NPL 1 dis-close related technologies for detecting spoofing as described above.
25 [0004]
In the related technology described in PTL 1, a face of a person is imaged as a moving image, and a blink of the person is detected on the basis of a difference between time-series images. Then, when the blink of
2

the person has never been detected within a predetermined period, it is de¬
termined that the face of the person is not authentic. The fact that a face
of a human is not authentic means that the face of the human is not the
face of the person oneself. In this way, in the related technology de-
5 scribed in PTL 1, spoofing is determined.
[0005]
In the related technology described in NPL 1, spoofing is deter¬
mined by using machine learning. Specifically, features of an authentic
face are learned using a convolutional neural network. Then, a discrimi-
10 nator is generated for discriminating between a face image appearing on a
print or a display and the authentic face. It is determined whether the
face of the person is fake or authentic by using the learned discriminator.
[Citation List]
[Patent Literature]
15 [0006]
[PTL 1] JP 5061563 B2 [Non Patent Literature] [0007]
[NPL 1] Koichi Ito et al., “A Liveness Detection Method Using Convolu-
20 tional Neural Network” (IEICE Transactions A), December 1, 2017, Vol.
J100-A, No. 12, pp. 455-464
[NPL 2] SHEN Linlin, “Gabor Features and Support Vector Machine for
Face Identification”, Biomedical fuzzy and human sciences, the official
journal of the Biomedical Fuzzy Systems Association 14(1), pp. 61-66,
25 2009-01-08
[Summary] [Technical Problem] [0008]
In the related technology described in PTL 1, spoofing is identified
3

by detecting a blink of a person. For that reason, when a moving image
obtained by imaging the face of another person who blinks is displayed on
a display, there is a possibility that a face image displayed on the display
is erroneously identified as being authentic in the related technology de-
5 scribed in PTL 1. Thus, it is not possible to prevent a malicious person
from illegally passing the face authentication while holding the display on which the face image of another person is displayed in the hand as illus-trated in Fig. 8. [0009]
10 In the related technology described in NPL 1, as the resolution of
the face image displayed on the display increases, the accuracy of the dis-criminator decreases, and there is a high possibility that the face image is erroneously identified as the authentic face. [0010]
15 The present disclosure has been made in view of the above prob-
lems, and an object of the present disclosure is to provide an image pro-cessing apparatus and the like capable of accurately identifying spoofing using a face image. [Solution to Problem]
20 [0011]
An image processing apparatus according to an aspect of the pre-sent embodiments includes: a motion detection means for detecting, from time-series images, a feature relating to a motion of a head portion of a person and a feature relating to a motion of a body portion that is a region
25 other than the head portion of the person; and an index value calculation
means for calculating an index value indicating a degree of consistency between the feature relating to the motion of the head portion of the per-son and the feature relating to the motion of the body portion of the per¬son.
4

[0012]
An image processing method according to an aspect of the present
embodiments includes: detecting, from time-series images, a feature relat¬
ing to a motion of a head portion of a person and a feature relating to a
5 motion of a body portion that is a region other than the head portion of
the person; and calculating an index value indicating a degree of con¬
sistency between the feature relating to the motion of the head portion of
the person and the feature relating to the motion of the body portion of
the person.
10 [0013]
A program according to an aspect of the present embodiments
causes a computer to execute: detecting, from time-series images, a fea¬
ture relating to a motion of a head portion of a person and a feature relat¬
ing to a motion of a body portion that is a region other than the head por-
15 tion of the person; and calculating an index value indicating a degree of
consistency between the feature relating to the motion of the head portion of the person and the feature relating to the motion of the body portion of the person.
[Advantageous Effects]
20 [0014]
According to an aspect of the present embodiments, spoofing using
a face image can be accurately identified.
[Brief Description of Drawings]
[0015]
25 Fig. 1 is a block diagram illustrating a configuration of an image
processing apparatus according to a first example embodiment.
Fig. 2 is a diagram illustrating a relationship between a motion of a head/body portion of a person and spoofing.
Fig. 3 is a flowchart illustrating a flow of a process for identifying
5

spoofing according to the first example embodiment.
Fig. 4 is a block diagram illustrating a configuration of an image processing apparatus according to a second example embodiment.
Fig. 5 is a diagram illustrating a standard value that is a spoofing
5 determination criterion in the second example embodiment.
Fig. 6 is a flowchart illustrating a flow from acquisition of an im-age to a spoofing determination process in the image processing apparatus according to the second example embodiment.
Fig. 7 is a diagram illustrating a hardware configuration of an ap-
10 paratus according to a third example embodiment.
Fig. 8 is a diagram illustrating an example of a person who per¬
forms spoofing.
[Example Embodiment]
[0016]
15 The arrows drawn in the drawings referred to in the following de-
scription indicate a flow direction of a certain signal or data, and are not
intended to exclude communication of the signal or data in both directions
or in a direction opposite to the direction of the arrow.
[0017]
20 [First Example Embodiment]
(Image processing apparatus 100)
Fig. 1 is a block diagram illustrating a configuration of an image
processing apparatus 100 according to a first example embodiment . As il¬
lustrated in Fig. 1, the image processing apparatus 100 includes a motion
25 detection unit 104 and an index value calculation unit 106 . The motion
detection unit 104 is an example of a motion detection means. The index value calculation unit 106 is an example of an index value calculation means. [0018]
6

The motion detection unit 104 detects, from time-series images,
features relating to a motion of a head portion of a person and features re¬
lating to a motion of a body portion that is a region other than the head
portion of the person. For example, the motion detection unit 104 detects
5 the features relating to the motion of the head portion of the person and
the features relating to the motion of the body portion for each image from the time-series images by using a deep learning neural network. [0019]
Here, the head portion of the person is a region including the neck,
10 face, head, and back of the head of the person. The body portion is at
least a part of a region of the entire person excluding the head portion. Specifically, the body portion is the torso, arm, leg, or a combination thereof. The time-series images are, for example, data of a plurality of frame images of a moving image obtained by imaging the person by one or
15 more imaging devices (not illustrated). Hereinafter, data of a frame image
is also referred to as an image for convenience. The time-series images may be a plurality of still images obtained by repeatedly imaging the same person by an imaging device. [0020]
20 An example will be described of a configuration of the motion de-
tection unit 104 described above. Upon acquiring the time-series images, the motion detection unit 104 analyzes those images to detect each of an area of the head portion and an area of the body portion of the person. [0021]
25 Then, the motion detection unit 104 detects first information indi-
cating a change in a position of the head portion from images of the area of the head portion among the time-series images, and extracts the fea¬tures relating to the motion of the head portion from the first in formation. The motion detection unit 104 detects second information indicating a
7

change in a position of the body portion from images of the area of the
body portion among the time-series images, and extracts the feature relat¬
ing to the motion of the body portion from the second information. Here¬
inafter, features relating to motions are abbreviated as motion features.
5 [0022]
The first information indicating the change in the position of the head portion is, for example, information indicating a change (displace¬ment) in a position of a tracking point detected from the area of the head portion in the time-series images. The second information indicating the
10 change in the position of the body portion is, for example, information in-
dicating a change (displacement) in a position of a tracking point detected from the area of the body portion between the time-series images. [0023]
Motion features of the head portion include, for example, a motion
15 vector indicating a direction in which the head portion moves . The mo-
tion detection unit 104 may calculate the change in the position of the head portion in a certain period of time on the basis of the first infor-mation, and may calculate a direction in which the head portion moves in the certain period of time. Alternatively, the motion features of the head
20 portion may include a magnitude of a displacement of the head portion, or
a period of the motion of the head portion. On the basis of the first infor¬mation, the motion detection unit 104 may calculate, as the period of the motion of the head portion, an average time during which the position of the head portion changes from the uppermost vertex to the next uppermost
25 vertex.
[0024]
Information indicating the motion features of the head portion may be data regarding the motion vector indicating the direction in which the head portion moves, data representing the magnitude of the displacement
8

of the head portion, or data representing the period of the motion of the
head portion. For example, information of the motion vector regarding
the motion of the head portion includes data regarding a motion vector of
the tracking point in the area of the head portion. The data may include
5 position coordinates before and after the tracking point is displaced, and
data representing the direction and magnitude of the vector. [0025]
On the other hand, motion features of the body portion include, for
example, a motion vector indicating a direction in which the body portion
10 moves. Alternatively, the motion features of the body portion may in-
clude a magnitude of a displacement of the body portion or a period of the
motion of the body portion. Information indicating the motion features of
the body portion may be data regarding the motion vector indicating the
direction in which the body portion moves, data representing the magni-
15 tude of the displacement of the body portion, or data representing the pe¬
riod of the motion of the body portion. For example, information of the
motion vector regarding the motion of the body portion may include data
regarding a motion vector of the tracking point in the area of the body
portion. The data may include position coordinates before and after the
20 tracking point is displaced, and data representing the direction and magni¬
tude of the vector.
[0026]
The motion detection unit 104 transmits the information indicating
the motion features of the head portion and the information indicating the
25 motion features of the body portion to the index value calculation unit
106. [0027]
The index value calculation unit 106 receives the information indi-
9

cating the motion features of the head portion and the information indicat¬
ing the motion features of the body portion from the motion detection unit
104. The index value calculation unit 106 calculates an index value indi¬
cating a degree of consistency between the motion of the head portion of
5 the person and the motion of the body portion of the person on the basis
of the information indicating the motion features of the head portion and the information indicating the motion features of the body portion. [0028]
The consistency mentioned here is not a concept including only
10 that the motion of the head portion of the person and the motion of the
body portion of the person completely match each other. A degree of sim¬ilarity indicating closeness between the motion features of the head por-tion of the person and the motion features of the body portion of the per¬son is also included in a category of the concept of consistency.
15 [0029]
For example, the index value calculated by the index value calcu-lation unit 106 is an output from the deep learning neural network having the motion vector indicating the direction in which the head portion moves and the motion vector indicating the direction in which the body
20 portion moves as inputs. Alternatively, the index value is a distance be-
tween the motion vector indicating the direction in which the head portion moves and the motion vector indicating the direction in which the body portion moves. Alternatively, the index value may be an angle between the motion vector indicating the direction in which the head portion
25 moves and the motion vector indicating the direction in which the body
portion moves. A calculation method is not limited as long as the index value indicates the degree of consistency between the motion of the head portion of the person and the motion of the body portion of the person. A
10

relationship between the index value calculated by the index value calcu-lation unit 106 and spoofing by the person will be descri bed below. [0030]
(Relationship between index value and spoofing)
5 As described above, the index value indicates the degree of con-
sistency between the motion of the head portion of the person and the mo¬
tion of the body portion of the person. Spoofing is an action of pretend¬
ing that the person is another person. Here, a case will be described
where the person performs spoofing another person by using a face image
10 of the another person displayed on a print or display.
[0031]
A relationship between the index value and a spoofing action by
the person will be described with reference to Fig. 2. Fig. 2 is a diagram
illustrating a relationship between the index value and presence or ab-
15 sence of spoofing.
[0032]
A case (a) in Fig. 2 illustrates a case where the index value is low.
A low index value indicates that there is a contradiction between the mo¬
tion features of the head portion and the motion features of the body por-
20 tion. In the case (a), a person is performing spoofing by using a face im¬
age of another person. The person holds a display in the hand and raises
the display in front of the face like the person illustrated in Fig. 8 . For
that reason, in the case (a), a contradiction may occur as a motion of one
human, between the motion of the head portion and the motion of the body
25 portion. More specifically, in walk-through authentication, when the per¬
son holding the display in the hand walks, a deviation unintended by the
person occurs between the motion of the body portion and the motion of
the arm (elbow) with respect to the period and magnitude of the motion,
11

and the contradiction occurs as the motion of one human, between the mo¬
tion of the head portion displayed on the display and the motion of the
body portion of the person. Thus, the lower the index value, the higher
the possibility that the person is performing spoofing.
5 [0033]
In (b) of Fig. 2, a case is illustrated where the index value is high .
A high index value indicates that the motion features of the head portion
and the motion features of the body portion are consistent with each other.
In the case (b), the person is not performing spoofing. For that reason,
10 the direction in which the head portion of the person moves is synchro-
nized with the direction in which the body portion moves, and there is al¬
most no difference between the displacement of the head portion and the
displacement of the body portion. That is, in the case (b), the motion of
the head portion and the motion of the body portion match each other or
15 are at least similar to each other. Thus, the lower the index value, the
lower the possibility that a person is performing spoofing. [0034]
(Flow of index value calculation process)
An index value calculation process executed by the image pro-
20 cessing apparatus 100 according to the first example embodiment will be
described with reference to Fig. 3. Fig. 3 is a flowchart illustrating a
flow of the index value calculation process by the image processing appa¬
ratus 100.
[0035]
25 As illustrated in Fig. 3, the motion detection unit 104 acquires
time-series images. For example, the motion detection unit 104 acquires a moving image imaged by a monitoring camera (not illustrated), and ex¬tracts a plurality of images including a person from the moving image. [0036]
12

The motion detection unit 104 detects each of an area of the head
portion of the person and an area of the body portion of the person from
the time-series images acquired (S101).
[0037]
5 The motion detection unit 104 analyzes the area of the head por-
tion of the person and the area of the body portion of the person detected
from the time-series images to detect features relating to the motion of the
head portion of the person (motion features of the head portion) and fea¬
tures relating to the motion of the body portion (motion feat ures of the
10 body). (S102).
[0038]
The motion detection unit 104 transmits the information indicating
the motion features of the head portion and the information indicating the
motion features of the body portion detected in step S102 to the index
15 value calculation unit 106.
[0039]
The index value calculation unit 106 receives the motion features
of the head portion and the motion features of the body portion from the
motion detection unit 104. Then, the index value calculation unit 106 cal-
20 culates the index value described above from the motion features of the
head portion and the motion features of the body portion received (S103). [0040]
Although not illustrated, after step S103, the index value calcula¬
tion unit 106 may output the calculated index value to an external device
25 (not illustrated) such as a display device. In addition, in the first example
embodiment, the image processing apparatus 100 may determine spoofing using a face image from a magnitude of the index value and output an alert. [0041]
13

After the process described above, the index value calculation pro¬
cess executed by the image processing apparatus 100 according to the first
example embodiment ends.
[0042]
5 (Effects of present example embodiment)
According to the configuration of the present example embodi¬ment, the motion detection unit 104 detects the features relating to the motion of the head portion of the person and the features relating to the motion of the body portion that is the region other than the head portion
10 of the person. The index value calculation unit 106 calculates the index
value indicating the degree of consistency between the features relating to the motion of the head portion of the person and the features relating to the motion of the body portion. The index value represents a result of identifying spoofing.
15 [0043]
When the face of the person is the face image displayed on the print or the display, a contradiction occurs between the motion of the head portion of the person and the motion of the region other than the head por¬tion. For that reason, since the motion features of the head portion and
20 the motion features of the body portion calculated by the motion detection
unit 104 do not match each other, the index value calculated by the index value calculation unit 106 is low. That is, the index value is a parameter reflecting the possibility that the person is performing spoofing. Accord¬ing to the first example embodiment, spoofing using a face image can be
25 accurately identified.
[0044] [Second Example Embodiment]
A second example embodiment includes a configuration that not
14

only calculates an index value representing a result of identifying spoof¬
ing, but also determines whether a person is performing spoofing on the
basis of the index value and outputs a determination result. A spoofing
score described in the second example embodiment is an example of a pa-
5 rameter based on the index value.
[0045]
(Image processing apparatus 200)
A configuration of an image processing apparatus 200 according to the second example embodiment will be described with reference to Fig.
10 4. Fig. 4 is a block diagram illustrating a configuration of the image pro-
cessing apparatus 200. As illustrated in Fig. 4, the image processing ap-paratus 200 includes a motion detection unit 210, an index value calcula-tion unit 230, and a spoofing determination unit 240. The motion detec-tion unit 210 is an example of a motion detection means. The index value
15 calculation unit 230 is an example of an index value calculation means .
The spoofing determination unit 240 is an example of a spoofing determi-nation means. [0046]
The image processing apparatus 200 is connected to an input unit
20 10 and an output unit 20. The input unit 10 inputs time-series images to a
head portion detection unit 211 and a body portion detection unit 212 in-cluded in the motion detection unit 210 of the image processing apparatus 200. The input unit 10 may include an imaging device that generates time-series images.
25 [0047]
The output unit 20 receives a determination result (described as a spoofing determination result in Fig. 4) indicating whether a person is performing spoofing, and data of the spoofing score to be described later,
15

from the spoofing determination unit 240 of the image processing appa¬
ratus 200. The output unit 20 outputs the spoofing score and spoofing de¬
termination result received to an external device such as a display device.
[0048]
5 As illustrated in Fig. 4, the image processing apparatus 200, the
input unit 10, the output unit 20, a weight value storage unit 30, a stand¬ard value storage unit 40, and a threshold value storage unit 50 constitute a spoofing detection device 1. [0049]
10 As illustrated in Fig. 4, the motion detection unit 210 includes the
head portion detection unit 211, the body portion detection unit 212, a head portion motion features extraction unit 215, a body portion motion features extraction unit 216, and a feature integration unit 220. [0050]
15 The head portion detection unit 211 is an example of a head por-
tion detection means. The body portion detection unit 212 is an example of a body portion detection means. [0051]
The head portion detection unit 211 detects an area corresponding
20 to the head portion of the person in the time-series images. As described
in the first example embodiment, the head portion of the person is a re-gion including the neck, face, head, and back of the head of the person . The body portion detection unit 212 detects an area corresponding to the body portion of the person in the time-series images. The body portion is
25 at least a part of a region of the entire person excluding the head portion.
[0052]
For example, the head portion detection unit 211 detects the con-tour of the head portion of the person from each of the time -series images by pattern matching. For example, the body portion detection unit 212
16

detects a contour of the body portion of the person from each of the time -
series images by pattern matching. In this case, the head portion detec¬
tion unit 211 detects the contour of the head portion of the person from
the time-series images by matching a sample image of the contour of the
5 head portion collected in advance with the time-series images acquired
from the input unit 10. Similarly, the body portion detection unit 212 de¬tects the contour of the body portion of the person from the time-series images by a pattern matching method. [0053]
10 The head portion detection unit 211 detects a tracking point of the
head portion of the person in the area of the head portion of the person detected. The tracking point of the head portion is, for example, a posi-tion of a part on the face such as the eyes, nose, or ears, or a position of the neck or top of the head. The body portion detection unit 212 detects a
15 tracking point of the body portion of the person in the area of the body
portion of the person detected by the body portion detection unit 212 . The tracking point of the body portion is, for example, a position of a joint on the torso, arm, or leg. However, the tracking points of the head portion and the body portion are not limited to the examples described
20 here.
[0054]
Specifically, the head portion detection unit 211 detects the track¬ing point of the head portion of the person from each of the time-series images by using information for specifying the tracking point. The body
25 portion detection unit 212 detects the tracking point of the body portion
of the person from each of the time-series images by using information for
specifying the tracking point.
[0055]
The information for specifying the tracking point is, for example, a
17

feature value of the tracking point. The feature value of the tracking
point may be a Haar-like feature value related to a luminance difference
between a pixel corresponding to the tracking point and its surrounding
pixels. Alternatively, the information indicating the tracking point may
5 be obtained by converting a luminance or hue of the pixel corresponding
to the tracking point into vector data (numerical data string) by using a
Gabor filter (NPL 2). The vector data is also referred to as a feature vec¬
tor.
[0056]
10 As another method for converting data of the time-series images
into vector data, a scale-invariant feature transform (SIFT) method or a
histograms of oriented gradients (HOG) method may be used. The head
portion detection unit 211 and the body portion detection unit 212 may be
designed to select an appropriate feature value conversion filter depend-
15 ing on a type of the images.
[0057]
The head portion detection unit 211 transmits detection results of
the tracking point of the head portion in the plurality of time-series im¬
ages to the head portion motion features extraction unit 215 together with
20 data of the plurality of time-series images. The body portion detection
unit 212 transmits detection results of the tracking point of the body por¬
tion in the plurality of time-series images to the body portion motion fea¬
tures extraction unit 216 together with the data of the plurality of time -se¬
ries images.
25 [0058]
Alternatively, the head portion detection unit 211 may transmit de-tection results (for example, position information of the contour of the head portion) of the area of the head portion to the head portion motion features extraction unit 215 instead of the detection results of the tracking
18

point of the head portion. The body portion detection unit 212 may trans¬
mit detection results (for example, position information of the contour of
the body portion) of the area of the body portion to the body portion mo¬
tion features extraction unit 216 instead of the detection results of the
5 tracking point of the body portion.
[0059]
The head portion motion features extraction unit 215 receives the
detection results of the head portion of the person in the time -series im¬
ages together with the data of the plurality of time-series images from the
10 head portion detection unit 211. The body portion motion features extrac-
tion unit 216 receives the detection result of the body portion of the per¬
son in the time-series images together with the data of the plurality of
time-series images from the body portion detection unit 212.
[0060]
15 The head portion motion features extraction unit 215 extracts fea-
tures relating to the motion of the head portion (motion features of the
head portion) from the plurality of time-series images by using the detec¬
tion results of the head portion of the person in the time-series images.
The motion features of the head portion include the motion vector of the
20 tracking point of the head portion of the person.
[0061]
The body portion motion features extraction unit 216 extracts fea¬
tures relating to the motion of the body portion (motion features of the
body portion) from the plurality of time-series images by using the detec-
25 tion results of the body portion of the person in the time-series images.
The motion features of the body portion include the motion vector of the
tracking point of the body portion of the person.
[0062]
Specifically, the head portion motion features extraction unit 215
19

calculates a change in the position of the head portion in the time-series
images on the basis of the detection results of the head portion of the per¬
son received from the head portion detection unit 211. For example, the
head portion motion features extraction unit 215 detects the first infor-
5 mation indicating the change in the position of the head portion from im¬
ages of the area of the head portion among the time-series images. Then,
the head portion motion features extraction unit 215 calculates the motion
features of the head portion of the person from the calculated first infor¬
mation indicating the change in the position of the head portion. For ex-
10 ample, the head portion motion features extraction unit 215 calculates in¬
formation regarding a motion vector indicating the motion features of the
head portion of the person from the first information.
[0063]
The body portion motion features extraction unit 216 calculates a
15 change in the position of the body portion in the time-series images on the
basis of the detection results of the body portion of the person receiv ed
from the body portion detection unit 212. For example, the body portion
motion features extraction unit 216 detects the second information indi¬
cating the change in the position of the body portion from images of the
20 area of the body portion among the time-series images. The body portion
motion features extraction unit 216 calculates the motion features of the
body portion of the person from the calculated second information indi¬
cating the change in the position of the body portion. For example, the
body portion motion features extraction unit 216 calculates information
25 regarding a motion vector indicating the motion features of the body por-
tion of the person from the second information. [0064]
The head portion motion features extraction unit 215 transmits the information indicating the motion features of the head portion extracted
20

from the time-series images to the feature integration unit 220. The body
portion motion features extraction unit 216 transmits the information indi¬
cating the motion features of the body portion extracted from the time-se¬
ries images to the feature integration unit 220.
5 [0065]
The feature integration unit 220 receives the information indicat¬
ing the motion features of the head portion and the information indicating
the motion features of the body portion from the head portion motion fea¬
tures extraction unit 215 and the body portion motion features extraction
10 unit 216, respectively. The feature integration unit 220 generates one in-
tegration feature by integration of the features relating to the motion of
the head portion of the person and the features relating to the motion of
the body portion. In other words, the feature integration unit 220 gener¬
ates an integration feature relating to a combination of the motion of the
15 head portion of the person and the motion of the body portion of the per-
son from the motion features of the head portion and the motion features
of the body portion. An example of the integration feature is described
below.
[0066]
20 (Example of integration feature)
For example, the integration feature is a vector obtained by con¬
necting a motion vector of the head portion representing the motion fea¬
tures of the head portion and a motion vector of the head portion repre¬
senting the motion features of the body portion. Alternatively, the inte-
25 gration feature is a weighted sum of the motion features of the head por¬
tion and the motion features of the body portion. In the latter case, the
integration feature may be expressed as Expression (1) below. Here, an
identifier of the tracking point of the head portion is represented by a
symbol i (i is an integer of one or more), and an identifier of the tracking
21

point of the body portion is represented by a symbol j (j is an integer of one or more). [0067] [Expression 1]
5 �(�,�,��,��) = ∑(�| |��(�)+���(�)) ■ ■ ■ (1)
�,� [0068]
In Expression (1), F(i, j, xi, yj) is the integration feature, f(i) is the motion features of the tracking point i of the head portion, and g(j) is the motion features of the tracking point j of the body portion. The xi and yj
10 are a weight value of the motion features of the head portion and a weight value of the motion features of the body portion, respectively. A method in which the feature integration unit 220 sets the weight values xi and yj will be described later. [0069]
15 The feature integration unit 220 calculates the integration feature
F in accordance with Expression (1) using the motion features of the head portion and the motion features of the body portion. The feature integra¬tion unit 220 transmits information indicating the calculated integration feature F to the index value calculation unit 230.
20 [0070]
The index value calculation unit 230 receives the information indi¬cating the integration feature from the feature integration unit 220. The index value calculation unit 230 of the second example embodiment calcu¬lates the index value from the integration feature. Specifically, the index
25 value calculation unit 230 inputs the integration feature to the deep learn¬ing neural network and obtains the index value as an output value.
Alternatively, the index value calculation unit 230 calculates a dis¬tance between the integration feature and a standard value. The standard
22

value is a representative value of the integration feature obtained in ac¬
cordance with Expression (1) from a combination of the motion features
of the head portion and the motion features of the body portion of a per¬
son who is not performing spoofing. An example of the standard value
5 will be described later.
[0071]
For example, the distance between the integration feature and the standard value may be a Euclidean distance in a feature space, or may be a distance other than that. In the second example embodiment, the index
10 value calculation unit 230 calculates the distance between the integration
feature and the standard value as the index value. Similarly to the first example embodiment, the index value of the second example embodiment indicates the degree of consistency between the motion of the head por¬tion of the person and the motion of the body portion of the person.
15 [0072]
The index value calculation unit 230 transmits data of the index value calculated to the spoofing determination unit 240. [0073]
The spoofing determination unit 240 receives the data of the index
20 value from the index value calculation unit 230. The spoofing determina-
tion unit 240 determines whether the person is performing spoofing in ac-cordance with a spoofing determination criterion on the basis of the re-ceived index value. The spoofing determination criterion is a threshold value for comparison with the index value. A specific example of the
25 spoofing determination criterion used by the spoofing determination unit
240 will be described later with reference to Fig. 5. [0074]
Further, the spoofing determination unit 240 calculates a “spoofing
23

score” on the basis of the index value calculated by the index value calcu¬
lation unit 230. The spoofing score is a parameter indicating a degree of
possibility that the person is performing spoofing (see Fig. 2). For exam¬
ple, the spoofing score is a reciprocal of the index value . Alternatively,
5 the spoofing score may be obtained by subtracting the index value from
the maximum value of the index value. The maximum value of the index value is an index value when the motion of the head portion of the person and the motion of the body portion of the person completely match each other.
10 [0075]
According to the above definition, as the index value is larger, the spoofing score is smaller, and the possibility that the person is performing spoofing is lower. On the other hand, as the index value is smaller, the spoofing score is larger, and the possibility that a person is performing
15 spoofing is higher.
[0076]
The spoofing determination unit 240 transmits information indicat¬ing a spoofing determination result and the data of the spoofing score to the output unit 20. The output unit 20 outputs the spoofing determination
20 result and the spoofing score. An output destination may be a display de-
vice or a terminal of an operator monitoring illegality. Alternatively, the spoofing determination unit 240 may output only the spoofing determina¬tion result to the output unit 20. [0077]
25 (Weight value)
The feature integration unit 220 needs to set in adva nce the weight values x i , y j (i, j are the identifiers of the tracking points) of the motion features f(i), g (j) to generate the integration feature indicated in Expres¬sion (1).
24

[0078]
The feature integration unit 220 first acquires a combination (here¬
inafter referred to as a group A) of the motion features of the head portion
and the motion features of the body portion detected from a number of
5 sample images of a person who is performing spoofing (the case (a) in
Fig. 2), and a combination (hereinafter referred to as a group B) of the motion features of the head portion and the motion features of the body portion detected from a number of sample images of a person who is not performing spoofing (the case (b) in Fig. 2).
10 [0079]
Alternatively, the feature integration unit 220 may generate these groups A and B from a combination of a number of motion features re-ceived from the head portion motion features extraction unit 215 and the body portion motion features extraction unit 216.
15 [0080]
An integration feature when a combination of the motion features of the head portion and the motion features of the body portion belonging to the group A is substituted as f(i) and g(j) in Expression (1) described above is set as FA(xi, yj). An integration feature when a combination of
20 the motion features of the head portion and the motion features of the
body portion belonging to the group B is substituted as f(i) and g(j) in Ex¬pression (1) is set as FB(xi, yj). [0081]
The feature integration unit 220 sets the weight values x i and y j so
25 that the integration feature F A(xi, yj) and the integration feature F B(xi, yj)
can be distinguished. For example, the feature integration unit 220 sets the weight values x i , y j so that an absolute value of a difference between the FB(xi, yj) and an FB(xm, yn) (m, n each are identifiers of tracking points different from i, j) is always equal to or less than a predetermined
25

threshold value Th regardless of a combination of (i, j, m, n), and so that
an absolute value of a difference between the FA(xi, yj) and the FB(xi, yj)
always exceeds the predetermined threshold value Th regardless of a com¬
bination of (i, j). More specifically, the feature integration unit 220 only
5 needs to, for example, comprehensively obtain values of the integration
feature FA, the integration feature FB while changing values of i, j, m, n,
and obtain the weight values x i , y j that satisfy conditions described above.
In this case, it is assumed that the weight values x i, yj, xm, yn can take
some values of equal to or more than 0 and equal to or less than 1, for ex-
10 ample, and the feature integration unit 220 only needs to perform compre¬
hensive calculation described above while changing values to be substi¬
tuted for the weight values x i, yj, xm, yn.
[0082]
That is, the feature integration unit 220 sets the weight values x i
15 and y j so that integration features of persons who are not performing
spoofing are similar to each other while an integration feature of a person
who is not performing spoofing is not similar to an integration feature of
a person who is performing spoofing. The weight values x i , y j may be dif¬
ferent for each set (i, j) of the tracking point of the head portion and the
20 tracking point of the body portion. Data of the threshold value Th is
stored in advance in the threshold value storage unit 50 illustrated in Fig.
4.
[0083]
The feature integration unit 220 stores data of the set weight val-
25 ues x i , y j in the weight value storage unit 30 illustrated in Fig. 5.
[0084]
(Standard value)
The index value calculation unit 230 according to the second ex-
26

ample embodiment sets the standard value in advance to calculate the in¬
dex value described above. As described above, the standard value is the
representative value of the integration feature obtained in accordance with
Expression (1) from the combination of the motion features of the head
5 portion and the motion features of the body portion of the person who is
not performing spoofing. For example, the standard value may be a statis¬
tical value such as an average of the integration feature F B(x i, y j ) (i, j are
the identifiers of the tracking points) obtained from a plurality of sample
images of the person who is not performing spoofing.
10 [0085]
The index value calculation unit 230 stores data of the set standard value in the standard value storage unit 40 illustrated in Fig. 5. [0086]
(Example of spoofing determination criterion)
15 An example of a method in which the spoofing determination unit
240 determines spoofing will be described with reference to Fig. 5. Fig. 5
is a graph illustrating a feature space. The feature space is an N-dimen-
sional (N > 1) Euclidean space. Fig. 5 illustrates the N-dimensional (N >
1) Euclidean space in three dimensions. Axes of the feature space respec-
20 tively correspond to feature values different from each other included in
the integration feature.
[0087]
In the graph illustrated in Fig. 5, a standard value is indicated . As
described above, the standard value is a statistical level value of the int e-
25 gration feature obtained from the plurality of sample images of the person
who is not performing spoofing. With the standard value as the center,
mesh is applied to the outer side of a range up to a certain distance
(threshold value Th). A distance from the standard value to the integra-
27

tion feature F is represented by d. In Fig. 5, d < Th. That is, in the inte¬
gration feature F, the integration feature is within the range up to the cer¬
tain distance (threshold value Th) with the standard value as the center.
[0088]
5 In the feature space illustrated in Fig. 5, if the integration feature
F is within the range up to the certain distance (threshold value Th) with the standard value as the center, the spoofing determination unit 240 de-termines that the person is authentic (that is, the person is not performing spoofing). On the other hand, if the integration feature is out of the range
10 up to the certain distance (threshold value Th) with the standard value as
the center, the spoofing determination unit 240 determines that a person is
performing spoofing.
[0089]
Regarding Fig. 5, the closer the integration feature F is to the
15 standard value, the lower the possibility that the person is spoofing an-
other person. Conversely, as the integration feature F deviates from the standard value, there is a higher possibility that the person is spoofing an-other person. Thus, when the distance d (corresponding to the index value) between the integration feature and the standard value is equal to
20 or less than the threshold value Th, the spoofing determination unit 240
determines that the person is not performing spoofing. On the other hand, when the distance d between the integration feature and the standard value exceeds the threshold value Th, the spoofing determination unit 240 deter¬mines that the person is performing spoofing.
25 [0090]
(Operation of image processing apparatus 200)
Operation executed by the image processing apparatus 200 accord¬ing to the second example embodiment will be described with reference to
28

Fig. 6. Fig. 6 is a flowchart illustrating a flow of a process from acquisi¬
tion of an image by the image processing apparatus 200 to spoofing deter¬
mination.
[0091]
5 As illustrated in Fig. 6, each of the head portion detection unit 211
and the body portion detection unit 212 acquires a plurality of time-series images from the input unit 10 (S201). [0092]
For example, the head portion detection unit 211 and the body por-
10 tion detection unit 212 acquire data of frame images of a moving image
imaged within a predetermined period (for example, 10 seconds) by one
camera.
[0093]
The head portion detection unit 211 detects the head portion of the
15 person from each of the time-series images acquired. The body portion
detection unit 212 detects the body portion of the person from each of the
same time-series images acquired (S202).
[0094]
In step S202, the head portion detection unit 211 may first extract
20 an area of the head portion of the person from each image by pattern
matching or the like, and then detect a tracking point of the head portion
from the extracted area of the head portion of the person. Similarly, the
body portion detection unit 212 may first extract an area of the body por¬
tion of the person from each image by pattern matching or the like, and
25 then detect a tracking point of the body portion from the extracted area of
the body portion of the person. [0095]
The head portion detection unit 211 transmits a detection result of the head portion of the person together with data of the time -series images
29

to the head portion motion features extraction unit 215. The body portion
detection unit 212 transmits a detection result of the body portion of the
person to the body portion motion features extraction unit 216 together
with the data of the time-series images.
5 [0096]
The head portion motion features extraction unit 215 receives the
detection result of the head portion of the person together with the data of
the time-series images from the head portion detection unit 211. The
body portion motion features extraction unit 216 receives the detection re-
10 sult of the body portion of the person together with the data of the time -
series images from the body portion detection unit 212.
[0097]
The head portion motion features extraction unit 215 extracts the
motion features of the head portion of the person from the time-series im-
15 ages. The body portion motion features extraction unit 216 extracts the
motion features of the body portion of the person from the time -series im-ages (S203). [0098]
The head portion motion features extraction unit 215 transmits in-
20 formation indicating the motion features of the head portion detected to
the feature integration unit 220. The body portion motion features extrac¬
tion unit 216 transmits information indicating the motion features of the
body portion detected to the feature integration unit 220.
[0099]
25 The feature integration unit 220 receives information indicating
the motion features of the head portion and information indicating the mo-tion features of the body portion from the head portion motion features extraction unit 215 and the body portion motion features extraction unit 216, respectively.
30

[0100]
The feature integration unit 220 generates an integration feature by
integration of the motion features of the head portion and the motion fea¬
tures of the body portion (S204).
5 [0101]
Specifically, using the weight values x i and y j (i, j are the identifi¬ers of the tracking points of the head, body, respectively) stored in ad¬vance in the weight value storage unit 30 (see Fig. 4), the feature integra¬tion unit 220 generates the integration feature F(i, j, x i , y j ) associated to
10 the weight values x i and y j , the motion features f(i) of the head portion,
and the motion features g(j) of the body portion, in accordance with Ex¬pression (1) described above. The feature integration unit 220 transmits the information indicating the integration feature generated to the index value calculation unit 230.
15 [0102]
The index value calculation unit 230 receives the information indi-cating the integration feature from the feature integration unit 220 . The index value calculation unit 230 acquires the standard value stored in ad-vance in the standard value storage unit 40 (see Fig. 5), and calculates, as
20 the index value, the distance d (see Fig. 5) between the integration feature
and the standard value in the feature space (S205). [0103]
Alternatively, in step S205, the index value calculation unit 230 may calculate an index value based on the distance d. For example, the
25 index value calculation unit 230 calculates a value of a function depend-
ing on the distance d as the index value. The index value calculation unit 230 transmits data of the index value calculated to the spoofing determi¬nation unit 240. [0104]
31

The spoofing determination unit 240 receives the data of the index
value from the index value calculation unit 230. The spoofing determina¬
tion unit 240 refers to the threshold value storage unit 50 to acquire the
threshold value Th. The spoofing determination unit 240 determines
5 whether the index value is equal to or less than the threshold value Th
(see Fig. 5) (S206). [0105]
After step S206, the spoofing determination unit 240 determines
presence or absence of spoofing as follows.
10 [0106]
When the distance d (corresponding to the index value) between
the integration feature and the standard value is equal to or less than the
threshold value Th (Yes in S206), the spoofing determination unit 240 de¬
termines that the person is not performing spoofing (S207A).
15 [0107]
On the other hand, when the distance d between the integration
feature and the standard value exceeds the threshold value Th (No in
S206), the spoofing determination unit 240 determines that the person is
performing spoofing (S207B).
20 [0108]
Thereafter, the spoofing determination unit 240 outputs a spoofing
determination result in step S207A or S207B, and data of the spoofing
score (S208).
[0109]
25 An output destination may be a display device or a terminal of an
operator. In step S208, the spoofing determination unit 240 may output the index value calculated by the index value calculation unit 230 together with the spoofing determination result and the data of the spoofing score. [0110]
32

After the process described above, the spoofing determination pro¬
cess executed by the image processing apparatus 200 according to the sec¬
ond example embodiment ends.
[0111]
5 (Effects of present example embodiment)
According to the configuration of the present example embodi¬
ment, the motion detection unit 210 detects, from the time-series images,
the feature relating to the motion of the head portion of the person and the
feature relating to the motion of the body portion that is a region other
10 than the head portion of the person. The index value calculation unit 230
calculates the index value indicating the degree of consistency between
the feature relating to the motion of the head portion of the person a nd the
feature relating to the motion of the body portion.
[0112]
15 When the person is performing spoofing using the face image dis-
played on the print or display, a contradiction occurs between the feature
relating to the motion of the head portion of the person and the feature re¬
lating to the motion of the body portion, and the consistency is lost . In
the second example embodiment, spoofing can be accurately identified
20 from the index value indicating the degree of consistency between the fea-
ture relating to the motion of the head portion of the person and the fea¬ture relating to the motion of the body portion. [0113]
The spoofing determination unit 240 determines whether the per-
25 son is spoofing another person on the basis of the index value. The index
value indicates the degree of consistency between the motion of the head
portion of the person and the motion of the body portion of the person.
For that reason, the spoofing determination unit 240 can determine with
high accuracy whether the person is spoofing another person.
33

[0114]
Further, the index value calculation unit 230 calculates the index
value on the basis of the integration feature generated by the feature inte¬
gration unit 220 and the standard value set in advance. The integration
5 feature may be a weighted sum of the feature relating to the motion of the
head portion of the person and the feature relating to the motion of the
body portion of the person. Since the standard value is the representative
value of the integration feature obtained from the combination of the mo¬
tion features of the head portion and the motion features of the body por-
10 tion of the person who is not performing spoofing, the closer the integra¬
tion feature is to the standard value, the higher the possibility that the
person is not performing spoofing. By using the integration feature and
the standard value, it is possible to calculate the index value indicating
with high accuracy the degree of consistency between the feature relating
15 to the motion of the head portion of the person and the feature relating to
the motion of the body portion.
[0115]
[Third Example Embodiment]
A third example embodiment will be described below with refer-
20 ence to Fig. 7.
[0116]
(Regarding Hardware Configuration)
Components of the image processing apparatuses 100, 200 de¬
scribed in the first and second example embodiments each indicate a block
25 of a functional unit. Some or all of these components are implemented by
an information processing apparatus 900 as illustrated in Fig. 7, for exam-ple. Fig. 7 is a block diagram illustrating an example of a hardware con¬figuration of the information processing apparatus 900. [0117]
34

As illustrated in Fig. 7, the information processing apparatus 900 includes the following configuration as an example. [0118]
• Central Processing Unit (CPU) 901 5 • Read Only Memory (ROM) 902
• Random Access Memory (RAM) 903
• Program 904 loaded into RAM 903
• Storage device 905 storing program 904
• Drive device 907 performing read/write from/to recording me-
10 dium 906
• Communication interface 908 for connecting to communication network 909
• Input/output interface 910 for inputting/outputting data
• Bus 911 for connecting each component
15 The components of the image processing apparatuses 100, 200 de-
scribed in the first and second example embodiments are implemented by the CPU 901 reading and executing the program 904 that implements functions of these components. The program 904 that implements the functions of the components is stored in the storage device 905 or the
20 ROM 902 in advance, for example, and the CPU 901 loads the program
into the RAM 903 and executes the program as necessary. The program 904 may be supplied to the CPU 901 via the communication network 909, or may be stored in the recording medium 906 in advance, and the drive device 907 may read the program and supply the program to the CPU 901.
25 [0119]
(Effects of present example embodiment)
According to the configuration of the present example embodi¬ment, the image processing apparatus described in the above ex ample em¬bodiment is implemented as hardware. Thus, effects similar to the effects
35

described in the above example embodiment can be obtained.
In the above, the present embodiments have been described with
reference to the example embodiments; however, the present embodiments
are not limited to the above-described example embodiments. Various
5 modifications that can be understood by those skilled in the art can be
made to the configuration and details of the present embodiments within
the scope of the present disclosure.
This application is based upon and claims the benefit of priority
from Japanese patent application No. 2019-055164, filed on March 22,
10 2019, the disclosure of which is incorporated herein in its entirety by ref-
erence.
[Reference signs List]
[0120]
100 Image processing apparatus
15 104 Motion detection unit
106 Index value calculation unit
200 Image processing apparatus
extraction unit extraction unit
210 Motion detection unit
211 Head portion detection unit 20 212 Body portion detection unit

215 Head portion motion features
216 Body portion motion features 220 Feature integration unit 230 Index value calculation unit
25 240 Spoofing determination unit

WE CLAIM:

[Claim 1]An image processing device comprising:
a motion detection means for detecting, from time-series images, a
5 feature relating to a motion of a head portion of a person and a feature re-
lating to a motion of a body portion that is a region other than the head portion of the person; and
an index value calculation means for calculating an index value in¬
dicating a degree of consistency between the feature relating to the mo-
10 tion of the head portion of the person and the feature relating to the mo¬
tion of the body portion of the person.
[Claim 2]
The image processing device according to claim 1, wherein
the index value is any one of:
15 a distance between a motion vector indicating a direction in which
the head portion of the person moves and a motion vector indicating a di¬rection in which the body portion of the person moves;
an angle between the motion vector indicating the direction in
which the head portion of the person moves and the motion vector indicat-
20 ing the direction in which the body portion of the person moves; or
an output value of a function including a deep learning neural net¬
work, the function having the motion vector indicating the direction in
which the head portion of the person moves and the motion vector indicat¬
ing the direction in which the body portion of the person moves as inputs.
25 [Claim 3]
The image processing device according to claim 1 or 2, wherein
the motion detection means includes a feature integration means for generating one integration feature by integration of the feature relat¬ing to the motion of the head portion of the person and the feature relating
37

to the motion of the body portion of the person, and
the index value calculation means calculates the index value from
the integration feature.
[Claim 4]
5 The image processing device according to claim 3, wherein
the feature integration means calculates, as the integration feature,
a weighted sum of the feature relating to the motion of the head portion of
the person and the feature relating to the motion of the body portion of
the person.
10 [Claim 5]
The image processing device according to any one of claims 1 to 4, further comprising
a spoofing determination means for determining whether the per¬
son is spoofing another person, based on the index value.
15 [Claim 6]
The image processing device according to any one of claims 1 to 5, wherein
the motion detection means
includes:
20 a head portion detection means for detecting the head portion of
the person from the time-series images;
a body portion detection means for detecting the region other than the head portion of the person from the time-series images;
a head portion motion features extraction means for extracting the
25 feature relating to the motion of the head portion of the person from a de-
tection result of the head portion of the person in the time-series images; and
a body portion motion features extraction means for extracting the feature relating to the motion of the body portion of the person from the
38

detection result of the head portion of the person in the time-series im-ages. [Claim 7]
The image processing device according to any one of claims 1 to 6,
5 wherein
the feature relating to the motion of the head portion of the person includes a motion vector of the head portion of the person, and
the feature relating to the motion of the body portion of the person
includes a motion vector of the body portion of the person.
10 [Claim 8]
The image processing device according to any one of claims 1 to 7, wherein
the motion detection means detects the feature relating to the mo¬
tion of the head portion of the person and the feature relating to the mo-
15 tion of the region other than the head portion of the person from the time -
series images by using a neural network.
[Claim 9]
An image processing method comprising:
detecting, from time-series images, a feature relating to a motion
20 of a head portion of a person and a feature relating to a motion of a body
portion that is a region other than the head portion of the person; and
calculating an index value indicating a degree of consistency be¬
tween the feature relating to the motion of the head portion of the person
and the feature relating to the motion of the body portion of the person.
25 [Claim 10]
A recording medium storing a program for causing a computer to execute:
detecting, from time-series images, a feature relating to a motion of a head portion of a person and a feature relating to a motion of a body
39

portion that is a region other than the head portion of the person; and
calculating an index value indicating a degree of consistency be¬tween the feature relating to the motion of the head portion of the person and the feature relating to the motion of the body portion of the person.