Title of Invention: Abnormal Physical Condition Determination System, Abnormal Physical Condition Determining Method, and Computer Program
Technical field
[0001]
　TECHNICAL FIELD The present invention relates to a technical field of an abnormality determination system, an abnormality determination method, and a computer program for determining abnormality of a subject's physical condition.
Background technology
[0002]
　As this type of system, a system that determines the condition of a subject based on an image of the subject is known. For example, Patent Literature 1 discloses a technique for estimating the state of a driver from the position of the face, the direction of the face, the line of sight, etc. obtained by analyzing the face image. Patent Literature 2 discloses a technique for detecting a face position from a face image and estimating the state of a driver using the degree of eye opening, line-of-sight direction, face orientation, and the like.
[0003]
　As another related technique, for example, Patent Document 3 discloses a technique for detecting a feature point vector from a plurality of feature points extracted from face image data.
prior art documents
patent literature
[0004]
Patent Document 1: International Publication No. 2017/209225
Patent Document 2: International Publication No. 2017/208529
Patent Document 3: International Publication No. 2016/013090
SUMMARY OF THE INVENTION
Problems to be Solved by the Invention
[0005]
　If an attempt is made to determine the state of a subject by directly using features such as the position of the face, the orientation of the face, and the direction of the line of sight, it is assumed that the determination will be affected by, for example, individual differences and environmental variations such as lighting conditions. In this case, depending on the environment in which the state is determined, there is a possibility that the accuracy of the determination may be degraded. That is, the techniques described in Patent Documents 1, 2, and 3 have a technical problem that the subject's condition cannot be accurately determined depending on the situation.
[0006]
　SUMMARY OF THE INVENTION It is an object of the present invention to provide an abnormal physical condition determination system, an abnormal physical condition determination method, and a computer program capable of appropriately determining an abnormal physical condition of a subject. do.
Means to solve problems
[0007]
　One aspect of the physical condition abnormality determination system of the present invention is an extraction means for extracting a plurality of feature amounts indicating the condition of the subject from the image of the subject, and an accumulation means for accumulating the plurality of feature amounts as time series data. and calculating means for calculating the relationship between each feature quantity from the plurality of feature quantities accumulated in the accumulating means, and determination means for determining the physical condition abnormality of the subject based on the relationship. Prepare.
[0008]
　One aspect of the physical condition abnormality determination method of the present invention is to extract a plurality of feature amounts indicating the condition of the subject from the image of the subject, accumulate the plurality of feature amounts as time-series data, and store the accumulated From the plurality of feature amounts, the relationship between each feature amount is calculated, and the subject's physical condition abnormality is determined based on the relationship.
[0009]
　One aspect of the computer program of the present invention extracts a plurality of feature amounts indicating the state of the subject from an image of the subject, accumulates the plurality of feature amounts as time-series data, and stores the accumulated plurality of feature amounts. A computer is operated so as to calculate the relationship between each feature amount from the feature amount and determine the physical condition abnormality of the subject based on the relationship.
Effect of the invention
[0010]
　According to one aspect of each of the abnormal physical condition determination system, the abnormal physical condition determination method, and the computer program described above, it is possible to appropriately determine the abnormal condition of the subject.
Brief description of the drawing
[0011]
1 is a block diagram showing the overall configuration of a physical condition abnormality determination system according to a first embodiment; FIG.
2 is a block diagram showing the overall configuration of a physical condition abnormality determination system according to a modification; FIG.
3 is a block diagram showing the hardware configuration of the physical condition abnormality determination system according to the first embodiment; FIG.
4 is a flow chart showing the operation flow of the physical condition abnormality determination system according to the first embodiment;
5 is a table showing specific examples of feature vectors; FIG.
6 is a block diagram showing the overall configuration of a physical condition abnormality determination system according to a second embodiment; FIG.
7 is a schematic diagram showing the interior configuration of a vehicle to which a physical condition abnormality determination system according to a second embodiment is applied; FIG.
8 is a diagram showing an example of a display area of ​​a display unit; FIG.
9 is a diagram (Part 1) showing a display example in the display area when car sickness occurs. FIG.
10 is a diagram (part 2) showing a display example in the display area when car sickness occurs. FIG.
11] A diagram showing an example of an icon indicating the degree of car sickness. [FIG.
12 is a diagram showing another display example in the display area of ​​the display unit; FIG.
13 is a block diagram showing the overall configuration of a physical condition abnormality determination system according to a third embodiment; FIG.
14 is a block diagram showing the overall configuration of a physical condition abnormality determination system according to a fourth embodiment; FIG.
MODE FOR CARRYING OUT THE INVENTION
[0012]
　Hereinafter, embodiments of an abnormality determination system, an abnormality determination method, and a computer program will be described with reference to the drawings.
[0013]

　A physical condition abnormality determination system according to a first embodiment will be described with reference to FIGS. 1 to 5. FIG. In the following, an example will be described in which the abnormal physical condition determination system is a device for determining abnormal physical condition (specifically, motion sickness) of a passenger in a vehicle.
[0014]
　(System Configuration)
　First, the overall configuration of the physical condition abnormality determination system according to the first embodiment will be described with reference to FIGS. 1 and 2. FIG. FIG. 1 is a block diagram showing the overall configuration of the physical condition abnormality determination system according to the first embodiment. FIG. 2 is a block diagram showing the overall configuration of a physical condition abnormality determination system according to a modification.
[0015]
　As shown in FIG. 1 , the physical condition abnormality determination system 1 according to the first embodiment is installed in a vehicle 10 having a vehicle interior camera 50 . The physical condition abnormality determination system 1 includes, as functional blocks for realizing its functions, an image acquisition unit 101, a subject detection unit 102, a feature extraction unit 103, a feature storage unit 104, a feature relationship calculation unit 105, and a physical condition abnormality determination unit 106 .
[0016]
　The image acquisition unit 101 is configured to be able to acquire an image captured by the vehicle interior camera 50 (specifically, an image in which a passenger of the vehicle 10 is captured). The image acquired by the image acquisition unit 101 is configured to be output to the subject detection unit 102 .
[0017]
　The subject detection unit 102 is configured to be able to detect a subject whose physical condition is abnormal from the image acquired by the image acquisition unit 101 . The target person detection unit 102 detects, for example, an area in which the target person is shown (for example, the target person's face area, etc.) from the captured image. As a specific method for detecting a target person from a captured image, an existing technique can be appropriately adopted, so detailed description thereof will be omitted here. The detection result of the subject detection unit 102 is configured to be output to the feature extraction unit 103 .
[0018]
　The feature extraction unit 103 is configured to be able to extract a feature amount for determining the subject's physical condition abnormality from the area detected by the subject detection unit 102 . The feature amount extracted by the feature extraction unit 103 includes, for example, the position of the passenger's pupils, the orientation of the face, the direction of the line of sight, the color of the face, the position of facial feature points, and the degree of opening of the eyelids. A feature extraction unit 103 extracts a plurality of types of feature amounts from the passenger. The feature amount extracted by the feature extraction unit 103 is configured to be output to each of the feature storage unit 104 and the feature relationship calculation unit 105 .
[0019]
　The feature storage unit 104 is configured to store the feature amount extracted by the feature extraction unit 103 as time-series data. The feature accumulating unit 104 accumulates the feature amount for each frame arranged in time series, for example. The feature amount accumulated in the feature accumulation unit 104 is configured to be output to the feature relation calculation unit 105 as appropriate.
[0020]
　The feature relationship calculation unit 105 is configured to be able to calculate relationships between feature amounts (that is, relationships between feature amounts of different types) from time-series data of a plurality of types of feature amounts. More specifically, the feature relation calculation unit 105 calculates the feature amount extracted by the feature extraction unit 103 (in other words, the current feature amount) and the feature amount read out from the feature storage unit 104 (in other words, the past feature amount). feature amount), the relationship between each feature amount is calculated. The relationship between feature amounts is calculated as a feature vector having multiple dimensions, for example. The relationship between the feature amounts calculated by the feature relationship calculation unit 105 is configured to be output to the physical condition abnormality determination unit 106 .
[0021]
　The physical condition determination unit 106 determines the physical condition of the passenger based on the relationship between the feature amounts calculated by the feature relationship calculation unit 105 . The physical condition determination unit 106 determines, for example, whether or not the passenger has an abnormal physical condition, or determines the degree of the physical condition of the passenger. It should be noted that the determination of the physical condition abnormality can be determined by setting in advance the relationship between the relationship between the feature amount and the physical condition abnormality. For example, a threshold may be set for the feature vector, and if the calculated feature vector exceeds the threshold, it may be determined that an abnormality in physical condition has occurred. Also, the physical condition abnormality determination unit 106 may be configured to learn a threshold for determination. For example, the physical condition abnormality determination unit 106 may perform learning using correct data obtained by the passenger's input.
[0022]
　As shown in FIG. 2 , the physical condition abnormality determination system 1 may be provided in an external server 20 located outside the vehicle 10 . That is, the physical condition abnormality determination system 1 does not necessarily have to be mounted on the vehicle 10 . Note that not all the components of the abnormality determination system 1 but some components of the abnormality determination system 1 may be provided in the external server 20 .
[0023]
　(Hardware Configuration)
　Next, the hardware configuration of the physical condition abnormality determination system 1 according to the first embodiment will be described with reference to FIG. FIG. 3 is a block diagram showing the hardware configuration of the physical condition abnormality determination system according to the first embodiment.
[0024]
　As shown in FIG. 3, the physical condition abnormality determination system 1 according to the first embodiment includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, and a storage device 14. and The physical condition abnormality determination system 1 may further include an input device 15 and an output device 16 . The CPU 11 , RAM 12 , ROM 13 , storage device 14 , input device 15 and output device 16 are connected via a data bus 17 .
[0025]
　The CPU 11 reads computer programs. For example, the CPU 11 is configured to read computer programs stored in at least one of the RAM 12, ROM 13 and storage device 14. FIG. Alternatively, the CPU 11 may read a computer program stored in a computer-readable recording medium using a recording medium reader (not shown). The CPU 11 may acquire (that is, read) a computer program from a device (not shown) arranged outside the physical condition abnormality determination system 1 via a network interface. The CPU 11 controls the RAM 12, the storage device 14, the input device 15 and the output device 16 by executing the read computer program. Particularly in this embodiment, when the computer program read by the CPU 11 is executed, a functional block for judging the physical condition abnormality is realized in the CPU 11 .
[0026]
　The RAM 12 temporarily stores computer programs executed by the CPU 11 . RAM 12 temporarily stores data temporarily used by CPU 11 while CPU 11 is executing a computer program. The RAM 12 may be, for example, a D-RAM (Dynamic RAM).
[0027]
　The ROM 13 stores computer programs executed by the CPU 11 . The ROM 13 may also store other fixed data. The ROM 13 may be, for example, a P-ROM (Programmable ROM).
[0028]
　The storage device 14 stores data that the physical condition abnormality determination system 1 saves for a long period of time. The storage device 14 may operate as a temporary storage device for the CPU 11 . The storage device 14 may include, for example, at least one of a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device.
[0029]
　The input device 15 is a device that receives input instructions from the user of the physical condition abnormality determination system 1 . Input device 15 may include, for example, at least one of a keyboard, mouse, and touch panel.
[0030]
　The output device 16 is a device that outputs information about the physical condition abnormality determination system 1 to the outside. For example, the output device 16 may be a display device (for example, a display) capable of displaying information regarding the physical condition abnormality determination system 1 .
[0031]
　(Flow of Operation)
　Next, the flow of operation of the physical condition abnormality determination system 1 according to the first embodiment will be described with reference to FIG. FIG. 4 is a flow chart showing the operation flow of the physical condition abnormality determination system according to the first embodiment.
[0032]
　As shown in FIG. 4, when the physical condition abnormality determination system 1 according to the first embodiment operates, the image acquisition unit 101 first acquires an image of the passenger captured by the vehicle interior camera 50 (step S101). Then, the target person detection unit 102 detects a face area from the image of the passenger (step S102). It should be noted that when the feature amount is detected from areas other than the face area, other areas corresponding to the feature amount may be detected.
[0033]
　Subsequently, the feature extraction unit 103 extracts feature amounts from the face area (step S103). Then, the feature accumulation unit 104 accumulates (that is, stores) the extracted feature amount (step S104).
[0034]
　Subsequently, the feature relation calculation unit 105 determines whether or not the feature amount for a predetermined number of frames is accumulated in the feature 2201 accumulation unit 104 (step S105). Note that the “predetermined frame” here is a value corresponding to a sufficient amount of feature amounts for calculating the relationship between the feature amounts, and is an optimal value in advance (for example, the number of frames for 5 seconds). is set. If it is determined that the feature amount is not accumulated for the predetermined number of frames (step S105: NO), the process is executed again from step S101. That is, the process of extracting the feature amount from the image of the passenger and accumulating it is repeated.
[0035]
　On the other hand, if it is determined that the feature amount is accumulated for the predetermined number of frames (step S105: YES), the feature relationship calculation unit 105 calculates the relationship of the feature amount (step S106). Then, the physical condition determination unit 106 determines the physical condition of the passenger (here, motion sickness) based on the relationship between the calculated feature amounts (step S107).
[0036]
　The series of processes described above are executed while the vehicle 10 is running. The physical condition abnormality determination system 1 may start operating, for example, when the vehicle 10 starts running. Alternatively, the operation may be started after a predetermined period of time (a period of time during which it is assumed that the passenger's physical condition begins to become abnormal) after the start of running. Here, the predetermined time may be set arbitrarily. Alternatively, the time from the start of running until the physical condition of the passenger begins to appear abnormal may be stored as history information, and the setting may be made based on this history information. By doing so, it is possible to reduce unnecessary processing during the period in which the passenger does not have an abnormality in physical condition. Alternatively, the physical condition abnormality determination system 1 may start operating at the timing when the passenger's face can be detected normally or at the timing when the passenger gets into the vehicle. By doing so, it is possible to track the face of the passenger, so it is possible to prevent a situation in which the face of the passenger cannot be detected even though the passenger has an abnormality in physical condition. . Alternatively, the operation may be started based on the behavior information of the vehicle (for example, at the timing when the threshold value of the acceleration sensor becomes equal to or greater than a predetermined threshold value). By doing so, it is possible to reduce unnecessary processing when the vehicle behaves in a manner that is unlikely to cause an abnormality in physical condition.
[0037]
　Further, the physical condition abnormality determination system 1 ends its operation when the vehicle 10 finishes running, for example. Alternatively, the operation may be terminated based on vehicle behavior information (for example, at timing when the threshold value of the acceleration sensor becomes equal to or less than a predetermined threshold value).
[0038]
　(Specific example of feature vector)
　Next, referring to FIG. 5, the feature vector used in the physical condition abnormality determination system 1 according to the first embodiment (ie, the relationship ) will be specifically explained. FIG. 5 is a table showing specific examples of feature vectors.
[0039]
　In FIG. 5, the feature extracting unit 103 extracts the pupil positions of the passenger's right and left eyes, face orientation angles (pan/tilt/roll directions), and line-of-sight angles (horizontal/vertical directions) as feature quantities. and In this case, the feature relationship calculation unit 105 calculates the relationship between the plurality of types of feature amounts as a multidimensional feature vector.
[0040]
　Specifically, the feature relation calculation unit 105 calculates (1) the correlation between the face position and the position of the right eye pupil, (2) the correlation between the face position and the position of the left eye pupil, (3) the panning direction of the face and the horizontal line of sight. (4) Correlation between face tilt direction and line-of-sight vertical direction, (5) ratio of movement speed between face center of gravity position and right eye position, (6) movement speed between face center of gravity position and left eye position. 8-dimensional feature vectors are calculated including the ratio, (7) the angular velocity ratio between the face pan direction and the line-of-sight horizontal direction, and (8) the angular velocity ratio between the face tilt direction and the line-of-sight vertical direction. Taking (1) as an example, the correlation is a value obtained by dividing the covariance between the face position and the right eye pupil by the product of the standard deviation of the face position and the standard deviation of the right eye pupil.
[0041]
　Each element of the feature vector may be appropriately weighted. The weight in this case may be determined according to the state of the face estimated from the image of the face region. Specifically, when the passenger wears glasses, the accuracy of the feature amount obtained for the eyes is considered to decrease, so the weight of the feature amount for the eyes may be reduced. Also, the feature vector does not have to be eight-dimensional, and may have an appropriately set number of dimensions.
[0042]
　The physical condition determination unit 106 determines that the passenger has an abnormal physical condition based on the relationship between the calculated feature amounts. Specifically, when the correlation and the ratio calculated by the characteristic relation calculation unit 105 are equal to or greater than a predetermined value, it is determined that the passenger is in an abnormal physical condition. When the eight-dimensional feature vectors including the above (1) to (8) are calculated, if one or more of the correlations or ratios are equal to or greater than a predetermined value, it is determined that the passenger is in an abnormal physical condition. Alternatively, it may be determined that the occupant is in an abnormal physical condition when all eight correlations and ratios are equal to or greater than corresponding predetermined values.
[0043]
　When motion sickness occurs, the passenger feels dizziness due to rotation or dizziness when opening the eyes. On the other hand, the feature vectors (1) to (8) described above are normally calculated as the relationship between pairs of feature amounts that change in the same way. Therefore, according to the feature vector described above, it is possible to determine a situation in which the movement of the face and eyes do not correspond due to dizziness, and it is possible to suitably determine the occurrence of motion sickness.
[0044]
　Note that the feature vector described above is merely an example, and other relationships of feature amounts may be used. For example, a pair of feature amounts for which the relationship is to be calculated may be determined according to the type of physical condition to be determined.
[0045]
　(Technical Effects)
　Next, technical effects obtained by the system 1 for judging abnormal physical condition according to the first embodiment will be described.
[0046]
　As described with reference to FIGS. 1 to 5, according to the physical condition abnormality determination system 1 according to the first embodiment, motion sickness of the passenger can be determined based on the feature amount extracted from the image of the passenger. can. Especially in this embodiment, motion sickness is determined based on the relationship between a plurality of feature values, instead of directly using the extracted feature values. Therefore, for example, the effects of environmental variations such as individual differences and lighting conditions can be suppressed, and motion sickness can be determined with high accuracy. Moreover, since the feature amount, which is time-series data, is used, unlike the case where the feature amount is extracted from only one image, changes in the feature amount per unit time can also be taken into consideration. Therefore, motion sickness can be determined more accurately.
[0047]
　In the above-described embodiment, an example of determining the motion sickness of the passengers of the vehicle 10 has been described. good. Furthermore, it is also possible to determine abnormal physical condition other than motion sickness. Specifically, determination regarding dizziness (so-called VR sickness) that may occur when using a VR (Virtual Reality) device may be performed. By using the feature amount related to the subject's face as in the present embodiment, it is possible to appropriately determine a physical condition abnormality, especially accompanied by dizziness. However, by using an appropriate relationship between feature amounts, it can also be applied to determination of other physical abnormalities that are not accompanied by dizziness.
[0048]

　Next, a physical condition abnormality determination system 1 according to a second embodiment will be described with reference to FIGS. 6 to 12. FIG. It should be noted that the second embodiment differs from the already-described first embodiment only in a part of configuration and operation, and other parts are generally the same. Therefore, in the following, portions different from the first embodiment will be described in detail, and descriptions of other overlapping portions will be omitted as appropriate.
[0049]
　(System Configuration)
　First, the overall configuration of the physical condition abnormality determination system 1 according to the second embodiment will be described with reference to FIGS. 6 and 7. FIG. FIG. 6 is a block diagram showing the overall configuration of the physical condition abnormality determination system according to the second embodiment. FIG. 7 is a schematic diagram showing the interior configuration of a vehicle to which the physical condition abnormality determination system according to the second embodiment is applied. In FIG. 6, the same symbols are assigned to the same components as those shown in FIG.
[0050]
　As shown in FIG. 6, the physical condition abnormality determination system 1 according to the second embodiment includes a determination result output unit 107 in addition to the components of the physical condition abnormality determination system 1 according to the first embodiment (see FIG. 1). ing.
[0051]
　The physical condition determination unit 106 may identify the cause of the physical condition of the passenger from the information indicating the behavior of the vehicle 10 . In this case, the physical condition determination unit 106 may be configured to be able to acquire information indicating the behavior of the vehicle 10 from a sensor (eg, an acceleration sensor, etc.) provided on the vehicle 10 (not shown). Further, the physical condition determination unit 106 may identify the cause of the physical condition of the passenger from the analysis result of the image of the passenger. In this case, the physical condition determination unit 106 may be configured to be able to extract a feature amount that can determine the cause of the physical condition abnormality from an image or the like including the whole body of the passenger. In addition, an unillustrated physical condition cause identification unit may identify the cause of the physical condition abnormality of the passenger.
[0052]
　For example, when it is determined that the passenger is in poor physical condition, the physical condition determination unit 106 acquires the acceleration of the vehicle at the determined timing. The physical condition determination unit 106 compares the acquired acceleration with a pre-stored threshold value (for example, acceleration likely to cause physical condition abnormality) to determine whether the physical condition abnormality of the passenger is caused by the acceleration of the vehicle 10. determine whether Note that the acceleration to be acquired may be the acceleration at the timing a predetermined period before the timing at which the physical condition is determined to be abnormal. Alternatively, the acceleration during a predetermined period up to the timing when the physical condition is determined to be abnormal may be used. Further, instead of or in addition to acceleration, velocity, angular velocity, yaw angular velocity, yaw angular acceleration, pitch angular velocity, pitch angular acceleration, roll angular velocity, and roll angular acceleration You may obtain at least one of The cause may be specified based on at least one of them, or may be specified based on all of them.
[0053]
　Note that the physical condition determination unit 106 may identify the cause of the physical condition without comparing the detected acceleration or the like with a threshold value. For example, with respect to the acceleration detected at the timing when the physical condition is determined to be abnormal, it can be determined that the acceleration itself is the cause of the physical condition. Therefore, the acceleration or the like detected at the timing when the physical condition is determined to be abnormal may be identified as the cause of the physical condition.
[0054]
　Further, for example, when it is determined that the passenger is in an unhealthy physical condition, the unhealthy physical condition determination unit 106 determines what kind of behavior the passenger is doing (for example, reading, eating and drinking, etc.) from the camera image at the determined timing. Identify what you are taking. The abnormal physical condition determination unit 106 determines whether or not the identified behavior corresponds to a pre-stored cause of abnormal physical condition (for example, table information of behaviors likely to cause abnormal physical condition). It is determined whether or not it is due to the action of Note that the camera image used to identify the cause may be captured at a timing a predetermined period before the timing at which the physical condition is determined to be abnormal. Alternatively, the image may be taken during a predetermined period up to the timing when the physical condition is determined to be abnormal. Furthermore, the table information of behaviors that are likely to cause an abnormality in physical condition may be associated with the degree of an abnormality in physical condition for each action. Alternatively, a plurality of behaviors may be obtained and the cause may be identified based on at least one of them, or may be identified based on all of them.
[0055]
　The determination result output unit 107 is configured to be able to output the determination result of the physical condition abnormality determination unit 106 to the display unit 301 mounted on the vehicle 10 . That is, the determination result output unit 107 outputs information for presenting the information about the passenger's physical condition abnormality to the passenger or fellow passenger. Note that the determination result output unit 107 may be configured to be capable of outputting information regarding the passenger's physical condition abnormality to a speaker (not shown) or the like.
[0056]
　As shown in FIG. 7, the display unit 301 is configured as a display or the like provided in front of the vehicle. The vehicle interior cameras 50 are provided as a camera 50a capable of imaging the faces of the passengers 201 in the driver's seat and the front passenger's seat, and a camera 50b capable of imaging the faces of the passengers 202 in the rear seats.
[0057]
　(Example of display by display unit)
　Next, an example of display by the physical condition abnormality determination system 1 according to the second embodiment (that is, a specific example of display by the display unit 301) will be described with reference to FIGS. 8 to 12. FIG. FIG. 8 is a diagram illustrating an example of a display area of ​​a display unit; FIG. 9 is a diagram (part 1) showing a display example in the display area when car sickness occurs. FIG. 10 is a diagram (part 2) showing a display example in the display area when car sickness occurs. FIG. 11 is a diagram showing an example of an icon indicating the degree of car sickness. FIG. 12 is a diagram showing another display example in the display area of ​​the display unit.
[0058]
　The display area of ​​the display unit 301 may include a cause/countermeasure information display area. Also, the display area of ​​the display unit 301 may include the passenger information display area. An example is shown in FIG. In the cause/countermeasure information display area on the left side, at least one of the cause and the countermeasure is displayed according to the motion sickness determination result. If there is no passenger suffering from motion sickness, nothing may be displayed in the cause/measure information display area. In the passenger information display area on the right side, information about the passengers of the vehicle 10 and information about instructions to the passengers are displayed. In this example, there are three passengers in total, one in the driver's seat, one in the front passenger seat, and one in the left rear seat. ) is displayed.
[0059]
　In the example shown in FIG. 9, the passenger in the left rear seat has motion sickness. Therefore, the icon corresponding to the passenger in the left rear seat in the passenger information display area is changed to an icon indicating motion sickness. Also, the arrow icon (icon in front of the vehicle) in the passenger information display area is illuminated. This icon is an icon indicating that the vehicle is recommended to stop on the shoulder of the road. In such a case, the driving route for parking the vehicle 10 on the road shoulder or parking space may be displayed in conjunction with the navigation device.
[0060]
　In the cause/countermeasure information display area, the cause of the motion sickness of the passenger in the left rear seat (here, braking and reading) is displayed. As described above, the cause of motion sickness may be identified by the abnormality determination unit 106 or the abnormality cause identification unit (not shown). If there are a plurality of passengers with motion sickness, the cause/countermeasure information display area may be divided according to the number of passengers, and displays corresponding to each of them may be displayed. In addition, although not shown here, as countermeasures against physical ailments, based on the identified causes of physical ailments, the following instructions were given: ``Drivers should be careful not to apply the brakes''; You may display more specific instruction content such as "Please stop
[0061]
　It should be noted that information of a template prepared in advance may be used for displaying the cause and displaying the countermeasure. The information in the template should display at least one cause, such as "sudden braking" and "reading," and instructions such as "drivers, please be careful when driving" and "passengers should stop reading, eating, and drinking." It should be information. Further, the template information may be an icon image indicating a cause or a countermeasure. Furthermore, when detecting the physical condition of the passenger, the template information may be used without specifying the cause of the physical condition.
[0062]
　Alternatively, in addition to or instead of the display on the display unit 301, voice guidance may be provided.
[0063]
　The example shown in FIG. 10 is a display example after the vehicle 10 has stopped on the road shoulder after the situation shown in FIG. 9 has occurred. In the passenger information display area, since the vehicle 10 has stopped on the shoulder of the road, the arrow icon that was illuminated in FIG. 9 is extinguished. Meanwhile, the arrow icon near the passenger in the left rear seat is illuminated. This arrow icon is an icon that recommends that the passenger in the left rear seat get out of the vehicle 10 .
[0064]
　In the cause/measure information display area, an instruction content of "Please go outside and take a deep breath" is displayed. That is, countermeasures for reducing motion sickness are displayed. Note that the display example here is merely an example, and other countermeasures may be displayed. For example, display may be made to encourage the user to look out the window, to adjust the seat so that the user can easily see the outside, to move to the front passenger seat, or to take motion sickness medicine. Alternatively, in conjunction with a navigation device, a travel route suitable for implementing countermeasures may be displayed. Specifically, it displays a choice of whether motion sickness prevention is necessary or not, accepts the passenger's answer (a question may be asked by voice, and the passenger's answer may be accepted by voice recognition), and the passenger answers that it is necessary. In this case, the driving route to the nearby pharmacy may be displayed.
[0065]
　One of the causes of motion sickness is the inability of passengers other than the driver to predict the behavior of the vehicle 10 . As a countermeasure against such a cause, when there is a passenger suffering from motion sickness, display or voice guidance regarding the behavior of the vehicle 10 may be performed. For example, based on information about the road on which the vehicle 10 is traveling, voice guidance may be provided as to which direction the vehicle 10 will turn after how many seconds. Alternatively, the direction in which the vehicle 10 turns may be displayed on the display unit 301, or the movement in seconds may be displayed.
[0066]
　As shown in FIG. 11, the passenger icon displayed in the passenger information display area changes according to the degree of motion sickness. For example, in Example 1, the facial expression changes according to the degree of motion sickness. In Example 2, the shading and color of the icon change according to the degree of motion sickness. When changing the color of the icon, it is preferable to adopt a color that allows the urgency to be intuitively grasped. Specifically, if the degree of motion sickness is mild, it may be changed to "green", if it is moderate, it will be changed to "yellow", and if it is severe, it may be changed to "red". Although an example in which icons are changed in three stages is shown here, the types of icons may be increased or decreased according to the degree, such as in five stages, for example.
[0067]
　As shown in FIG. 12, icons may be displayed in the cause/measure information display area. Icons in the figure indicate causes of motion sickness, and icons in the upper row indicate "reading", "sudden braking", and "not looking outside" in order from the left. The icons in the lower row indicate "eating and drinking", "sudden start", and "vibration" in order from the left. In the example of FIG. 12, "reading" and "sudden braking" are presumed to be the causes of motion sickness, so these icons are illuminated.
[0068]
　(Technical Effects)
　Next, technical effects obtained by the system 1 for judging abnormal physical condition according to the second embodiment will be described.
[0069]
　As described with reference to FIGS. 6 to 12, according to the physical condition abnormality determination system 1 according to the second embodiment, by displaying the motion sickness determination result, the passenger of the vehicle 10 is accurately notified of the current situation. be able to. In addition, by displaying the causes of motion sickness and countermeasures, it becomes easier to eliminate motion sickness. It should be noted that, like the first embodiment, the second embodiment can also be applied to motion sickness that occurs in a moving object other than the vehicle 10, and physical condition other than motion sickness.
[0070]
　
　Next, a physical condition abnormality determination system 1 according to a third embodiment will be described with reference to FIG. It should be noted that the third embodiment differs from the already-described first and second embodiments only in part in the configuration and operation, and the other parts are generally the same. Therefore, in the following, portions different from those in the first and second embodiments will be described in detail, and descriptions of other overlapping portions will be omitted as appropriate.
[0071]
　(System Configuration)
　First, the overall configuration of the physical condition abnormality determination system 1 according to the third embodiment will be described with reference to FIG. FIG. 13 is a block diagram showing the overall configuration of the physical condition abnormality determination system according to the third embodiment. In FIG. 13, the same symbols are assigned to the same components as those shown in FIG.
[0072]
　As shown in FIG. 13, the physical condition abnormality determination system 1 according to the third embodiment includes a control information output unit 108 in addition to the components of the physical condition abnormality determination system 1 according to the first embodiment (see FIG. 1). ing.
[0073]
　The control information output unit 108 generates and outputs control information for controlling the vehicle 10 based on the determination result of the physical condition abnormality. The control information output from the control information output section 108 is configured to be input to the vehicle control section 302 of the vehicle 10 . The vehicle control section 302 is a control unit capable of controlling each section of the vehicle 10, and executes control according to control information.
[0074]
　(Details of Control)
　Next, the details of the control executed by the control information output from the control information output section 108 (in other words, the details of the control executed by the vehicle control section 302) will be specifically described.
[0075]
　The control information output unit 108 outputs control information related to automatic driving of the vehicle 10, for example. Specifically, the control information output unit 108 may output control information that removes low-frequency motion in a motion control unit that controls motion of the vehicle 10 . In this way, minute vibrations that cause motion sickness can be suppressed. The control information output unit 108 may output control information for avoiding sudden braking and sudden starting. The control information output unit 108 may output control information that causes the vehicle 10 to slowly stop on the road shoulder. The control information output unit 108 sets the automatic driving level to any one of 0 to 2 (that is, driving level at which the driver is involved in driving), and control information may be output to switch to a state in which the driver can stop on the shoulder of the road.
[0076]
　Also, the control information output unit 108 may output control information for controlling the air conditioning in the passenger compartment. That is, information for changing the state of the air in the passenger compartment to a state in which motion sickness is less likely to occur may be output. Alternatively, the control information output unit 108 may set the positions of the movable navigation display unit, the movable smartphone holder, etc. in the passenger compartment so that when the passenger looks at the display screen, the scenery outside is within the field of view. Control information for adjusting the height may be output. That is, information may be output to change the passenger's field of view to a state in which motion sickness is less likely to occur.
[0077]
　(Technical Effects)
　Next, technical effects obtained by the system 1 for judging abnormal physical condition according to the third embodiment will be described.
[0078]
　As described with reference to FIG. 13 , according to the physical condition abnormality determination system 1 according to the third embodiment, by executing control according to the determination result of motion sickness, a situation in which motion sickness is likely to be eliminated or motion sickness is easily eliminated. A situation that is unlikely to occur can be realized. It should be noted that the third embodiment can also be applied to motion sickness that occurs in a moving body other than the vehicle 10, and physical condition other than motion sickness, as in the first and second embodiments.
[0079]
　
　Next, a physical condition abnormality determination system 1 according to a fourth embodiment will be described with reference to FIG. It should be noted that the fourth embodiment differs from the already described first to third embodiments only in a part of configuration and operation, and other parts are generally the same. Therefore, in the following, portions different from those of the first to third embodiments will be described in detail, and descriptions of other overlapping portions will be omitted as appropriate.
[0080]
　(System Configuration)
　First, the overall configuration of the physical condition abnormality determination system 1 according to the fourth embodiment will be described with reference to FIG. FIG. 14 is a block diagram showing the overall configuration of the physical condition abnormality determination system according to the fourth embodiment. In FIG. 14, the same symbols are attached to the same components as those shown in FIG.
[0081]
　As shown in FIG. 14 , the physical condition determination system 1 according to the fourth embodiment includes a feature extraction unit 103 , a feature accumulation unit 104 , a feature relationship calculation unit 105 and a physical condition determination unit 106 . That is, the physical condition abnormality determination system 1 according to the fourth embodiment includes the image acquisition unit 101 and the subject detection unit 102 among the components of the physical condition abnormality determination system 1 according to the first embodiment (see FIG. 1). do not have.
[0082]
　The feature extraction unit 103 according to the fourth embodiment acquires an image captured by the vehicle interior camera 50, and is configured to be able to extract a feature amount for determining an abnormality in the physical condition of the subject from the image. . Components other than the feature amount extraction unit 103 (that is, the feature storage unit 104, the feature relationship calculation unit 105, and the physical condition abnormality determination unit 106) are configured in the same manner as those in the first to third embodiments. there is
[0083]
　Especially in the fourth embodiment, detection of a target person (for example, detection of a target person's face area, etc.) by the target person detection unit 102 is not executed as in the first to third embodiments already described. Therefore, the image acquired by the feature extraction unit 103 preferably includes an area suitable for extracting feature amounts (for example, an image obtained by enlarging the periphery of the subject's face). In order to capture such an image, for example, the imaging range of the vehicle interior camera 50 may be adjusted in advance to an appropriate range, or the vehicle interior camera 50 may have a function to automatically adjust the imaging range. You can have Alternatively, the feature extraction unit 103 may have a function of automatically detecting an area suitable for extracting a feature amount from the captured image and extracting the feature amount from that area.
[0084]
　(Technical Effect)
　Next, the technical effect obtained by the physical condition abnormality determination system 1 according to the fourth embodiment will be described.
[0085]
　As described in FIG. 14, according to the physical condition abnormality determination system 1 according to the fourth embodiment, similar to the first embodiment, based on the relationship of the feature amount extracted from the image of the passenger, of motion sickness can be determined. In addition, especially in the fourth embodiment, since the system configuration is simplified, it is possible to suppress the cost and increase in size of the device. It should be noted that, like the first to third embodiments, the fourth embodiment can also be applied to motion sickness that occurs in a moving object other than the vehicle 10, and physical condition other than motion sickness.
[0086]
　
　The above-described embodiments can be further described as the following additional remarks, but are not limited to the following.
[0087]
　(Supplementary note 1)
　The physical condition abnormality determination system according to Supplementary note 1 includes extraction means for extracting a plurality of feature amounts indicating the condition of the subject from the image of the subject, and storing the plurality of feature amounts as time series data. accumulating means; calculating means for calculating a relationship between the feature quantities from the plurality of feature quantities accumulated in the accumulating means; and determination means for judging abnormal physical condition of the subject based on the relationship. It is a physical condition abnormality determination system characterized by comprising
[0088]
　(Supplementary note 2)
　The physical condition abnormality determination system according to Supplementary note 2 is characterized in that the calculation means calculates the relationship from the degree of change per unit time of the plurality of feature values. is a physical condition abnormality determination system.
[0089]
　(Supplementary Note 3)
　The physical condition abnormality determination system according to Supplementary Note 3 is characterized in that the calculation means calculates a correlation or ratio between different feature quantities as the relationship, and the physical condition abnormality according to Supplementary Note 1 or 2. judgment system.
[0090]
　(Supplementary Note 4)
　The physical condition abnormality determination system according to Supplementary Note 4 is characterized in that the plurality of feature amounts include feature amounts related to the subject's pupil position, face orientation, or line of sight direction. 4. The physical condition abnormality determination system according to any one of 3.
[0091]
　(Supplementary Note 5)
　The physical condition abnormality determination system according to Supplementary Note 5 is characterized in that the relationships are the correlation between the face position and the eye position, the correlation between the face pan direction and the line of sight horizontal direction, the face tilt direction and the line of sight vertical 5. The physical condition determination system according to any one of appendices 1 to 4, characterized by including at least one correlation with direction.
[0092]
　(Supplementary Note 6)
　The physical condition abnormality determination system according to Supplementary Note 6 is characterized in that the relationship is the ratio of the moving speed between the center of gravity of the face and the eye position, the ratio of the angular velocity between the panning direction of the face and the horizontal direction of the line of sight, and the direction of the face. 6. The physical condition determination system according to any one of appendices 1 to 5, characterized in that at least one of the ratios of angular velocities between the tilt direction and the line-of-sight vertical direction is included.
[0093]
　(Supplementary Note 7)
　The physical condition abnormality determination system according to Supplementary Note 7 is any one of Supplementary Notes 1 to 6, wherein the determination means determines and outputs the degree of the physical condition abnormality based on the relationship. 1. The physical condition abnormality determination system according to item 1.
[0094]
　(Supplementary Note 8)
　The physical condition abnormality determination system according to Supplementary Note 8 is configured to transmit the information regarding the physical condition abnormality of the subject according to the determination result of the determination means to at least the subject and persons existing around the subject. 8. The physical condition abnormality determination system according to any one of Appendices 1 to 7, further comprising presentation means for presenting to one side.
[0095]
　(Supplementary Note 9)
　In the physical condition abnormality determination system according to Supplementary Note 9, the target person is a passenger of a vehicle, and the vehicle is controlled according to the physical condition abnormality based on the determination result of the determination means. 9. The physical condition abnormality determination system according to any one of appendices 1 to 8, further comprising output means for outputting control information for.
[0096]
　(Supplementary note 10)
　The physical condition determination system according to supplementary note 10 is the physical condition determination system according to any one of supplementary notes 1 to 9, wherein the physical condition is a state of motion sickness.
[0097]
　(Supplementary note 11)
　The physical condition abnormality determination method according to Supplementary note 11 extracts a plurality of feature amounts indicating the state of the subject from the image of the subject, accumulates the plurality of feature amounts as time series data, and accumulates and calculating a relationship between the feature amounts from the plurality of feature amounts, and determining the physical condition of the subject based on the relationship.
[0098]
　(Supplementary note 12)
　The computer program according to Supplementary note 12 extracts a plurality of feature amounts indicating the state of the subject from the image of the subject, accumulates the plurality of feature amounts as time-series data, and accumulates the accumulated A computer program characterized by operating a computer so as to calculate a relationship between each feature amount from a plurality of feature amounts and determine the physical condition of the subject based on the relationship.
[0099]
　The present invention can be modified as appropriate within the scope not contrary to the gist or idea of ​​the invention that can be read from the scope of claims and the entire specification. A program is also included in the technical concept of the present invention.
Code explanation
[0100]
　1 physical condition abnormality determination system
　10 vehicle
　11 CPU
　12 RAM
　13 ROM
　14 storage device
　15 input device
　16 output device
　17 data bus
　20 external server
　50 vehicle interior camera
　101 image acquisition unit
　102 subject detection unit
　103 feature extraction unit
　104 feature storage unit
　105 Characteristic relationship calculation unit
　106 Abnormal physical condition determination unit
　107 Determination result output unit
　108 Control information output units
　201 and 202 Passenger
　301 Display unit
　302 Vehicle control unit
The scope of the claims
[Claim 1]
　extracting means for extracting a plurality of feature quantities indicating the state of the subject from an image of the subject;
　storage means for accumulating the plurality of feature quantities as time-series data; and the plurality of features accumulated in the storage
　means A physical condition determination system , comprising: calculation means for calculating relationships between respective feature quantities from quantities; and
　determination means for determining abnormal physical conditions of said subject based on said relationships
　.
[Claim 2]
　2. A physical condition abnormality determination system according to claim 1, wherein said calculation means calculates said relationship from the degree of change per unit time of said plurality of feature values.
[Claim 3]

　3. A physical condition abnormality determination system according to claim 1 　, wherein said calculation means calculates a correlation or a ratio between different feature quantities as said relationship .
[Claim 4]
　4. The physical condition abnormality determination system according to any one of claims 1 to 3, wherein the plurality of feature amounts include feature amounts relating to the subject's eye position, face orientation, or line-of-sight direction.
[Claim 5]
　The relationship includes at least one of a correlation between a face position and a pupil position, a correlation between a face pan direction and a horizontal line-of-sight direction, and a correlation between a face tilt direction and a vertical line-of-sight direction. The physical condition abnormality determination system according to any one of claims 1 to 4.
[Claim 6]
　The relationship is at least one of a ratio of moving speed between the center of gravity of the face and the position of the eyes, a ratio of angular velocities between the face pan direction and the line-of-sight horizontal direction, and an angular speed ratio between the face tilt direction and the line-of-sight vertical direction. 6. The physical condition abnormality determination system according to any one of claims 1 to 5, comprising:
[Claim 7]
　7. The physical condition abnormality determination system according to any one of claims 1 to 6, wherein the determination means determines and outputs the degree of the physical condition abnormality based on the relationship.
[Claim 8]
　The present invention further comprises presenting means for presenting the information about the physical condition abnormality of the subject according to the determination result of the determining means to at least one of the subject and persons present around the subject. Item 8. The physical condition abnormality determination system according to any one of Items 1 to 7.
[Claim 9]
　The target person is a passenger of a vehicle,
　and further includes output means for outputting control information for executing control of the vehicle according to the abnormality in physical condition based on the determination result of the determination means
　. The physical condition abnormality determination system according to any one of claims 1 to 8.
[Claim 10]
　10. The system for determining abnormality in physical condition according to any one of claims 1 to 9, wherein the abnormality in physical condition is a state of motion sickness.
[Claim 11]
　Extracting a plurality of feature amounts indicating the state of the subject from an image of the subject,
　accumulating the plurality of feature amounts as time-series data, and
　determining relationships between the feature amounts from the plurality of accumulated feature amounts. is calculated,
　and based on the relationship, an abnormality in physical condition of the subject is
　determined.
[Claim 12]
　Extracting a plurality of feature amounts indicating the state of the subject from an image of the subject,
　accumulating the plurality of feature amounts as time-series data, and
　determining relationships between the feature amounts from the plurality of accumulated feature amounts. and 　operating a computer to
　determine the subject's physical condition abnormality based on the relationship .