Description:TECHNICAL FIELD
The present disclosure relates to healthcare. More particularly, the present disclosure relates to a system, an apparatus, and a method for Heart Rate determination of a user wearing a face mask.
BACKGROUND
Heart Rate (HR) is a crucial human vital parameter that can provide essential information about an individual’s physical and physiological condition, thereby aiding doctors in diagnosing medical conditions. Traditional HR determination techniques demand proper contact between skin and sensors which limits user acceptability during the COVID-19 pandemic. As a result, remote photoplethysmography (rPPG) utilization is proliferated in HR determination which is performed by analyzing face videos that do not require any physical contact between humans and sensors. Unfortunately, the existing rPPG techniques are incompetent when the facial region is covered with a face mask.
The existing works for remote HR determination are based on either of, Lagrangian and Eulerian principle where the Lagrangian approach focuses on the motion variation of facial landmark points, and the Eulerian approach relies on the color variation of facial landmark points.
However, the existing works primarily neglect the masked region for extracting relevant rPPG information, thereby providing limited performance. The light reflected by the face mask lacks relevant rPPG color variations which limits the efficacy of the Eulerian approach for masked patients. Moreover, the remote HR determination by Eulerian approach can be further influenced by inadequate illumination, unsuitable camera focus, or human factors such as hair or headwear obscuring the observational area, resulting in erroneous HR determination. On the other hand, the remote HR determination based on Lagrangian approach is time-consuming and may produce unreliable results when only a few discriminatory features are available for tracking, particularly in poor lighting conditions and thus needs further improvement. Here it is observed that either of these approaches for remote HR determination has limited performance.
Therefore, there is a need for a system and a method that provides a technical solution towards the challenges mentioned hereinabove for enhanced accuracy of remote Heart Rate (HR) determination.
SUMMARY:
In an aspect of the present disclosure, a data processing apparatus includes processing circuitry configured to combine first set of noise cancelled signals to generate a consolidated Eulerian signal and second set of noise cancelled signals to generate a consolidated Lagrangian signal, respectively. The processing circuitry is further configured to generate first and second pulse spectrum from the consolidated Eulerian signal and the consolidated Lagrangian signal, respectively, using Fast Fourier Transformation (FFT) technique. Furthermore, the processing circuitry is configured to select one of, the first and second pulse spectrum as best consolidated signal based on a set of quality parameters of the consolidated Eulerian signal and the consolidated Lagrangian signal. Furthermore, the processing circuitry is configured to determine Heart Rate (HR) based on the best consolidated signal. Furthermore, the processing circuitry is configured to generate a validation signal based on a comparison of the determined HR with a predetermined heart rate.
In some aspects, the validation signal may be configured to generate a first acknowledgement, when the determined HR is matched with the predefined heart rate and generate a first alert, when the determined HR is mismatched with the predefined heart rate.
In some aspects, prior to the generation of the first and second sets of pulse spectrums, the processing circuitry is configured to generate first and second sets of temporal signals based on sensing of color and motion variation of first and second sets of non-overlapping windows using Eulerian and Lagrangian approach respectively and filter the first and the second sets of temporal signals to generate first and second sets of noise cancelled signals.
In some aspects, to generate the first and the second sets of noise cancelled signals, the processing circuitry is configured to apply a band pass filter within a range of 0.7 Hz to 4.0 Hz on the first and the second sets of temporal signals to generate first and second sets of filtered signals and generate the first and the second sets of noise cancelled signals from the first and the second sets of filtered signals using blind source separation (BSS) technique.
In another aspect of the present disclosure, a system includes a sensing unit and a data processing apparatus. The sensing unit is configured to capture a plurality of images. The data processing apparatus is coupled to the sensing unit and includes processing circuitry. The processing circuitry is configured to combine first set of noise cancelled signals to generate a consolidated Eulerian signal and second set of noise cancelled signals to generate a consolidated Lagrangian signal, respectively. The processing circuitry is further configured to generate first and second pulse spectrum from the consolidated Eulerian signal and the consolidated Lagrangian signal, respectively, using Fast Fourier Transformation (FFT) technique. Furthermore, the processing circuitry is configured to select one of, the first and second pulse spectrum as best consolidated signal based on a set of quality parameters of the consolidated Eulerian signal and the consolidated Lagrangian signal. Furthermore, the processing circuitry is configured to determine Heart Rate (HR) based on the best consolidated signal. Furthermore, the processing circuitry is configured to generate a validation signal based on a comparison of the determined HR with a predetermined heart rate.
In yet another aspect of the present disclosure, a method includes combining, by way of processing circuitry, first set of noise cancelled signals to generate a consolidated Eulerian signal and second set of noise cancelled signals to generate a consolidated Lagrangian signal, respectively. The method further includes generating, by way of the processing circuitry, first and second pulse spectrum from the consolidated Eulerian signal and the consolidated Lagrangian signal, respectively, using Fast Fourier Transformation (FFT) technique. Furthermore, the method includes selecting, by way of the processing circuitry, one of, first and second pulse spectrum as best consolidated signal based on a set of quality parameters of the consolidated Eulerian signal and the consolidated Lagrangian signal. Furthermore, the method includes determining, by way of the processing circuitry, Heart Rate (HR) based on the best consolidated signal. Furthermore, the method includes generating, by way of the processing circuitry, a validation signal based on a comparison of the determined HR with a predetermined heart rate.
BRIEF DESCRIPTION OF DRAWINGS:
The above and still further features and advantages of aspects of the present disclosure becomes apparent upon consideration of the following detailed description of aspects thereof, especially when taken in conjunction with the accompanying drawings, and wherein:
FIG. 1 illustrates a block diagram of a system for heart rate determination, in accordance with an exemplary aspect of the present disclosure;
FIG. 2 illustrates a block diagram of a data processing apparatus of FIG. 1, in accordance with an exemplary aspect of the present disclosure; and
FIG. 3 illustrates a flow chart of a method for the heart rate determination, in accordance with an exemplary aspect of the present disclosure.
To facilitate understanding, reference numerals have been used, where possible, to designate elements common to the figures.
DETAILED DESCRIPTION
Various aspects of the present disclosure provide a system, an apparatus, and a method for Heart Rate (HR) determination. The following description provides specific details of certain aspects of the disclosure illustrated in the drawings to provide a thorough understanding of those aspects. It should be recognized, however, that the present disclosure can be reflected in additional aspects and the disclosure may be practiced without some of the details in the following description.
The various aspects including the example aspects are now described more fully with reference to the accompanying drawings, in which the various aspects of the disclosure are shown. The disclosure may, however, be embodied in different forms and should not be construed as limited to the aspects set forth herein. Rather, these aspects are provided so that this disclosure is thorough and complete, and fully conveys the scope of the disclosure to those skilled in the art. In the drawings, the sizes of components may be exaggerated for clarity.
It is understood that when an element is referred to as being “on,” “connected to,” or “coupled to” another element, it can be directly on, connected to, or coupled to the other element or intervening elements that may be present. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items.
The subject matter of example aspects, as disclosed herein, is described with specificity to meet statutory requirements. However, the description itself is not intended to limit the scope of this patent. Rather, the inventor/inventors have contemplated that the claimed subject matter might also be embodied in other ways, to include different features or combinations of features similar to the ones described in this document, in conjunction with other technologies. Generally, the various aspects including the example aspects relate to the system, and the method for HR determination.
As mentioned, there is a need for a system, an apparatus, and a method for accurate and precise determination of remote HR determination that efficiently utilize the available resources and results in least amount of irrelevant data captured by the system to cover the desired area. The present aspects, therefore: provide a system 100, a data processing apparatus 106, and a method 300 that provides an improvised technical solution to overcome the aforementioned problems.
The aspects herein and the various features and advantageous details thereof are explained more fully with reference to the non-limiting aspects that are illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known components and processing techniques are omitted so as to not unnecessarily obscure the aspects herein. The examples used herein are intended merely to facilitate an understanding of ways in which the aspects herein may be practiced and to further enable those of skill in the art to practice the aspects herein. Accordingly, the examples should not be construed as limiting the scope of the aspects herein.
FIG. 1 illustrates a block diagram of the system 100 for HR determination of a user wearing a face mask, in accordance with an exemplary aspect of the present disclosure. The system 100 may include a sensing unit 102, a user device 104, a data processing apparatus 106. In some aspects of the present disclosure, the sensing unit 102 and the user device 104 may be communicatively coupled to the data processing apparatus 106, by way of either of, a first wired communication medium and a first wireless communication medium. In some aspects of the present disclosure, the sensing unit 102, the user device 104, and the data processing apparatus 106 may be communicatively coupled to each other, by way of a communication network 108.
The sensing unit 102 may be configured to capture a plurality of images. In some aspects of the present disclosure, the sensing unit 102 may include a power supply (not shown), one or more camera sensors (not shown), a first processing unit (not shown), a first memory (not shown), and a first communication interface (not shown). In some aspects of the present disclosure, various components of the sensing unit 102 may be coupled to each other, by way of one or more wired or wireless communication mediums (not shown).
Examples of the one or more camera sensors (not shown) of the sensing unit 102 may include but not limited to, a stationary camera, a Pan-Tilt-Zoom (PTZ) camera, a depth sensing camera pair, and the like. Aspects of the present disclosure are intended to include or otherwise cover any type of camera sensor including known, related art, and/or later developed camera sensors.
The first processing unit (not shown) within the sensing unit 102 may include suitable logic, instructions, circuitry, interfaces, and/or codes for executing various operations or it may include one or more processors such as Arduino or raspberry pi or the like. In some aspects of the present disclosure, the first processing unit (not shown) of the sensing unit 102 may be configured to select ROIs from the plurality of images captured by the one or more camera sensors. Examples of the associated first processing unit (not shown) of sensing unit 102 may include, but are not limited to, an application-specific integrated circuit (ASIC) processor, a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a field-programmable gate array (FPGA), a Programmable Logic Control unit (PLC), and the like. Aspects of the present disclosure are intended to include or otherwise cover any type of processing unit including known, related art, and/or later developed processing units.
The first memory (not shown) associated with the sensing unit 102 may be configured to store the logic, instructions, circuitry, interfaces, and/or codes of the associated first processing unit (not shown) of the sensing unit 102, and/or data associated with the system 100. In some aspects of the present disclosure, the sensing unit 102 associated first memory (not shown) may be configured to store a variety of inputs received from the data processing apparatus 106 or to temporarily store the plurality of images. Examples of the sensing unit 102 associated first memory (not shown) may include, but are not limited to, a Read-Only Memory (ROM), a Random-Access Memory (RAM), a flash memory, a removable storage drive, a hard disk drive (HDD), a solid-state memory, a magnetic storage drive, a Programmable Read Only Memory (PROM), an Erasable PROM (EPROM), and/or an Electrically EPROM (EEPROM). Aspects of the present disclosure are intended to include or otherwise cover any type of sensing unit 102 associated first memory (not shown) including known, related art, and/or later developed memories.
The first communication interface (not shown) may be configured to enable the sensing unit 102 to communicate with the data processing apparatus 106 and the user device 104. Examples of the first communication interface (not shown) associated with the sensing unit 102 may include, but are not limited to, a modem, a network interface such as an Ethernet card, a communication port, and/or a Personal Computer Memory Card International Association (PCMCIA) slot and card, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, and a local buffer circuit.
The user device 104 may be configured to enable a user to submit a set of inputs. The user device 104 may further be configured to enable the user to select and/or input one or more parameters associated with the sensing unit 102. In some aspects of the present disclosure, the user device 104 may be configured to facilitate the user to provide input(s) to register on the system 100. Furthermore, the user device 104 may facilitate the user to enable a password protection for logging-in (i.e., user authentication) to the system 100. In some aspects of the present disclosure, the user device 104 may include a user interface 110, a second processing unit 112, a second memory 114, a console 116, and a second communication interface 118.
The user interface 110 may include an input interface (not shown) for receiving inputs from the user. In some aspects of the present disclosure, the input interface (not shown) may be configured to enable the user to submit the set of inputs for selection of ROIs. The input interface (not shown) may further be configured to enable the user to select and/or input the one or more parameters associated with the sensing unit 102. Furthermore, the input interface (not shown) may be configured to enable the user to select and/or provide inputs for registration and/or authentication of the user to use one or more functionalities of the system 100. In some aspects of the present disclosure, the input interface (not shown) may be configured to enable the user to provide inputs to enable password protection for logging-in to the system 100. Examples of the input interface (not shown) may include, but are not limited to, a touch interface, a mouse, a keyboard, a motion recognition unit, a gesture recognition unit, a voice recognition unit, or the like. Aspects of the present disclosure are intended to include and/or otherwise cover any type of the input interface (not shown) including known and/or related, or later developed technologies.
The user interface 110 may further include an output interface (not shown) for displaying (or presenting) an output to the user. Examples of the output interface (not shown) may include, but are not limited to, a digital display, an analog display, a touch screen display, a graphical user interface, a website, a webpage, a keyboard, a mouse, a light pen, an appearance of a desktop, and/or illuminated characters. Aspects of the present disclosure are intended to include and/or otherwise cover any type of the output interface (not shown) including known and/or related, or later developed technologies.
The second processing unit 112 may include suitable logic, instructions, circuitry, interfaces, and/or codes for executing various operations, such as the operations associated with the user device 104, and/or the like. In some aspects of the present disclosure, the second processing unit 112 may utilize one or more processors such as Arduino or raspberry pi or the like. Further, the second processing unit 112 may be configured to control one or more operations executed by the user device 104 in response to the input received at the user interface 110 from the user. Examples of the second processing unit 112 may include, but are not limited to, an application-specific integrated circuit (ASIC) processor, a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a field-programmable gate array (FPGA), a Programmable Logic Control unit (PLC), and the like. Aspects of the present disclosure are intended to include or otherwise cover any type of second processing unit 112 including known, related art, and/or later developed processing units.
The second memory 114 may be configured to store the logic, instructions, circuitry, interfaces, and/or codes of the second processing unit 112, data associated with the user device 104, and/or data associated with the system 100. In some aspects of the present disclosure, the second memory 114 may be configured to store a variety of inputs received from the user. Examples of the second memory 114 may include, but are not limited to, a Read-Only Memory (ROM), a Random-Access Memory (RAM), a flash memory, a removable storage drive, a hard disk drive (HDD), a solid-state memory, a magnetic storage drive, a Programmable Read Only Memory (PROM), an Erasable PROM (EPROM), and/or an Electrically EPROM (EEPROM). Aspects of the present disclosure are intended to include or otherwise cover any type of second memory 114 including known, related art, and/or later developed memories.
The console 116 may be configured as a computer-executable application, to be executed by the second processing unit 112. The console 116 may include suitable logic, instructions, and/or codes for executing various operations and may be controlled by the data processing apparatus 106. The one or more computer executable applications may be stored in the second memory 114. Examples of the one or more computer executable applications may include, but are not limited to, an audio application, a video application, a social media application, a navigation application, or the like. Aspects of the present disclosure are intended to include or otherwise cover any type of the computer executable application including known, related art, and/or later developed computer executable applications.
The second communication interface 118 may be configured to enable the user device 104 to communicate with the sensing unit 102, and the data processing apparatus 106. Examples of the second communication interface 118 may include, but are not limited to, a modem, a network interface such as an Ethernet card, a communication port, and/or a Personal Computer Memory Card International Association (PCMCIA) slot and card, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, and a local buffer circuit. It will be apparent to a person of ordinary skill in the art that the second communication interface 118 may include any device and/or apparatus capable of providing wireless or wired communications of the user device 104 with the sensing unit 102 and the data processing apparatus 106.
The data processing apparatus 106 may be a network of computers, a software framework, or a combination thereof, that may provide a generalized approach to create the server implementation. Examples of the data processing apparatus 106 may include, but are not limited to, personal computers, laptops, mini-computers, mainframe computers, any non-transient and tangible machine that can execute a machine-readable code, cloud-based servers, distributed server networks, or a network of computer systems. The data processing apparatus 106 may be realized through various web-based technologies such as, but not limited to, a Java web-framework, a .NET framework, a personal home page (PHP) framework, or any web-application framework. The data processing apparatus 106 may include processing circuitry 120 and one or more Database (hereinafter, collectively referred to and designated as “Database 122”).
The processing circuitry 120 may include suitable logic, instructions, circuitry, interfaces, and/or codes for executing various operations of the system 100. The processing circuitry 120 may be configured to host and enable the console 116 running on (or installed on) the user device 104 to execute the operations associated with the system 100 by communicating one or more commands and/or instructions over the communication network 108.
The Database 122 may be configured to store the logic, instructions, circuitry, interfaces, and/or codes of the processing circuitry 120 for executing several operations. The Database 122 may be further configured to store therein, data associated with users registered with the system 100. Some aspects of the present disclosure are intended to include and/or otherwise cover any type of the data associated with the users registered with the system 100. Examples of the Database 122 may include but are not limited to, a ROM, a RAM, a flash memory, a removable storage drive, a HDD, a solid-state memory, a magnetic storage drive, a PROM, an EPROM, and/or an EEPROM. In some aspects of the present disclosure, the Database 122 may be configured to store one or more user data, instructions data, one or more configuration parameters of the sensing unit 102, and the like corresponding to the system 100.
The communication network 108 may include suitable logic, circuitry, and interfaces that may be configured to provide several network ports and several communication channels for transmission and reception of data related to operations of various entities (such as the sensing unit 102, the user device 104, and the data processing apparatus 106) of the system 100. Each network port may correspond to a virtual address (or a physical machine address) for transmission and reception of the communication data. For example, the virtual address may be an Internet Protocol Version 4 (IPV4) (or an IPV6 address) and the physical address may be a Media Access Control (MAC) address. The communication network 108 may be associated with an application layer for implementation of communication protocols based on one or more communication requests from the sensing unit 102, the user device 104, and the data processing apparatus 106. The communication data may be transmitted or received via the communication protocols. Examples of the communication protocols may include, but are not limited to, Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Simple Mail Transfer Protocol (SMTP), Domain Network System (DNS) protocol, Common Management Interface Protocol (CMIP), Transmission Control Protocol and Internet Protocol (TCP/IP), User Datagram Protocol (UDP), Long Term Evolution (LTE) communication protocols, or any combination thereof. In some aspects of the present disclosure, the communication data may be transmitted or received via at least one communication channel of several communication channels in the communication network 108. The communication channels may include, but are not limited to, a wireless channel, a wired channel, a combination of wireless and wired channel thereof. The wireless or wired channel may be associated with a data standard which may be defined by one of a Local Area Network (LAN), a Personal Area Network (PAN), a Wireless Local Area Network (WLAN), a Wireless Sensor Network (WSN), Wireless Area Network (WAN), Wireless Wide Area Network (WWAN), a metropolitan area network (MAN), a satellite network, the Internet, an optical fiber network, a coaxial cable network, an infrared (IR) network, a radio frequency (RF) network, and a combination thereof. Aspects of the present disclosure are intended to include or otherwise cover any type of communication channel, including known, related art, and/or later developed technologies.
In operation, the data processing apparatus 106, by way of the processing circuitry 120, may be configured to receive the set of inputs from the user device 104. The data processing apparatus 106, by way of the processing circuitry 120, may further be configured to determine non-occluded region and masked region from each image of the plurality of images collected from the sensing unit 102 of a user wearing a face mask. Furthermore, the data processing apparatus 106, by way of the processing circuitry 120, may be configured to determine Regions of Interests (ROIs) of the non-occluded and the masked region from each image of the plurality of images. Furthermore, the data processing apparatus 106, by way of the processing circuitry 120, may be configured to determine first and second sets of non-overlapping windows corresponding to ROIs of the non-occluded and masked region of each image, respectively. Furthermore, the data processing apparatus 106, by way of the processing circuitry 120, may be configured to generate first and second sets of temporal signals from the first and second sets of non-overlapping windows based on sensing of color and motion variation using Eulerian and Lagrangian approach, respectively. Furthermore, the data processing apparatus 106, by way of the processing circuitry 120, may be configured to filter the first and second sets of temporal signals to generate first and second sets of noise cancelled signals. Upon generation of the first and second sets of noise cancelled signals, the data processing apparatus 106, by way of the processing circuitry 120, may be configured to combine the first set of noise cancelled signals to generate a consolidated Eulerian signal and the second set of noise cancelled signals to generate a consolidated Lagrangian signal, respectively. The data processing apparatus 106, by way of the processing circuitry 120, may further be configured to generate first and second pulse spectrum from the consolidated Eulerian signal and the consolidated Lagrangian signal, respectively, using Fast Fourier Transformation (FFT) technique. Furthermore, the data processing apparatus 106, by way of the processing circuitry 120, may be configured to select one of, the first and second pulse spectrum as best consolidated signal based on a set of quality parameters of the consolidated Eulerian signal and the consolidated Lagrangian signal. Upon determination of the best consolidated signal, the data processing apparatus 106, by way of the processing circuitry 120, may be configured to determine Heart Rate (HR) from the best consolidated signal using a mathematical expression presented as: HR = round (mf × 60), where mf denotes the frequency in Hz associated with the highest magnitude of the best consolidated signal and round denotes the rounding off operation. Furthermore, the data processing apparatus 106, by way of the processing circuitry 120, may be configured to generate a validation signal based on a comparison of the determined HR with a predetermined heart rate. Based on the validation signal, the data processing apparatus 106, by way of the processing circuitry 120, may be configured to generate one of, a first acknowledgement when the determined HR is matched with the predefined heart rate, and a first alert when the determined HR is mismatched with the predefined heart rate.
FIG. 2 is a block diagram that illustrates the data processing apparatus 106 of FIG. 1, in accordance with an exemplary aspect of the present disclosure. The data processing apparatus 106 may include the processing circuitry 120 and the Database 122. The data processing apparatus 106 may further include a network interface 200 and an input/output (I/O) interface 202. The processing circuitry 120, the database 122, the network interface 200, and the input/output (I/O) interface 202 may be configured to communicate with each other, by way of a first communication bus 203.
In some aspects of the present disclosure, the processing circuitry 120 may include a data exchange engine 204, a registration engine 206, an authentication engine 208, a data segregation engine 210, a data processing engine 212, a first ROI determination engine 214, a second ROI determination engine 216, a first filtration engine 218, a second filtration engine 220, a consolidated signal engine 222, and a notification engine 224, communicatively coupled to each other, by way of a second communication bus 226. It will be apparent to a person having ordinary skill in the art that the data processing apparatus 106 is for illustrative purposes and not limited to any specific combination of hardware circuitry and/or software.
The data exchange engine 204 may be configured to enable transfer of data from the database to various engines of the processing circuitry. The data exchange engine 204 may further be configured to enable transfer of data and/or instructions from the user device 104 and/or the sensing unit 102 to the data processing apparatus 106.
The registration engine 206 may be configured to enable the user to register into the system 100 by providing registration data through a registration menu (not shown) of the console 116 that may be displayed, by way of the user device 104.
The authentication engine 208, by way of the data exchange engine 204 may be configured to fetch the registration data of the user and authenticate the registration data of the user. The authentication engine 208, upon successful authentication of the registration data of the user, may be configured to enable the user to log-in or sign up to the system 100. In some aspects of the present disclosure, the authentication engine 208 may enable the user to set the password protection for logging-in to the system 100. In such a scenario, the authentication engine 208 may be configured to verify a password entered by the user for logging-in to the system 100 by comparing the password entered by the user with the set password protection. In some aspects, when the password entered by the user is verified by the authentication engine 208, the authentication engine 208 may enable the user to log-in to the system 100. In some other aspects of the present disclosure, when the password entered by the user is not verified by the authentication engine 208, the authentication engine 208 may generate a signal for the notification engine 224 to generate a login failure notification for the user.
Upon authentication of the registration data of the user, the data segregation engine 210 may be configured to segregate each image of the plurality of images collected by the sensing unit 102 into two different portions separately, coming from the non-occluded region and the masked region.
Upon segregation of each image of the plurality of images collected by the sensing unit 102 into two different portions separately, the data processing engine 212 may be configured to generate first and second sets of temporal signals from first and second sets of non-overlapping windows based on sensing of color and motion variation of using Eulerian and Lagrangian approach respectively. The data processing engine 212 may also be configured to receive the set of inputs from the user device 104.
Upon segregation of image using the data segregation engine 210 but before generation of first set of temporal signals from the first sets of non-overlapping windows using the data processing engine 212 engine the first ROI determination engine 214 may be configured to determine ROIs of the non-occluded region from each image. Furthermore, the first ROI determination engine 214 may be configured to determine the first sets of non-overlapping windows corresponding to ROIs of the non-occluded region of each image.
Upon segregation of image using the data segregation engine 210 but before generation of second set of temporal signals from the second sets of non-overlapping windows using the data processing engine 212 the second ROI determination engine 216 may be configured to determine ROIs of the non-occluded region from each image. Furthermore, the second ROI determination engine 216 may be configured to determine the second sets of non-overlapping windows corresponding to ROIs of the non-occluded region of each image.
Upon generation of first set of temporal signals from first set of non-overlapping windows using the data processing engine 212, the first filtration engine 218 may be configured to filter the first sets of temporal signals to generate first set of filtered signals using a band pass filter within the range from 0.7 Hz to 4.0 Hz. The first filtration engine 218 may further be configured to generate the first set of noise cancelled signals from the first set of filtered signals using a blind source separation (BSS) technique. In some aspects of the present disclosure, the first filtration engine 218 may be configured to generate the first set of noise cancelled signals using a multiuser kurtosis (MUK) technique.
Upon generation of the second sets of temporal signals from the second sets of non-overlapping windows using the data processing engine 212 engine, the second filtration engine 220 may be configured to filter the second sets of temporal signals to generate the second set of filtered signals using the band pass filter within the range from 0.7 Hz to 4.0 Hz. The second filtration engine 220 may further be configured to generate the second set of noise cancelled signals from the second set of filtered signals using the blind source separation (BSS) technique. In some aspects of the present disclosure, the second filtration engine 220 may be configured to generate the second set of noise cancelled signals using the blind source separation (BSS) technique.
Upon generation of the first and second sets of noise cancelled signals, the consolidated signal engine 222 may be configured to combine the first set of noise cancelled signals to generate the consolidated Eulerian signal. The consolidated signal engine 222 may be configured to combine the second set of noise cancelled signals to generate the consolidated Lagrangian signal. Furthermore, the consolidated signal engine 222 may be configured to generate the first and second pulse spectrum from the consolidated Eulerian signal and the consolidated Lagrangian signal, respectively, using Fast Fourier Transformation (FFT) technique.
The selection engine 223 may be configured to receive the first and second pulse spectrum and the consolidated Eulerian and consolidated Lagrangian signals from the consolidated signal engine 222. The selection engine 223 may further be configured to select one of, the first and second pulse spectrum as best consolidated signal based on a set of quality parameters of the consolidated Eulerian signal and the consolidated Lagrangian signal. In some aspects of the present disclosure, to select one of, the consolidated Eulerian signal and the consolidated Lagrangian signal (or the associated first and second pulse spectrum), the selection engine 223 may be configured to determine first and second quality scores for the consolidated Eulerian signal and the consolidated Lagrangian signal, respectively, based on the set of quality parameters. The selection engine 223 may further compare the first and second quality scores. Furthermore, when the first quality score is greater than the second quality score, the selection engine 223 may be configured to select the first spectra associated with the consolidated Eulerian signal as the best consolidated signal, else when the second quality score is greater than the first quality score, the selection engine 223 may be configured to select the second spectra associated with the consolidated Lagrangian signal as the best consolidated signal.
The selection engine 223 may further be configured to determine the Heart Rate (HR) value based on the best consolidated signal. Furthermore, the selection engine 223 may be configured to generate the validation signal based on the comparison of the determined HR value with the pre-determined set of heart rate values. In some aspects of the present disclosure, the selection engine 223 may be configured to enables one of, generation of a first acknowledgement, when the determined HR is matched with the predefined heart rate and generation of a first alert, when the determined HR is mismatched with the predefined heart rate.
Upon generation of either of, the first acknowledgement or the first alert, the notification engine 224 may be configured to generate one or more notifications corresponding to the system 100 that may be presented to the user, by way of the user device 104. It will be apparent to a person skilled in the art that the aspects of the present disclosure are intended to include or cover any type of notification generated by the system 100 and/or presented to the user by the system 100.
The database 122 may be configured to store data and/or metadata associated with the data processing apparatus 106. In some aspects of the present disclosure, the database 122 may be segregated into one or more repositories that may be configured to store a specific type of data. In an exemplary aspect of the present disclosure, the database 122 may include an instructions repository 228, a user data repository 230, and an image data repository 232, a ROI repository 234, a signal data repository 236, a filtered signal repository 238, a consolidated signal repository 240 and a HR repository 242.
The instructions repository 228 may be configured to store instructions data corresponding to the data processing apparatus 106. The instructions data may include data and metadata of one or more instructions corresponding to the various entities of the data processing apparatus 106 such as the processing circuitry 120, the I/O interface 202 and/or the network interface 200. It will be apparent to a person skilled in the art that the aspects of the present disclosure are intended to include or cover any type of instructions data of the data processing apparatus 106, and thus must not be considered as a limitation of the present disclosure.
The user data repository 230 may be configured to store data associated with a user of the system 100. Specifically, the user data may include data and/or metadata of the data of authenticated users that may be registered on the system 100. In some aspects of the present disclosure, the user data repository 230 may further be configured to temporarily store data and/or metadata of one or more users that fail to register and/or authenticate on the system 100. Furthermore, the user data repository 230 may be configured to store the set of inputs received from the user, by way of the user device 104. It will be apparent to a person skilled in the art that the aspects of the present disclosure are intended to include or cover any type of user data and/or metadata of the user data of the system 100, and thus must not be considered as a limitation of the present disclosure.
The image data repository 232 may be configured to store the plurality of images captured by the sensing unit 102. The image data repository 232 may further be configured to store a combined (or fused) image that may be generated by combining the plurality of images by the processing circuitry 120. The ROI repository 234 may be configured to store the first and second sets of ROIs and the corresponding first and second sets of non-overlapping windows, respectively that may be determined from the non-occluded and masked regions, respectively, of each image of the plurality of images. The signal data repository 236 may be configured to store the first and second sets of temporal signals that may be extracted from the non-occluded and the masked regions based on sensing of color and motion variation of the first and second sets of non-overlapping windows using Eulerian and Lagrangian approach, respectively. The filtered signal repository 238 may be configured to store the first and second sets of noise cancelled signals. The consolidated signal repository 240 may be configured to store the consolidated Eulerian signal and the consolidated Lagrangian signal and the associated first and second spectrum. The consolidated signal repository 240 may further be configured to store the best consolidated signal determined by the processing circuitry 120. The HR repository 242 may be configured to store the Heart Rate (HR) information data determined from the best consolidated signal.
FIG. 3 illustrates a flow chart of a method 300 for the heart rate (HR) determination of a user wearing a face mask, in accordance with an exemplary aspect of the present disclosure.
At step 302, the data processing apparatus 106 may receive the plurality of images from the sensing unit 102.
At step 304, the data processing apparatus 106 may determine separately a non-occluded region and a masked facial region from each image of the plurality of images.
At step 306, the data processing apparatus 106 may determine the first and second sets of regions of interests (ROIs) for non-occluded and masked region of each image of the plurality of images, respectively.
At step 308, the data processing apparatus 106 may determine the first and second sets of non-overlapping windows corresponding to the first and second sets of ROIs of the non-occluded and masked region of each image of the plurality of images, respectively.
At step 310, the data processing apparatus 106 may generate the first and second sets of temporal signals based on sensing of color and motion variation of the first and second sets of non-overlapping windows using Eulerian and Lagrangian approach respectively.
At step 312, the data processing apparatus 106 may filter the first and the second sets of temporal signals to generate the first and second sets of noise cancelled signals.
In some aspects of the present disclosure, to generate the first and the second sets of noise cancelled signals, the data processing apparatus 106 may apply the band pass filter within a range of 0.7 Hz to 4.0 Hz on the first and the second sets of temporal signals to generate first and second sets of filtered signals. The data processing apparatus 106 may further generate the first and second sets of noise cancelled signals from the first and second sets of filtered signals using the blind source separation (BSS) technique.
At step 314, the data processing apparatus 106 may combine the first set of noise cancelled signals (i.e., the set of Eulerian signals) to generate the consolidated Eulerian signal and second set of noise cancelled signals (i.e., the set of Lagrangian signals) to generate the consolidated Lagrangian signal, respectively.
At step 316, the data processing apparatus 106 may generate the first and second pulse spectrum from the consolidated Eulerian signal and the consolidated Lagrangian signal, respectively, using the Fast Fourier Transformation (FFT) technique.
At step 318, the data processing apparatus 106 may select one of, the first and second pulse spectrum as best consolidated signal based on the set of quality parameters of the consolidated Eulerian signal and the consolidated Lagrangian signal.
At step 320, the data processing apparatus 106 may determine the Heart Rate (HR), based the best consolidated signal.
In some aspects of the present disclosure, the data processing apparatus 106 may determine the Heart Rate (HR) using the mathematical expression given by: HR = round (mf × 60), where mf denotes the frequency in Hz associated with the highest magnitude of the best consolidated signal and round denotes the rounding off operation.
At step 322, the data processing apparatus 106 may generate the validation signal based on the comparison of determined Heart Rate (HR) with the predetermined heart rate.
In some aspects of the present disclosure, the data processing apparatus 106 may be configured to enables one of, the generation of the first acknowledgement, when the determined HR is matched with the predefined heart rate and the generation of a first alert, when the determined HR is mismatched with the predefined heart rate.
As discussed earlier, there is a need for a system, a data processing apparatus, and a method for accurate and precise heart rate (HR) determination remotely using the rPPG signal collected from the non-occluded and the masked region separately from each image of the plurality of images, using a simultaneous consolidation of both the Eulerian and the Lagrangian approach. The system 100, by way of the data processing apparatus 106 through the method 300 provides accurate and precise determination of the heart rate remotely for a masked patient that efficiently utilizes the available resources. The system 100 provides a simultaneous consolidation of both Eulerian and Lagrangian approach, that provides reliable and accurate remote HR determination. The system 100 further provides a solution to accurate determination of Heart Rate remotely in case of inadequate illumination, unsuitable camera focus, or errors driven by human factors such as hair or headwear obscuring the observational area, resulting in erroneous HR determination.
The foregoing discussion of the present disclosure has been presented for purposes of illustration and description. It is not intended to limit the present disclosure to the form or forms disclosed herein. In the foregoing Detailed Description, for example, various features of the present disclosure are grouped together in one or more aspects, configurations, or aspects for the purpose of streamlining the disclosure. The features of the aspects, configurations, or aspects may be combined in alternate aspects, configurations, or aspects other than those discussed above. This method of disclosure is not to be interpreted as reflecting an intention the present disclosure requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed aspect, configuration, or aspect. Thus, the following claims are hereby incorporated into this Detailed Description, with each claim standing on its own as a separate aspect of the present disclosure.
Moreover, though the description of the present disclosure has included description of one or more aspects, configurations, or aspects and certain variations and modifications, other variations, combinations, and modifications are within the scope of the present disclosure, e.g., as may be within the skill and knowledge of those in the art, after understanding the present disclosure. It is intended to obtain rights which include alternative aspects, configurations, or aspects to the extent permitted, including alternate, interchangeable and/or equivalent structures, functions, ranges, or steps to those claimed, whether or not such alternate, interchangeable and/or equivalent structures, functions, ranges or steps are disclosed herein, and without intending to publicly dedicate any patentable subject matter.
As one skilled in the art will appreciate, the system 100 includes a number of functional blocks in the form of a number of units and/or engines. The functionality of each unit and/or engine goes beyond merely finding one or more computer algorithms to carry out one or more procedures and/or methods in the form of a predefined sequential manner, rather each engine explores adding up and/or obtaining one or more objectives contributing to an overall functionality of the system 100. Each unit and/or engine may not be limited to an algorithmic and/or coded form, rather may be implemented, by way of one or more hardware elements operating together to achieve one or more objectives contributing to the overall functionality of the system 100. Further, as it will be readily apparent to those skilled in the art, all the steps, methods and/or procedures of the system 100 are generic and procedural in nature and are not specific and sequential.
Certain terms are used throughout the following description and claims to refer to particular features or components. As one skilled in the art will appreciate, different persons may refer to the same feature or component by different names. This document does not intend to distinguish between components or features that differ in name but not structure or function. While various aspects of the present disclosure have been illustrated and described, it will be clear that the present disclosure is not limited to these aspects only. Numerous modifications, changes, variations, substitutions, and equivalents will be apparent to those skilled in the art, without departing from the spirit and scope of the present disclosure, as described in the claims. , Claims:1. A data processing apparatus (106) comprising:
processing circuitry (120) configured to (i) combine first set of noise cancelled signals to generate a consolidated Eulerian signal and second set of noise cancelled signals to generate a consolidated Lagrangian signal, respectively, (ii) generate first and second pulse spectrum from the consolidated Eulerian signal and the consolidated Lagrangian signal, respectively, using Fast Fourier Transformation (FFT) technique, (iii) select one of, the first and second pulse spectrum as best consolidated signal based on a set of quality parameters of the consolidated Eulerian signal and the consolidated Lagrangian signal, (iv) determine a Heart Rate (HR) value based on the best consolidated signal, and (v) generate a validation signal based on a comparison of the determined HR value with a pre-determined set of heart rate values.
2. The data processing apparatus (106) as claimed in claim 1, wherein, the validation signal enables one of, (i) generation of a first acknowledgement, when the determined HR is matched with the predefined heart rate and (ii) generation of a first alert, when the determined HR is mismatched with the predefined heart rate.
3. The data processing apparatus (106) as claimed in claim 1, wherein, prior to the generation of the first and second sets of pulse spectrums, the processing circuitry (120) is configured to (i) determine a non-occluded region and a masked region from each image of a plurality of images received from a sensing unit (102), (ii) determine first and second sets of Regions of Interests (ROIs) for the non-occluded region and the masked region respectively of each image of the plurality of images, respectively (iii) determine first and second set of non-overlapping windows corresponding to the first and second sets of Region of Interests (ROIs) respectively of each image from the plurality of images, (iv) generate first and second sets of temporal signals based on sensing of color and motion variation of the first and second sets of non-overlapping windows using Eulerian and Lagrangian approach respectively, and (v) filter the first and the second sets of temporal signals to generate the first and second sets of noise cancelled signals.
4. The data processing apparatus (106) as claimed in claim 1, wherein, to generate the first and the second sets of noise cancelled signals, the processing circuitry (120) is configured to (i) apply a band pass filter within a range of 0.7 Hz to 4.0 Hz on the first and the second sets of temporal signals to generate first and second sets of filtered signals and (ii) generate the first and second sets of noise cancelled signals from the first and second sets of filtered signals using a blind source separation (BSS) technique.
5. A system (100) comprising:
a sensing unit (102) configured to capture a plurality of images;
a data processing apparatus (104) that is coupled to the sensing unit (102), and comprising processing circuitry (120), wherein the processing circuitry (120) is configured to combine first set of noise cancelled signals to generate a consolidated Eulerian signal and second set of noise cancelled signals to generate a consolidated Lagrangian signal, respectively, (ii) generate first and second pulse spectrum from the consolidated Eulerian signal and the consolidated Lagrangian signal, respectively, using Fast Fourier Transformation (FFT) technique, (iii) select one of, the first and second pulse spectrum as best consolidated signal based on a set of quality parameters of the consolidated Eulerian signal and the consolidated Lagrangian signal, (iv) determine a Heart Rate (HR) value based on the best consolidated signal, and (v) generate a validation signal based on a comparison of the determined HR value with a pre-determined set of heart rate values.

6. The system (100) as claimed in claim 5, wherein, the validation signal enables one of, (i) generation of a first acknowledgement, when the determined HR is matched with the predefined heart rate and (ii) generation of a first alert, when the determined HR is mismatched with the predefined heart rate.
7. The system (100) as claimed in claim 5, wherein, prior to the generation of the first and second sets of pulse spectrums, the processing circuitry (120) is configured to (i) determine a non-occluded region and a masked facial region from each image of the plurality of images received from a sensing unit (102), (ii) determine first and second sets of Regions of Interests (ROIs) for the non-occluded region and the masked region respectively of each image of the plurality of images, (iii) determine first and second set of non-overlapping windows corresponding to the first and second sets of Region of Interests (ROIs) respectively of each image from the plurality of images, (iv) generate first and second sets of temporal signals based on sensing of color and motion variation of the first and second sets of non-overlapping windows using Eulerian and Lagrangian approach respectively, and (v) filter the first and the second sets of temporal signals to generate the first and second sets of noise cancelled signals.
8. The system (100) as claimed in claim 5, wherein, to generate the first and the second sets of noise cancelled signals, the processing circuitry (120) is configured to (i) apply a band pass filter within a range of 0.7 Hz to 4.0 Hz on the first and the second sets of temporal signals to generate first and second sets of filtered signals and (ii) generate the first and the second sets of noise cancelled signals from the first and the second sets of filtered signals using a blind source separation (BSS) technique, respectively.
9. A method (300) for determining heart rate of a user wearing a face mask comprising:
combining, by way of processing circuitry (120), first set of noise cancelled signals to generate a consolidated Eulerian signal and second set of noise cancelled signals to generate a consolidated Lagrangian signal, respectively;
generating, by way of the processing circuitry (120), first and second pulse spectrum from the consolidated Eulerian signal and the consolidated Lagrangian signal, respectively, using Fast Fourier Transformation (FFT) technique;
selecting, by way of the processing circuitry (120), one of, the first and second pulse spectrum as best consolidated signal based on a set of quality parameters of the consolidated Eulerian signal and the consolidated Lagrangian signal;
determining, by way of the processing circuitry (120) Heart Rate (HR) based on the best consolidated signal; and
generating, by way of the processing circuitry (120) a validation signal based on a comparison of the determined HR with a pre-determined heart rate.
10. The method (300) as claimed in claim 9, wherein, for generating the validation signal, the method (300) comprising (i) generating, by way of the processing circuitry (120), a first acknowledgement, when the determined HR is matched with the predefined heart rate, and (ii) generating, by way of the processing circuitry (120), a first alert, when the determined HR is mismatched with the predefined heart rate.
11. The method (300) as claimed in claim 9, wherein, prior to the generation of the first and second sets of pulse spectrums, the method (300) comprising (i) determining, by way of the processing circuitry (120), the non-occluded region and the masked facial region from each image of the plurality of images received from a sensing unit (102), (ii) determining, by way of the processing circuitry (120), first and second sets of Regions of Interests (ROIs) for the non-occluded region and the masked region respectively of each image of the plurality of images, and (iii) determining, by way of the processing circuitry (120), the first and the second set of non-overlapping windows corresponding to the Region of Interests (ROIs) of the non-occluded and the masked region respectively of each image from the plurality of images. (iv) generating, by way of the processing circuitry (120), first and second sets of temporal signals based on sensing of color and motion variation of first and second sets of non-overlapping windows using Eulerian and Lagrangian approach, respectively and (v) filtering, by way of the processing circuitry (120) the first and the second sets of temporal signals to generate the first and second sets of noise cancelled signals.
12. The method (300) as claimed in claim 9, for generating the first and the second sets of noise cancelled signals, wherein the method comprising (i) applying, by way of a band pass filter working within a range of 0.7 Hz to 4.0 Hz on the first and the second sets of temporal signals, by way of the processing circuitry (120) to generate first and second sets of filtered signals, and (ii) generating, by way of the processing circuitry (120) the first and second sets of noise cancelled signals from the first and second sets of filtered signals using a blind source separation (BSS) technique, respectively.