Sign In to Follow Application
View All Documents & Correspondence

Method And System For Self Supervised Training Of Deep Learning Based Time Series Models

Abstract: This disclosure relates to method and system for training of deep learning based time-series models based on self-supervised learning. The problem of missing data is taken care of by introducing missing-ness masks. The deep learning model for univariate and multivariate time series data is trained with the distorted input data using the self-supervised learning to reconstruct the masked input data. Herein, the one or more distortion techniques include quantization, insertion, deletion, and combination of the one or more such distortion techniques with random subsequence shuffling. Different distortion techniques in the form of reconstruction of masked input data are provided to solve. The deep learning model performs these different distortion techniques, which force the deep learning model to learn better features. It is to be noted that the system uses a lot of unlabeled data available cheaply as compared to the label or annotated data which is very hard to get.

Get Free WhatsApp Updates!
Notices, Deadlines & Correspondence

Patent Information

Application #
Filing Date
02 July 2021
Publication Number
01/2023
Publication Type
INA
Invention Field
COMPUTER SCIENCE
Status
Email
ip@legasis.in
Parent Application

Applicants

Tata Consultancy Services Limited
Nirmal Building, 9th Floor, Nariman Point, Mumbai - 400021, Maharashtra, India

Inventors

1. RATHORE, Pradeep
Tata Consultancy Services Limited, Tata Research Development & Design Centre, 54-B, Hadapsar Industrial Estate, Hadapsar, Pune - 411013, Maharashtra, India
2. BASAK, Arghya
Tata Consultancy Services Limited, Tata Research Development & Design Centre, 54-B, Hadapsar Industrial Estate, Hadapsar, Pune - 411013, Maharashtra, India
3. NISTALA, Sri Harsha
Tata Consultancy Services Limited, Tata Research Development & Design Centre, 54-B, Hadapsar Industrial Estate, Hadapsar, Pune - 411013, Maharashtra, India
4. RUNKANA, Venkataramana
Tata Consultancy Services Limited, Tata Research Development & Design Centre, 54-B, Hadapsar Industrial Estate, Hadapsar, Pune - 411013, Maharashtra, India

Specification

Claims: 1. A processor-implemented method (800) for self-supervised training of a deep learning based model with un-labelled time-series data comprising: receiving (802), via one or more hardware processors, a plurality of input data from one or more predefined data sources, wherein the plurality of input data is the un-labelled time-series data; preprocessing (804), via the one or more hardware processors, the received plurality of input data to: remove noise and outliers; and achieve uniform sampling frequency of the received plurality of input data; masking (806), via the one or more hardware processors, the preprocessed plurality of input data for one or more missing values of the plurality of input data; distorting (808), via the one or more hardware processors, the masked plurality of input data using one or more distortion techniques, wherein the one or more distortion techniques include quantization, insertion, deletion, and combination of the one or more such distortion techniques with random subsequence shuffling; and training (810), via the one or more hardware processors, the deep learning based model with the distorted plurality of input data using self-supervised learning to reconstruct the masked plurality of input data. 2. The processor-implemented method (800) of claim 1, wherein the time-series data includes univariate and multi-variate datasets. 3. The processor-implemented method (800) of claim 1, wherein the quantization technique includes discretization of the time series signal. 4. The processor-implemented method (800) of claim 1, wherein the insertion technique comprises of inserting one or more values between selected pairs of instances in the time series data. 5. The processor-implemented method (800) of claim 1, wherein the deletion technique comprises of deletion of one or more randomly selected instances in the time series data. 6. The system (100) for self-supervised training of a deep learning based model with un-labelled time-series data comprising: an input/output interface (104) for receiving a plurality of input data from one or more predefined data sources, wherein the plurality of input data is un-labelled time-series data, wherein the time-series data includes univariate and multi-variate datasets; one or more hardware processors; a memory in communication with the one or more hardware processors (108), wherein the one or more hardware processors are configured to execute programmed instructions stored in the memory, to: preprocess the received plurality of input data to remove noise and one or more outliers from the received plurality of input data and achieve uniform sampling frequency of the time-series data; mask the preprocessed plurality of input data for one or more missing values of the input data; distort the masked plurality of input data using one or more distortion techniques, wherein the one or more distortion techniques include quantization, insertion, deletion, and combination of the one or more distortion techniques with random subsequence shuffling; and train the deep learning based model with the distorted plurality of input data using the self-supervised learning to reconstruct the masked plurality of input data. 7. A non-transitory computer readable medium storing one or more instructions which when executed by one or more processors on a system, cause the one or more processors to perform method for self-supervised training of a deep learning based model with un-labelled time-series data comprising: receiving, via one or more hardware processors, a plurality of input data from one or more predefined data sources, wherein the plurality of input data is un-labelled time-series data; preprocessing, via the one or more hardware processors, the received plurality of input data to: remove noise and outliers from the received plurality of input data; and achieve uniform sampling frequency of the received plurality of input data; masking, via the one or more hardware processors, the preprocessed plurality of input data for one or more missing values of the input data; distorting, via the one or more hardware processors, the masked plurality of input data using one or more distortion techniques, wherein the one or more distortion techniques include quantization, insertion, deletion, and combination of the one or more distortion techniques with random subsequence shuffling; and training, via the one or more hardware processors, the deep learning based model with the distorted plurality of input data using the self-supervised learning to reconstruct the masked plurality of input of data. , Description:FORM 2 THE PATENTS ACT, 1970 (39 of 1970) & THE PATENT RULES, 2003 COMPLETE SPECIFICATION (See Section 10 and Rule 13) Title of invention: METHOD AND SYSTEM FOR SELF SUPERVISED TRAINING OF DEEP LEARNING BASED TIME SERIES MODELS Applicant: Tata Consultancy Services Limited A company Incorporated in India under the Companies Act, 1956 Having address: Nirmal Building, 9th Floor, Nariman Point, Mumbai 400021, Maharashtra, India The following specification particularly describes the invention and the manner in which it is to be performed. TECHNICAL FIELD The disclosure herein generally relates to the field of training of deep learning models and specifically, to a method and system for self-supervised training of a deep-learning based model with time-series data. BACKGROUND Deep learning based models have become a part of our life. These models are present in smart watches (for predicting abnormal heart disease), mobile phones, manufacturing industries (for predictive maintenance), self-driven cars etc. The training of deep learning based models is a very complex task and requires huge amount of labelled data. Getting labelled data in industries is very limited and difficult because it requires expensive manual labelling by experts. Large amount of labelled data is either very limited in industries or very expensive to acquire because labelling or annotation of unlabeled data require lot of human expertise and manual labor. In real-time system, many times there is lot of missing data due to sensor fault, communication fault or some unavoidable reasons but there is no suitable approach to utilize time series data containing lot of missing values without imputation for tasks such as classification, regression and anomaly detection. SUMMARY Embodiments of the disclosure present technological improvements as solutions to one or more of the above-mentioned technical problems recognized by the inventors in conventional systems. For example, in one embodiment, a system and method for self-supervised training of a deep learning based model with un-labelled time-series data is provided. In one aspect, a processor-implemented method for self-supervised training of a deep learning based model with un-labelled time-series data is provided. The method includes one or more steps such as receiving a plurality of input data from one or more predefined data sources, wherein the plurality of input data is un-labelled time-series data, and pre-processing the received plurality of input data to remove noise and outliers from the received plurality of input data and to achieve uniform sampling frequency of the received plurality of input data. The processor-implemented method further includes masking the preprocessed plurality of input data for one or more missing values of the input data, distorting the masked plurality of input data using one or more distortion techniques, and training the deep learning based model with the distorted plurality of input data using the self-supervised learning to reconstruct the masked plurality of input of data. Herein, the one or more distortion techniques include quantization, insertion, deletion, and combination of the one or more distortion techniques with random subsequence shuffling. In another aspect, a system for self-supervised training of a deep learning based model using un-labelled time-series data is provided. The system includes an input/output interface configured to receive a plurality of input data from one or more predefined data sources, at least one memory storing a plurality of instructions and one or more hardware processors communicatively coupled with the at least one memory, wherein the one or more hardware processors are configured to execute the plurality of instructions stored in the at least one memory. Herein, the plurality of input data is un-labelled time-series data and the time-series data includes univariate and multi-variate datasets. Further, the system is configured to pre-process the received plurality of input data to remove noise and one or more outliers from the received plurality of input data and achieve uniform sampling frequency of the time-series data. Further, the system is configured to mask the preprocessed plurality of input data for one or more missing values of the input data and distort the masked plurality of input data using one or more distortion techniques. Herein one or more distortion techniques include quantization, insertion, deletion, and combination of the one or more distortion techniques with random subsequence shuffling. And, finally, the system is configured to train the deep learning based model with the distorted plurality of input data using the self-supervised learning to reconstruct the masked plurality of input data. In yet another aspect, there are provided one or more non-transitory machine-readable information storage mediums comprising one or more instructions, which when executed by one or more hardware processors causes a method for self-supervised training of a deep learning based model with un-labelled time-series data is provided. The method includes one or more steps such as receiving a plurality of input data from one or more predefined data sources, wherein the plurality of input data is un-labelled time-series data, and pre-processing the received plurality of input data to remove noise and outliers from the received plurality of input data and to achieve uniform sampling frequency of the received plurality of input data. The processor-implemented method includes masking the preprocessed plurality of input data for one or more missing values of the input data, distorting the masked plurality of input data using one or more distortion techniques, and training the deep learning based model with the distorted plurality of input data using the self-supervised learning to reconstruct the masked plurality of input data. Herein, the one or more distortion techniques include quantization, insertion, deletion, and combination of the one or more distortion techniques with random subsequence shuffling. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed. BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and, together with the description, serve to explain the disclosed principles: FIG. 1 illustrates an exemplary system for training of time-series deep learning based models based on self-supervised learning, according to an embodiment of the present disclosure. FIG. 2 illustrates a functional flow diagram for training of time-series deep learning based models based on self-supervised learning, according to an embodiment of the present disclosure FIG. 3 is a graphical illustration showing working of deletion-based distortion technique in a self-supervised learning, according to an embodiment of the present disclosure. FIG. 4 is a graphical representation to illustrate working of insertion-based distortion technique in a self-supervised learning, according to an embodiment of the present disclosure. FIG. 5 is a graphical representation to illustrate working of quantization-based distortion technique in a self-supervised learning, in accordance with some embodiments of the present disclosure. FIG. 6 is a graphical representation to illustrate working of shuffling-based data distortion technique in a self-supervised learning, in accordance with some embodiments of the present disclosure. FIG. 7 is a graphical representation to illustrate the data of flame detector voltage of gas turbine combustor corresponding to abnormal operation, in accordance with some embodiments of the present disclosure. FIG. 8 illustrates a flowchart of a processor-implemented method for self-supervised training of time-series deep learning based models based on self-supervised learning, in according to an embodiment of the present disclosure. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed. DETAILED DESCRIPTION OF EMBODIMENTS Exemplary embodiments are described with reference to the accompanying drawings. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. Wherever convenient, the same reference numbers are used throughout the drawings to refer to the same or like parts. While examples and features of disclosed principles are described herein, modifications, adaptations, and other implementations are possible without departing from the spirit and scope of the disclosed embodiments. It is intended that the following detailed description be considered as exemplary only, with the true scope and spirit being indicated by the following claims. The embodiments herein provide a method and system for training a deep learning based model with a time-series data. Referring now to the drawings, and more particularly to FIG. 1 through 8, where similar reference characters denote corresponding features consistently throughout the figures, there are shown preferred embodiments and these embodiments are described in the context of the following exemplary system and/or method. FIG. 1 illustrates a block diagram of a system (100) for training of time-series deep learning based models based on self-supervised learning, in accordance with an example embodiment. Although the present disclosure is explained considering that the system (100) is implemented on a server, it may be understood that the system (100) may comprises one or more computing devices (102), such as a laptop computer, a desktop computer, a notebook, a workstation, a cloud-based computing environment and the like. It will be understood that the system 100 may be accessed through one or more input/output interfaces 104-1, 104-2... 104-N, collectively referred to as I/O interface (104). Examples of the I/O interface (104) may include, but are not limited to, a user interface, a portable computer, a personal digital assistant, a handheld device, a smartphone, a tablet computer, a workstation, and the like. The I/O interface (104) are communicatively coupled to the system (100) through a network (106). In an embodiment, the network (106) may be a wireless or a wired network, or a combination thereof. In an example, the network (106) can be implemented as a computer network, as one of the different types of networks, such as virtual private network (VPN), intranet, local area network (LAN), wide area network (WAN), the internet, and such. The network 106 may either be a dedicated network or a shared network, which represents an association of the different types of networks that use a variety of protocols, for example, Hypertext Transfer Protocol (HTTP), Transmission Control Protocol/Internet Protocol (TCP/IP), and Wireless Application Protocol (WAP), to communicate with each other. Further, the network (106) may include a variety of network devices, including routers, bridges, servers, computing devices, storage devices. The network devices within the network (106) may interact with the system (100) through communication links. The system (100) supports various connectivity options such as BLUETOOTH®, USB, ZigBee and other cellular services. The network environment enables connection of various components of the system (100) using any communication link including Internet, WAN, MAN, and so on. In an exemplary embodiment, the system (100) is implemented to operate as a stand-alone device. In another embodiment, the system (100) may be implemented to work as a loosely coupled device to a smart computing environment. The components and functionalities of the system (100) are described further in detail. Referring FIG. 2, illustrating a functional flow diagram to illustrate the system for training of time-series deep learning based models based on self-supervised learning. It would be appreciated that large amount of labelled data is either very limited in industries or very expensive to acquire because labelling or annotation of unlabeled data require lot of expertise and manual labor. Further, in real systems, many times there is a lot of missing data due to one or more sensors’ fault, communication fault or some unavoidable reasons but there is no guideline to utilize time series data containing lot of missing values without imputation for distortion techniques such as classification, regression and anomaly detection. In one embodiment, a concept of insertion, deletion, quantization, and masking of time series is introduced for self-supervised learning. Firstly, missingness mask is generated corresponding to missing values present in the data. The missingness mask is univariate and multivariate for univariate and multivariate data respectively. It is to be noted that herein the deep learning based models are trained to learn in a self-supervised way to predict the masked signal. Different distortion techniques for the deep learning based model are provided to solve using a masked time-series data. The deep learning based model performs these different distortion techniques, which force the deep learning based model to learn better features of the time-series data. One such task is insertion and deletion of timestamps in time-series data. The system is configured to reconstruct the original masked signal for the entire time window. The system ignores those timestamps which have missing data originally while calculating loss. It is to be noted that the system uses a lot of un-labeled time-series data available cheaply as compared to the label or annotated time-series data which is very hard to get. The system is configured to handle missing data appropriately without need of imputation because of using appropriate missingness mask In another embodiment, wherein the system (100) is configured for training of time-series deep learning based models based on self-supervised learning. The system (100) comprises at least one memory with a plurality of instructions, one or more databases (112), one or more input/output interfaces (104) and one or more hardware processors (108) which are communicatively coupled with the at least one memory to execute a plurality of modules (110) therein. Further, the system comprises a pre-processing module, a masking module, a data distortion module, and a training module. Herein, the input/output interface (106) is configured to receive a plurality of input data from one or more predefined data sources. Herein, the plurality of input data includes univariate and multi-variate datasets which is un-labelled time-series data. In the preferred embodiment, the pre-processing module of the system (100) is configured to perform pre-processing of the received plurality of input data from one or more sensors. Pre-processing involves outlier identification and removal of redundant data, filtering of data, and unification of sampling frequency of the time-series data. In the preferred embodiment, the masking module of the system (100) is configured to mask the preprocessed plurality of input data for one or more missing values of the input data. It is to be noted that the one or more missing values are present in the input univariate or multivariate data, therefore the system is configured to apply missingness mask corresponding to the one or more missing values. Let X?R^MXN be the input data to the deep learning based model where M are the number of samples or instance and N are the number of features to the deep learning based model. The system introduces mask, A?[0,1] of dimension MXN where 0 represent the missing value of the sensor and 1 represent the presence of the sensor value. In the preferred embodiment, the data distortion module of the system (100) is configured to distort the masked plurality of input data using one or more distortion techniques. The one or more distortion techniques include quantization, insertion, deletion, and combination of the one or more distortion techniques with random subsequence shuffling. It is to be noted that the one or more distortion techniques are known as distortion techniques for the deep learning based model. These distortion techniques include reconstruction of original signal from the distorted plurality of input data. Herein, each of the one or more distortion techniques comprise of few randomly chosen samples from the original masked data along with different types of distortion for the same set of samples. Kindly note that we provide randomly chosen masked input data along with the distorted version of these samples in the same minibatch while training deep learning based models. In another embodiment, wherein T1, T2, T3 ….Tn are the distortion techniques to be performed by the deep learning based model and x ? X is the subset of unlabeled input data. Let x1, x2, x3….xn be the input samples in a minibatch . The self-supervised training calculates x_(T )^1,x_T^2,x_T^3 ,……x_T^n for each of the distortion techniques T. For each sample in a minibatch, multiple manipulated samples are generated using different distortion techniques. Further, the system configured to calculate loss considering missingness mask appropriately and update weights using back propagation. 5th Position 0. 0.43 0.78 0.97 0.97 0.78 0.43 0. -0.43 -0.78 -0.97 -0.97 -0.78 -0.43 0 0 0.43 0.78 0.97 0.97 0.43 0 -0.43 -0.78 -0.97 -0.97 -0.78 -0.43 0 0 0 0.43 0.78 0.97 0.97 0.85 0.78 0.43 0. -0.43 -0.78 -0.97 -0.97 -0.78 -0.43 Table 1 Referring Table 1, wherein the reconstruction of original data from deleted and inserted timestamps is elaborated. In the table 1, the first row denotes the original univariate time series data, 2nd and 3rd rows represent the data having deletion and insertion operations, respectively. The system is configured to first select the index randomly to be deleted or inserted. In the case of deletion, the system deletes the values of the time series corresponding to the index selected and shift the remaining data so that there is no gap. Further, the system is configured to add padding towards the end or in the starting of the time-series data to maintain the length of the data same before and after the deletion operation. It would be appreciated that the deletion of an instance from data is different from the replacement. Similarly, in the case of insertion, the system inserts random values at randomly selected index positions. Herein, the system deletes the data from last or starting to maintain the length of the data after insertion operation same as before the insertion operation. It is to be noted that the length of the padding in case of deletion is same as the number of deleted instances. Similarly, in case of insertion the length of the cropped portion of time series data is same as number of newly inserted instances. This insertion and deletion operation can be at multiple positions of a time series data. Also, the structure of these deletion and insertion operations can be of different type e.g. random or intermittent. In case of multivariate time series data, the system performs insertion and deletion operations on each of the sensor data independently. In one example, wherein to explain concepts of deletion, insertion, quantization and shuffling respectively, a flame detector voltage of industrial gas turbine combustor is used in accordance with some embodiments of the present disclosure. During the startup of the gas turbine, the flame detector voltage of combustor follows a specific trend. In the starting it is zero and then follow an approximately linear increase till a predefined threshold. Whenever there is some problem in the startup of the gas turbine, it’s gets reflected in the abnormal trend of the combustor flame voltage as shown in Fig 7. One of the use case is to develop machine learning models to identify these abnormal trends. Since, manual generation of these abnormal patterns by introducing faults in the physical gas turbine is a major economic and implementation challenge. Herein, the self-supervised learning helps in generating more data during training of machine learning models and acts as data augmentation. FIG. 3 through FIG. 6 shows graphical representation to illustrate the novel concepts of deletion, insertion, quantization, and shuffling, respectively. Referring FIG. 3, a graphical representation (300) to illustrate original and modified time-series data of flame detector voltage of gas turbine combustor corresponding to a normal operation. The time-series data is normalized between 0 and 1. FIG. 3 illustrates the working of the deletion-based distortion technique in a self-supervised learning algorithm. Instances corresponding to the timesteps 36-45 is deleted and the corresponding modified data is represented by dotted line. Since, its necessary to keep the length of the data same before and after deletion-based data manipulation, the system has inserted padding towards the end of the data. The length of the padding is same as the length of the deleted part. Last value of the end data point has been taken as the value of each element in the padding. In another embodiment, referring FIG. 4, a graphical representation (400) represents the original and modified data of flame detector voltage of gas turbine combustor corresponding to normal operation. The data is normalized between 0 and 1. The graph depicts the working of the insertion-based distortion technique in a self-supervised learning algorithm. Herein, 10 new instances are inserted at index 36-45 and corresponding modified data is represented by a dotted line. The values corresponding to these instances is interpolated such that difference between successive inserted values remains same. Since, it is necessary to keep the length of the data same before and after insertion-based data manipulation, the system has deleted 10 instances of the data towards the end. The number of deleted instances is same as the length of the inserted part. In another embodiment, wherein to reconstruct the original data using quantized signal, the system is configured to generate quantized data x_q ??R?^AXN A

Documents

Application Documents

# Name Date
1 202121029887-STATEMENT OF UNDERTAKING (FORM 3) [02-07-2021(online)].pdf 2021-07-02
2 202121029887-REQUEST FOR EXAMINATION (FORM-18) [02-07-2021(online)].pdf 2021-07-02
3 202121029887-FORM 18 [02-07-2021(online)].pdf 2021-07-02
4 202121029887-FORM 1 [02-07-2021(online)].pdf 2021-07-02
5 202121029887-FIGURE OF ABSTRACT [02-07-2021(online)].jpg 2021-07-02
6 202121029887-DRAWINGS [02-07-2021(online)].pdf 2021-07-02
7 202121029887-DECLARATION OF INVENTORSHIP (FORM 5) [02-07-2021(online)].pdf 2021-07-02
8 202121029887-COMPLETE SPECIFICATION [02-07-2021(online)].pdf 2021-07-02
9 202121029887-Proof of Right [15-07-2021(online)].pdf 2021-07-15
10 Abstract1..jpg 2022-01-13
11 202121029887-Request Letter-Correspondence [08-04-2022(online)].pdf 2022-04-08
12 202121029887-Power of Attorney [08-04-2022(online)].pdf 2022-04-08
13 202121029887-FORM-26 [08-04-2022(online)].pdf 2022-04-08
14 202121029887-Form 1 (Submitted on date of filing) [08-04-2022(online)].pdf 2022-04-08
15 202121029887-Covering Letter [08-04-2022(online)].pdf 2022-04-08
16 202121029887-CERTIFIED COPIES TRANSMISSION TO IB [08-04-2022(online)].pdf 2022-04-08
17 202121029887 CORRESPONDANCE (IPO) WIPO DAS 12-04-2022.pdf 2022-04-12
18 202121029887-FORM 3 [05-05-2022(online)].pdf 2022-05-05
19 202121029887-FER.pdf 2023-03-02
20 202121029887-OTHERS [12-07-2023(online)].pdf 2023-07-12
21 202121029887-FER_SER_REPLY [12-07-2023(online)].pdf 2023-07-12
22 202121029887-COMPLETE SPECIFICATION [12-07-2023(online)].pdf 2023-07-12
23 202121029887-CLAIMS [12-07-2023(online)].pdf 2023-07-12

Search Strategy

1 SearchStrategy(2)E_02-03-2023.pdf