FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENT RULES, 2003
COMPLETE SPECIFICATION
(See Section 10 and Rule 13)
Title of invention:
SYSTEM AND METHOD TO PROVIDE ANALYTICAL PROCESSING OF DATA IN A DISTRIBUTED DATA STORAGE SYSTEMS
Applicant
TATA Consultancy Services Limited A Company Incorporated in India under The Companies Act, 1956
Having address: Nirmal Building, 9th Floor,
Nariman Point, Mumbai 400021, Maharashtra, India
The following specification particularly describes the invention and the manner in which it is to be performed.

FIELD OF INVENTION
The present invention in general relates to a method, system and a computer program product to process data in distributed data storage system. More particularly, the present invention discloses on analytical processing of data by using the processing power of the distributed data storage system.
BACKGROUND OF THE INVENTION
The fast and unlimited advancement in technology has started producing large amount of data for which large databases are needed where this data could not only be stored but which may also prove to be useful while doing research and analysis. Although, so many databases are there, where large data could be easily stored and used. One such type of database includes but is not limited to Hadoop where data in mass is stored. However, along with storing, management-of this big data (in terabytes or more) is-really difficult ProbTems are continuously faced while extracting data, transforming it into a desired format and then storing the same in a desired destination. Moreover, valuable visualization as per the user's requirement is an important factor while storing and using the data from the big data storage systems.
Further, handling big data with requires massive software and reasonably high number of servers. There are many existing ETL tools available in the market to counter the issues of analyzing big data; however, the existing ETL tools are either quite complex or insufficient to handle big data.
With this drawback, most of the industries in order to handle and maintain big data are utilizing distributed data storage systems like Hadoop technology and coming up with various ETL tools to support their business requirements. Distributed data storage systems has gathered momentum as a mechanism for dealing big data wherein enterprises seek to derive value from the rapidly growing amount of data in their computer systems. Most of the existing ETL operations in distributed environment are performed using map reduce codes, however, understanding them and coding them

require immense effort and also requires custom programming to develop, maintain and support.
To counter this issue, some of the existing technologies have come up by providing plugins on different technologies processing big data like Hadoop, but still there is a challenge as in almost all the existing technologies, the ETL processing is performed outside of the big data storage area and they extract and load data from or to any distributed data storage system. Moreover, in some of the cases, where processing power of distributed data storage systems has been utilized, no analysis could be done while extracting the data or while loading in order to refine the data for saving cost and time.
Therefore, to address the above discussed concerns and to create a better customer experience, a cost-effective and convenient E@TL (Extract analytical transformation and loading) tool or a similar system is needed that can make better decisions regarding data transformations and.enrichments of raw data utilizing distributed. storage_and_distributed processing power. The system or tool should be useful enough to eliminate a great deal of complexity from the data developer wherein the raw data is processed within the distributed environment.
OBJECTS OF THE INVENTION
The primary object of the present invention is to provide a system, method and computer program product capable of processing the data by utilizing processing power of a distributed data storage system.
The other object of the invention is to perform elementary analysis over the extracted data in order to extract the useful data.
Another object of the invention is to provide a various type of mapping (multi source to multi destination).
Yet another object of the invention is to categorize the data as valid and invalid after performing a repetitive validation process.

Yet another object of the invention is to provide a secondary analysis after doing the transformation in order to provide a refined data which may be used for resolving a user's search query to the distributed data storage system
Yet another object of the invention is to perform the extraction and transformation in a single stage i.e. to perform the transformation while data is being extracted.
Yet another object of the invention is to provide a computer program product in a form of an E@TL (Extract Analytical Transformation and Loading) tool which is capable of performing the analytical processing by utilizing the processing power of distributed data storage and processing environment like Hadoop.
SUMMARY OF THE INVENTION
The present invention provides a system for analytical processing of data in a distributed
data storage system. The system comprises of a data extraction module configured, to
perform one or more elementary analytical operations to extract data from one or more source database in one or more data format. Said data is extracted regardless of the presence of constraints and structure of the source database. The system further comprises of a processing engine configured to perform one or more operations for data refinement in order to categorize the data, while the data is being extracted. The processing engine further comprises of a mapping module configured to perform one or more type of mapping operations over the categorized data by using one or more predefined mapping rules and a transformation module configured to perform a secondary analytical transforming operation based on one or more pre-defined business rules, over the mapped categorized data in order to obtain a transformed categorized data. The data is processed in a manner such that, only the transformed categorized data is stored in a target area present inside the distributed data storage system for one or more utility purpose.
The present invention further provides a method for analytical processing of data in a distributed data storage system. The method comprises of processor implemented steps

of performing one or more elementary analytical operations to extract data from one or more source database in one or more data format, said data is extracted regardless of the presence of constraints on and structure of the source database and processing said extracted data by performing one or more operations for data refinement in order to categorize the data, while the data is being extracted. The processing further comprises of steps of performing one or more type of mapping operations over the categorized data by using one or more pre-defined mapping rules and performing a secondary analytical transforming operation based on one or more pre-defined business rules, over the mapped categorized data in order to obtain a transformed categorized data. The data is processed
in a manner such that, only the transformed categorized data is stored in a target area
i present inside the distributed data storage system for one or more utility purpose.
i
i The present invention also provides a computer program product for analytical
processingof data-in a distributed data storage system with respect to one or more query.
The computer program product comprises of a user interface configured to first
authenticate and receive said query from one or more user and a data extraction module
configured to perform an elementary analysis to extract data from one or more source
database in one or more data format with respect to the user's query; said data is
extracted regardless of the presence of constraints on and structure of the source
database. The computer program product further comprises of a processing engine
associated with the distributed data storage system configured to categorize the data in a
repetitive manner and further analytically transform said categorized data, as per the
user's requirements with respect to said query. The processing engine further maps the
data in one or more types in order to obtain a transformed categorized data with respect
to the user's query. The computer program product further comprises of an output
generation module configured to retrieve results from the transformed categorized data
with respect to the user's query.

BRIEF DESCRIPTION OF DRAWINGS
Figure 1 illustrates system architecture for analytical processing of data in a distributed data storage system in accordance with an embodiment of the invention.
Figure 2 illustrates one or more components associated with a computer program product for analytical data processing in accordance with an alternate embodiment of invention.
Figure 3(a) illustrates detailed system for data validation and transformation in accordance with an alternate embodiment of the invention.
Figure 3(b) illustrates analytical transformation/processing of data in accordance with an
i alternate embodiment of the invention.
i DETAILED DESCRIPTION
Some embodiments of this invention, illustrating its features, will now be discussed:
The words "comprising", "having", "containing", and "including", and other forms thereof, are intended to be equivalent in meaning and be open ended in that an item or items following any one of these words is not meant to be an exhaustive listing of such item or items, or meant to be limited to only the listed item or items.
It must also be noted that as used herein and in the appended claims, the singular forms "a", "an", and "the" include plural references unless the context clearly dictates otherwise. Although any systems, methods, apparatuses, and devices similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present invention, the preferred, systems and parts are now described. In the following description for the purpose of explanation and understanding reference has been made to numerous embodiments for which the intent is not to limit the scope of the invention.
One or more components of the invention are described as module for the understanding of the specification. For example, a module may include self-contained component in a

hardware circuit comprising of logical gate, semiconductor device, integrated circuits or any other discrete component. The module may also be a part of any software programme executed by any hardware entity for example processor. The implementation of module as a software programme may include a set of logical instructions to be executed by the processor or any other hardware entity. Further a module may be incorporated with the set of instructions or a programme by means of an interface.
The disclosed embodiments are merely exemplary of the invention, which may be embodied in various forms.
The present invention provides a system and method for performing an analytical processing in a distributed data storage system. The data is extracted from a plurality of source databases/systems after performing elementary analytical operations over the data thus extracted. Such data is then processed by using the proposed system for categorizing the data after performing operations for data refinement. The data is further processed by performing operations like mapping, transformation etc. All these operations are performed in a single stage i.e. at a time when the data is being extracted. At the time of transformation a secondary analysis is carried out to further categorize the data. The transformed categorized data is then stored in a target area present inside the distributed data storage system.
The use of Hadoop is merely an example of distributed or big data storage system and other similar databases and processing system may also be used. This is to be understood that the intent of using Hadoop is not to limit the scope of invention.
In accordance with an embodiment, referring to figure 1, the system (100) for analytical processing of data comprises of a data extraction module (104) configured to extract data from one or more source database/systems (102) in one or more format and a processing engine (106) for categorizing the data. The processing engine (106) being the most important component of the proposed invention further comprises of a mapping module

(108) and a transformation module (110). The transformed categorized data is further stored in a target area present inside the distributed data storage system (112).
The data extraction module (104) is configured to extract data from one or more source databases (102) in one or more format. The data is extracted by performing first stage i.e. elementary analytical operations over the extracted data as shown in figure 3(b). These elementary analytical operations form the crucial part of the E@TL (Extract Analytical Transformation and Loading) process. The present invention thus has the ability to ingest massive data in as-is condition and does not have to be sanitized. The distributed data storage (112) system may include but is not limited to Hadoop (data storage system).
The data could be both traditional transactional data that include POS (point-of-sale) transactions, call details records, general ledger records, call center transactions and the like. The data also could be unstructured data, i.e., consumer comments, descriptions,
web logs, etc. and also could be social.media sites such as LinkedIn, Facebook, Twitter,
or the like. Regardless of the structure of the data, the data could be rapidly loaded into distributed data storage in as-is condition and is available to downstream for analytical purposes. With the proposed system, the required data does not have to be loaded and can be operated when the data is copied into distributed data storage within the distributed system like HDFS.
Structured elements of a database may also be represented in various formats such as XML (extensible markup language), JSON, CSV etc. as input data. For example, the input data could be a database of journal articles having attributes such as author, title, publication, date etc., but included as a single data element.
In accordance with an embodiment, for the purpose of extraction, one or more transporters (not shown in figure) are used. These transporters are used to import the data from the source database (RDBMS) to distributed data storage system. At the time of data extraction, the elementary analysis is performed over this data. Elementary analysis may be done on the data which is going to be imported into the distributed data storage

system (112) by using various algorithms such as frequent pattern matching algorithm to come up with patterns. This elementary analysis helps in condition checking whether to import entire table/entire dataset or only same records after some transformation.
The system (100) further comprises of a processing engine (106) configured to refine the
data before storing it into the distributed data storage system. The refinement is done in
i
order to filter the extracted data so that the data could be categorized.
The input data obtained from the source system in various formats are then parsed into a single format that is appropriate for further analytical transformation processing. The data could be transformed at the time of extracting the source dataset without staging since the extraction of the data and the transformation of the data could be done simultaneously in a single step with this system (100). After the data has been staged in distributed or big data storage system, the data is further processed for transformation.
Referring now to figure 2, In case of flat files, the data is parsed and stored in distributed or big data storage system. Further, the one or more operations for data refinement like mapping and validation are applied over this parsed data. The parsed data having undergone extraction and transformation is then stored into the staging area (130) (distributed data storage system). The parsed data stored in the staging area of distributed data storage system (130) is then subjected to validation as shown in validation table (140) to control the type of data or values that needs to be validated. The validation table (140) validates the parsed data that includes data validation, structure validation and custom validation with generic scripts which can run in a distributed manner. The parsed data stored in the staging area of distributed data storage system (112) after being validated is then sent to the validation check (142) and then segregated into two different databases depending on the data being passed or failed. The validated data that gets passed through the validation steps of data, structure, and custom validation, is stored in a database as valid records and the data that gets failed is stored in a database as invalid records. The valid records stored in the valid database are then passed through a check point table (150) that includes iterations and updates where the check point table (150)

keeps track of the state of each iteration variable for each pass through the iteration and is updated accordingly.
The invalid records in the invalid database are processed again at least upto 3 times. After performing the iteration and updates in the check point table, the valid records stored in the valid database are then passed through a mapping table (160) to map the records based on the business requirement and concatenation by means of the mapping module. The mapping module (108) will perform one or more mapping operations over the categorized data (valid data).
Mapping could be from one-to- many, many-to-one, one-to-one, or many-to-many source and destinations respectively. The mapping includes data mapping from one table to another table from two or more data paths and also includes complex splitting of data into multiple output paths depending on the input conditions. The data obtained could be from various sources such as Oracle, DB2 having different schemas and hence are subjected to data transformation process, for each set of source data.
The proposed system (100) allows loading data from various sources with different schemas in a single step and the transformation process (170) is applied to the data that has been staged and stored into multiple destinations. For example, with a single transformation process, data could be loaded into distributed data storage system (112) from source systems (102) (like Oracle and DB2), both having different schemas, and mapping table could be applied in distributed file system through a single step for both source files i.e. Oracle and SQL, and the output could be stored in multiple files or tables in distributed orbig data storage area.
Referring to figure 3(b), the processing engine (106) further comprises of a transformation module (110) configured to perform a secondary analytical transformation based on pre-defined business rules over the mapped categorized data in order to obtain a transformed categorized data. Secondary analytics forms the second stage of the analytics performed by the E@TL (Extract Analytical Transformation and Loading) tool.

Secondary analytics can then be done on the transformed data based on the business use cases for analytics, for e.g.: Market Basket Analysis, Predictive analysis is performed on the raw data, once the data has been transformed in the required format as per the requirement of the analysis.
In an exemplary embodiment of the invention, consider retail data sets as source data.
i Secondary analytics such as market basket analysis can be easily performed using E@TL
(Extract Analysis Transformation and Loading) tool and based on this analysis the data
i
can be transformed accordingly and stored in the target area. Sentiment analysis or fraud analysis combining transaction data with textual and other data can also be performed using E@TL (Extract Analytical Transformation and Loading) tool.
After performing the transformation process, the valid records are transformed and are subjected to mapping, so that they can be reflected with fact and dimension tables. The status in checkpoint tahleJs updated again with the overwritten data with_a_mapped_ data of the existing information, obtained through the mapping table. The processed records are then sent to the distributed data storage system target area (180) and the invalid records information is sent back. The entire E@TL (Extract Analytical Transformation and Loading) process is performed using the distributed environment and the data is processed in the same database while processing.
In this manner, the mapped and analytically transformed (categorized) data is then stored in a target area (180) present inside the distributed data storage system (112).
Further, the transporters may then be used for transporting this categorized transformed data (stored in distributed data storage system) to RDBMS where the data may be further used for one or more utility purpose. The utility purpose may include but is not limited to a market based analysis, a predictive data analysis or a combination thereof.
Referring to figure 3 (a), the present invention further provides a computer program
i product (300) for analytical processing of data in a distributed data storage system (112)
with respect to one or more query.

The computer program product (300) comprises of a user interface (103) configured to first authenticate one or more user and then receive one or more query from said user. The user interface (103) is further provided with an authentication module (not shown in figure) configured to perform authentication through a security feature such as 'Kerberos' which is a networking authentication protocol to prove identity to one another in a secure manner.
After the authentication process, a user can request for specific information from the distributed data storage system. Upon request for information of data from a user the data extraction module (as described above) collects data from various source databases (102). The data could be in structured or unstructured form. Structured elements of a database may also be represented in various formats such as XML (extensible markup language), JSON, CSV etc. as input data. For example, the input data could be a
database-of journal articles having attributes such as-author, title, publication, date etc.,
but included as a single data element.
Further this extracted data (after performing elementary operations) is processed for refinement. The processing engine (106) is provided with the distributed data storage system (112) to perform this refinement by performing mapping operations and analytical transformation operations over the data thus extracted and further produce categorized transformed data.
The mapping included various type of mapping (as described above). The computer program product (300) then by way of its output generation module (114) retrieves results from the transformed categorized data with respect to the user's query.
For the purpose of understanding and by way of specific example, the computer program product refers to an E@TL (Extract Analytical Transform Loading) tool which is capable of processing the data in a unique manner i.e. by utilizing the processing power of a distributed data storage system in a very easy and efficient manner.

BEST MODE/EXAMPLE FOR WORKING OF THE INVENTION
The system and method illustrated to provide analytical processing of data in a distributed data storage system may be illustrated by working example stated in the following paragraph; the process is not restricted to the said example only:
Proposes system may be used for gaining affinity analysis of products from an e-commerce website.
Objective:
1. Analyzing Affinity of items over a long duration (6 - lOyrs) will provide key insights into running better promotions, planogram and price planning using affinity of items.
2. Impact of Competitor Store on Basket (Reduction in Basket)
3. Impact of-Competitor Store on Consumer Loyalty (Reduction, in trips per month) Solution provided by proposed system:
Steps involved while processing data process:
1. Extract data from any source system using the data extraction module and store it in distributed data storage system.
2. Transform the source data using the transformation module as per the business rules for finding affinity, and then the transformed data may be analyzed to identify the patterns and affinity among products by using frequent pattern matching algorithm.
3. The Competitor Impact may be derived by linking Segmentation & Competitor Data with Frequency of Store Visits, Basket Size & Total Spend of the Consumers
4. Store target as selected flattened structure for storing output.
5. The output can then be used to gain insights which have a lot of business value.
i

ADVANTAGES OF THE INVENTION
1. Data processing with the capacity of distributed and parallel processing of Bigdata, using/leveraging map reduce framework.
2. Constraint-free/schema free, source and destination independent mapping.
3. Multi-source to multi-destination mapping allowed (many to many mapping) in a single step.
4. Single-stage extract-transform capability.
5. Analytical Transformation: Process to perform elementary analysis over the extracted data in order to extract the useful data. Perform a secondary analysis after doing the transformation in order to provide a refined data which may be used for resolving a user's search query to the distributed data storage system.

WE CLAIM:
1. A system for analytical processing of data in a distributed data storage system, the
system comprising:
a data extraction module configured to perform one or more elementary analytical operations to extract data from one or more source database in one or more data format, said data is extracted regardless of the presence of constraints and structure of the source database;
a processing engine configured to perform one or more operations for data refinement in order to categorize the data, while the data is being extracted, the processing engine comprising:
a mapping module configured to perform one or more type of mapping operations over the categorized data by using one or more pre-defined
mapping rules;
a transformation module configured to perform a secondary analytical transforming operation based on one or more pre-defined business rules, over the mapped categorized data in order to obtain a transformed categorized data;
such that, only the transformed categorized data is stored in a target area present inside the distributed data storage system for one or more utility purpose.
2. The system as claimed in claim 9, wherein one or more utility purpose may include but is not limited to a market based analysis, a predictive data analysis or a combination thereof.
3. The system as claimed in claim 1, wherein the elementary analysis further comprises one or more algorithm based analysis.
4. The system as claimed in claim 1, wherein the source database further comprises of an oracle database, DB2 or alike.

5. The system as claimed in claim 1, wherein the system further comprises of a parsing module configured to parse the extracted data into a custom format.
6. The system as claimed in claim 1, wherein the system further comprises of a validation module configured to perform a repetitive data sorting operation in one or more stages in order to identify and categorize the extracted data as a valid data and an invalid data and store said categorized data in one or more respective
databases.
7. The system as claimed in claim 1, wherein the data further comprises of structured data, semi-structured data, unstructured data or a combination thereof.
8. The system as claimed in claim 7, wherein one or more data format further comprises of JSON, XML, CSV, or a combination thereof.
9. The system as claimed in claim 1, wherein the repetitive data sorting operation is performed over the data categorized as invalid data before storing it into the respective database.
10. The system as claimed in claim 1, wherein the one or more type of mapping operations further comprises of one-to-many, many-to-one, one-to-one and many-to-many source and destination mapping respectively in a single step.
11. A method for analytical processing of data in a distributed data storage system, the method comprising processor implemented steps of:
performing one or more elementary analytical operations to extract data from one or more source database in one or more data format, said data is extracted regardless of the presence of constraints on and structure of the source database;

processing said extracted data by performing one or more operations for data refinement in order to categorize the data, while the data is being extracted, the processing further comprising:
performing one or more type of mapping operations over the categorized data by using one or more pre-defined mapping rules; performing a secondary analytical transforming operation based on one or more pre-defined business rules, over the mapped categorized data in order to obtain a transformed categorized data;
such that, only the transformed categorized data is stored in a target area present inside the distributed data storage system for one or more utility purpose.
12. The method as claimed in claim 11, wherein one or more utility purpose includes
but is not limited to a market based analysis, a predictive data analysis or a
combination thereof.

13. The method as claimed in claim 11, wherein the elementary analysis further
i comprises of an algorithm based analysis.
14. The method as claimed in claim 11, wherein the source database further comprises of an oracle database, DB2 or alike.
15. The method as claimed in claim 11, wherein the method further comprises of step of parsing the extracted data into a custom format.
16. The method as claimed in claim 11, wherein the method further comprises of step of performing a repetitive data sorting operation in one or more stages in order to identify and categorize the extracted data as a valid data and an invalid data and store said categorized data in one or more respective databases.

17. The method as claimed in claim 11, wherein the data further comprises of structured data, unstructured data, semi-structured data or a combination thereof.
18. The method as claimed in claim 17, wherein the one or more data format further
comprises of JSON, XML, CSV, or a combination thereof.
i
19. The method as claimed in claim 11, wherein the repetitive data sorting operation is performed over the data categorized as invalid data before storing it into the respective database.
20. The method as claimed in claim 11, wherein the one or more type of mapping operations further comprises of one-to-many, many-to-one, one-to-one and many-to-many source and destination mapping respectively in a single step.
21. A computer program product for analytical processing of data in a distributed data storage system with respect to one or more query, the computer program product comprising:
a user interface configured to first authenticate and receive said query from one or
more user;
a data extraction module configured to perform an elementary analysis to extract

an output generation module configured to retrieve results from the transformed categorized data with respect to the user's query.