FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENT RULES, 2003
COMPLETE SPECIFICATION
(See Section 10 and Rule 13)
Title of invention:
SYSTEM AND METHOD FOR DATA MANAGEMENT IN AN OPEN SOURCE DISTRIBUTED COMPUTING PLATFORM
Applicant
TATA Consultancy Services Limited A Company Incorporated in India under The Companies Act. 1956
Having address:
Nirmal Building, 9th Floor,
Nariman Point, Mumbai 400021,
Maharashtra. India
The following specification particularly describes the invention and the manner in which it is to be performed.

TECHNICAL FIELD
[001] The present subject matter described herein, in general, relates to data
management systems, and more particularly to data management in an open source distributed computing platform.
BACKGROUND
[002] The existing document management tools use local file systems for storage of
the document data, wherein the document data may be in a structured or an unstructured format. As document content increases the document management tools use proprietary file systems for the storage of the document content. Management of the document content which is very large in size and stored in local file systems is time consuming and is not a scalable solution. Also, traditional document management tools are complex to implement because of large size of the document data and are difficult to integrate with main applications.
[003] Moreover, execution of business operations like upload, search, retrieve,
remove and de-duplication, on large size documents, using traditional methodologies is a high resource oriented and time-consuming task. Also, performing search through the document repository based on meta-data or other parameters requires a lot of time.
[004] Further, the existing data management tools do not provide for a mechanism for
analysis of the document content before uploading the document in repository. Also, when the document is uploaded without any analysis, there is possibility of errors in the file so uploaded. In general scenario, the analysis or fact-finding of the document content is done after the document is uploaded or stored in the repository. Today, there does not exist any method for scanning the document while uploading in order to perform analysis on the document content thereby leading to delay in raising alerts for unusual data patterns or errors in the document content.
SUMMARY
[005] This summary is provided to introduce aspects related to systems and methods
for data management in an open source distributed computing platform and the aspects are further described below in the detailed description. This summary is not intended to identify

essential features of the claimed subject matter nor is it intended for use in determining or limiting the scope of the claimed subject matter.
[006] In one implementation, a system for data management in an open source
distributed computing platform is disclosed. The system comprises a processor and a memory coupled to the processor for executing a plurality of modules present in the memory. The plurality of modules comprises an input module, a data uploading module, an extraction module, an analytical module and a processing module. The input module is constructed to configure one or more parameters to be used for performing one or more operations on one or more document, such that the parameters are further mapped with the document and are stored in a repository of the open source distributed computing platform. Further, a data uploading module is configured to select the document to be uploaded. The extraction module is configured to extract document content by performing a search in the document based on the parameters so configured before uploading the document. The analytical module is configured to analyze the document content so extracted by applying one or more logic rules before storing the document content in a distributed file system. Further, the processing module is configured to perform operations in a parallel mode on the document content stored in the distributed file system based on the parameters configured, without moving the document content to other locations.
[007] In one implementation, a method for data management in an open source
distributed computing platform is disclosed. The method comprises configuring one or more parameters to be used for performing one or more operations on one or more document, such that the parameters are further mapped with the document and are stored in a repository of the open source distributed computing platform. The method further comprises of selecting the document to be uploaded and extracting document content by performing a search in the document based on the parameters so configured before uploading the document. The method further comprises of analyzing the document content so extracted by applying one or more logic rules before storing the document content in a distributed file system and the operations are performed in a parallel mode on the document content stored in the distributed file system based on the parameters configured. The operations are performed without moving the document content to other locations. The steps of configuring, the selecting, the extracting, the analyzing and the performing operations in a parallel mode are performed by a processor.

[008] In one implementation, a computer program product having embodied thereon a
computer program for data management in an open source distributed computing platform is disclosed. The computer program product comprises of a program code for configuring one or more parameters to be used for performing one or more operations on one or more document, such that the parameters are further mapped with the document and are stored in a repository of the open source distributed computing platform. The computer program product further comprises of a program code for selecting the document to be uploaded and a program code for extracting document content by performing a search in the document based on the parameters so configured before uploading the document. The computer program product further comprises of a program code for analyzing the document content so extracted by applying one or more logic rules before storing the document content in a distributed file system and a program code for performing the operations in a parallel mode on the document content stored in the distributed file system based on the parameters configured, without moving the document content to other locations.
BRIEF DESCRIPTION OF THE DRAWINGS
[009] The detailed description is described with reference to the accompanying
figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The same numbers are used throughout the drawings to refer like features and components.
[0010] Figure 1 illustrates a network implementation of a system for data management
in an open source distributed computing platform is shown, in accordance with an embodiment of the present subject matter.
[0011] Figure 2 illustrates the system, in accordance with an embodiment of the present
subject matter.
[0012] Figure 3 illustrates architecture of the system, in accordance with an exemplary
embodiment of the present subject matter.
[0013] Figure 4 illustrates a method for uploading a document, in accordance with an
embodiment of the present subject matter.

[0014] Figure 5 illustrates a method for performing analysis on the document, in
accordance with an embodiment of the present subject matter.
[0015] Figure 6 illustrates a method for metadata indexing of a document, in
accordance with an embodiment of the present subject matter.
[0016] Figure 7 illustrates a method for data management in an open source distributed
computing platform, in accordance with an embodiment of the present subject matter.
DETAILED DESCRIPTION
[0017] Systems and methods for data management in an open source distributed
computing platform are described. Initially, one or more parameters are configured through the user interface in order to execute operations like upload; search, retrieve, de-duplication and remove on document data. The parameters configured are metadata, mapped with the document to perform search, retrieve operations on the document after storage in a distributed file system, content type of the document, search attributes, and rules.
[0018] Further, the document to be uploaded in the distributed file system is selected.
Document content is extracted by performing a search in the document based on the parameters so configured before uploading the document, wherein the parameters for extraction are the search attributes configured. Further, analysis is performed on the document content extracted, by applying one or more rules, while the document is stored in the distributed file system. Based on results of the analysis performed, alerts or notifications are raised in case of errors.
[0019] In the next step, various operations may be performed in a parallel mode on the
document content stored in the distributed file system, based on the parameters mapped with the document providing faster search and retrieval. The metadata mapped with the document is stored in a repository of the open source distributed computing platform.
[0020] While aspects of described system and method for data management in an open
source distributed computing platform may be implemented in any number of different computing systems, environments, and/or configurations, the embodiments are described in the context of the following exemplary system.

[0021] Referring now to Figure 1, a network implementation 100 of a system 102 for
data management in an open source distributed computing platform is illustrated, in accordance with an embodiment of the present subject matter. In one embodiment, the system 102 provides for configuring one or more parameters to be used for performing one or more operations on one or more document. The parameters are further mapped with the document and are stored in a repository of the open source distributed computing platform. Further, in order to perform analysis on document content, being uploaded, the document content is extracted from the document based on the parameters configured. Also, operations are performed in a parallel mode on the document content stored in a distributed file system based on the parameters configured, without moving the document content to other locations.
[0022] Although the present subject matter is explained considering that the system 102
is implemented on a server, it may be understood that the system 102 may also be implemented in a variety of computing systems, such as a laptop computer, a desktop computer, a notebook, a workstation, a mainframe computer, a server, a network server, and the like. It will be understood that the system 102 may be accessed by multiple users through one or more user devices 104-1, 104-2...104-N, collectively referred to as user 104 hereinafter, or applications residing on the user devices 104. Examples of the user devices 104 may include, but are not limited to, a portable computer, a personal digital assistant, a handheld device, and a workstation. The user devices 104 are communicatively coupled to the system 102 through a network 106.
[0023] In one implementation, the network 106 may be a wireless network, a wired
network or a combination thereof. The network 106 can be implemented as one of the different types of networks, such as intranet, local area network (LAN), wide area network (WAN), the internet, and the like. The network 106 may either be a dedicated network or a shared network. The shared network represents an association of the different types of networks that use a variety of protocols, for example, Hypertext Transfer Protocol (HTTP), Transmission Control Protocol/Internet Protocol (TCP/IP), Wireless Application Protocol (WAP), and the like, to communicate with one another. Further the network 106 may include a variety of network devices, including routers, bridges, servers, computing devices, storage devices, and the like.

[0024] Referring now to Figure 2, the system 102 is illustrated in accordance with an
embodiment of the present subject matter. In one embodiment, the system 102 may include at least one processor 202, an input/output (I/O) interface 204, and a memory 206. The at least one processor 202 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, centra] processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Among other capabilities, the at least one processor 202 is configured to fetch and execute computer-readable instructions stored in the memory 206.
[0025] The I/O interface 204 may include a variety of software and hardware
interfaces, for example, a web interface, a graphical user interface, and the like. The I/O interface 204 may allow the system 102 to interact with a user directly or through the client devices 104. Further, the I/O interface 204 may enable the system 102 to communicate with other computing devices, such as web servers and external data servers (not shown). The I/O interface 204 can facilitate multiple communications within a wide variety of networks and protocol types, including wired networks, for example, LAN, cable, etc., and wireless networks, such as WLAN, cellular, or satellite. The I/O interface 204 may include one or more ports for connecting a number of devices to one another or to another server.
[0026] The memory 206 may include any computer-readable medium known in the art
including, for example, volatile memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM), and/or non-volatile memory, such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes. The memory 206 may include modules 208 and data 210.
[0027] The modules 208 include routines, programs, objects, components, data
structures, etc., which perform particular tasks or implement particular abstract data types. In one implementation, the modules 208 may include an input module 212, a data uploading module 214, an extraction module 216, an analytical module 218, a processing module 220, a validation module 222, a categorization module 224 and other modules 226. The other modules 226 may include programs or coded instructions that supplement applications and functions of the system 102.

[0028] The data 210, amongst other things, serves as a repository for storing data
processed, received, and generated by one or more of the modules 208. The data 210 may also include a system database 228, and other data 230. The other data 230 may include data generated as a result of the execution of one or more modules in the other modules 226.
[0029] In one implementation, at first, a user may use the client device 104 to access
the system 102 via the I/O interface 204. The user may register them using the I/O interface 204 in order to use the system 102. The working of the system 102 may be explained in detail in Figures 3, 4, 5 and 6 explained below. The system 102 may be used for data management in an open source distributed computing platform. The system 102, further comprises of an input module 212 that is constructed to configure one or more parameters to be used for performing one or more operations on one or more document, such that the parameters are further mapped with the document and are stored in a repository of the open source distributed computing platform.
[0030] In one implementation, referring to figure 3, the parameters are configured
through the input module 212 by a client application (302(a) and 302 (b)). The input module 212 provides a user interface 304; the user interface 304 acts as an admin console through which the parameters may be configured. The client applications (302(a) and 302 (b)) may be built using heterogeneous technologies and the system 102 provides a technology independent platform to the client applications (302(a) and 302 (b)) to perform operations on the data. The one or more parameters further comprises of metadata, content type of the document, search attributes, rules or a combination thereof and the operations further comprises of uploading, search, retrieve, de-duplication and remove.
[0031] The system 102 further comprises of the data uploading module 214 that is
configured to select the document to be uploaded. Referring to figure 4, at step 402. the document to be uploaded is selected by the data uploading module 214. The documents may include rich text documents, text documents, image documents or video documents. In the next step, 404. the metadata configured is added to the document. Further, at step 406, a document upload service is called. Further, at step 408, the metadata is validated based on the rules configured by the validation module 222.

[0032] The validation module 222 is configured to validate the metadata based on the
rules so configured through the input module 212. The rules comprises of validation rules, wherein the validation rules includes standard types and business validations. The standard type validations include string, number and data format validation rules. The business validation includes email format and regular expression rules. The regular expression rules include PAN format and IP address check. Also, extension points for invoking business validations specific to the client application (302(a) and 302 (b)) requirement are provided. Referring to figure 3, the validation module 222 provides for a business layer 306. wherein validation of metadata is performed.
[0033] Still referring to figure 4, at step 410, if any errors are present in the metadata,
the errors are reported and the document upload service is terminated. Else if, no errors are present, at step 412, an integration service is called. Referring to figure 3, the integration service, at the integration layer 308, executes the operations on the document content. The operations are performed on the document content through the processing module 220. The processing module 220 is configured to perform operations in a parallel mode on the document content stored in the distributed file system based on the parameters configured, without moving the document content to other locations. The operations are performed based on the metadata mapped with the document.
[0034] Referring to figure 4, at the next step, 414. analysis is performed on the
document before the document is uploaded. In order to perform analysis on the document, the document content needs to be extracted. The document content is extracted through the extraction module 216. The extraction module 216 is configured to extract document content by performing a search in the document based on the parameters so configured before uploading the document. The document content is extracted based on the search attributes configured through the input module 212. Further, the analytical module 218 is configured to analyze the document content so extracted by applying one or more logic rules before storing the document content in a distributed file system. The logic rules further comprises of business rules and mathematical rules. By way of a specific example, rules may include extraction of keywords from the document content like error keywords extracted from log files, wherein the error keywords may be used for raising alerts and notifications. Business rules also include de-duplication logic applied on the document content. Mathematical rules may include rules

applied for extraction of customer information from mobile bills if usage crosses threshold limit.
[0035J Figure 5 illustrates the steps implemented by the extraction module 216 and the
analytical module 218 in order to perform analysis on the document content. The analysis is further a pre-emptive analysis. In the first step, 502, document content stream is read. Further, at step 504, a chunk of the document content stream is picked up based upon separator. In the next step, 506, required information is extracted for configured keywords. By way of a specific example, for documents pertaining to mobile bills, customer information for high usage of bill amount may be extracted. The customer information extracted may be further stored and used for performing analysis for purpose of marketing. Further at step 508, the information extracted is stored and values are calculated based on rules specific for the client application (302(a) and 302 (b)). Further, at step 510, DMS level reporting attribute is updated. Further, at step, 512, analysis is performed on the extracted information by applying business rules. In case of any errors or presence of any unusual data patterns, alerts or notifications are raised.
[0036] Still referring to figure 4, at step 416, metadata indexing is performed while
storing the metadata in the repository. Figure 6 illustrates the steps implemented in order to perform metadata indexing. At step 602. values of metadata attributes are concatenated for all permutations and combinations. By way of a specific example, if A, B and C are three metadata attributes, values may be A, B. C, AB, AC, BC and ABC. Further, in the next step 604, 16 bytes hash value is calculated for each combination. At step, 606, all the hash values mapped with the document id are stored. In the next step, 608. hash value keys are indexed and stored in RAM In the next step, 610. while performing the operation of search, indexed hash values are searched from RAM.
[0037] By way of a specific example, during the operation of search performed on the
document content, values are concatenated and converted into hash values. The values are fetched from RAM which is faster compared to any standard Input/output operation.
[0038] Further, the system 102 comprises of the categorization module 224 that is
configured to categorize the document based on the content type of the document before storing the document in the distributed file system. The system 102 supports storage of multiple content types of documents as actual document content is stored in binary format.

[0039] Further, the open source distributed computing platform includes Hadoop, the
repository further includes Hbase and the distributed file system further includes Hadoop distributed file system (HDFS). Also, in order to perform operations in a parallel mode on the document content, Mapreduce framework may be implemented.
ADVANTAGES
[0040] The system 102 is implemented as a light weight service oriented component
leveraging Hadoop technologies in order to build a large document repository. The distributed storage system and performance of operations in the parallel mode offers reduced cost and faster and easier processing on the document content.
[0041] Analysis performed using the analytical module 218, provides for analysis,
wherein the document may be scanned and analysis is performed on the document content before the upload operation is executed on the document content. Also, necessary alerts may be raised for fraudulent or error information which helps in taking early corrective action. Analysis of the document content before the document is uploaded helps in identifying new business attributes which can be used for more precise and real time information for making strategic decisions. The preemptive analysis (or simply analysis) of the document content before the document is uploaded brings in process improvement by early detection of data patterns.
[0042] As the system 102 is implemented as a light weight service oriented component,
business services are exposed which allow easy integration with the client applications or other products.
[0043] Efficient design for storage of metadata information mapped with the
documents helps in faster search and retrieval from large document repository.
[0044] HDFS eco system component of Big Data provides fault tolerance distributed
file system which facilitates scalable solution for large document repository spread across cluster of servers. The operations may be performed on the document content stored in the HDFS in a parallel mode without moving the document content to any other storage system.

[0045] Underlying hardware requirements for HDFS component for document
repository may be made of commodity hardware which is a cost effective solution compared to traditional data management systems.
[0046] Referring now to Figure 7, a method 700 for data management in an open
source distributed computing platform is shown, in accordance with an embodiment of the present subject matter. The method 700 may be described in the general context of computer executable instructions. Generally, computer executable instructions can include routines, programs, objects, components, data structures, procedures, modules, functions, etc., that perform particular functions or implement particular abstract data types. The method 700 may also be practiced in a distributed computing environment where functions are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, computer executable instructions may be located in both local and remote computer storage media, including memory storage devices.
[0047] The order in which the method 700 is described is not intended to be construed
as a limitation, and any number of the described method blocks can be combined in any order to implement the method 700 or alternate methods. Additionally, individual blocks may be deleted from the method 700 without departing from the spirit and scope of the subject matter described herein. Furthermore, the method can be implemented in any suitable hardware, software, firmware, or combination thereof. However, for ease of explanation, in the embodiments described below, the method 700 may be considered to be implemented in the above described media system 102.
[0048] At block 702, a method for configuring one or more parameters to be used for
performing one or more operations on one or more document is described. The parameters are further mapped with the document and are stored in a repository of the open source distributed computing platform. In one implementation, one or more parameters to be used for performing one or more operations on one or more document are configured by the input module 212,
[0049] At block 704, a method for selecting the document to be uploaded is described.
In one implementation, the document is selected by the data uploading module 214.
[0050] At block 706, a method for extracting document content by performing a search
in the document based on the parameters so configured before uploading the document, is

described. In one implementation, the document content is extracted by the extraction module 216.
[0051] At block 708. a method for analyzing the document content so extracted by
applying one or more logic rules before storing the document content in a distributed file system, is described. In one implementation, the document content is analyzed by the analyzing module 218.
[0052] At block 710, a method for performing operations in a parallel mode on the
document content stored in the distributed file system based on the parameters configured, without moving the document content to other locations, is described. In one implementation, the operations are performed by the processing module 220.
[0053] Although implementations for methods and systems for data management in an
open source distributed computing platform have been described in language specific to structural features and/or methods, it is to be understood that the appended claims are not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as examples of implementations for data management in an open source distributed computing platform.

WE CLAIM:
I. A method for data management in an open source distributed computing platform, the method comprising:
configuring one or more parameters to be used for performing one or more operations on one or more document, such that the parameters are further mapped with the document and are stored in a repository of the open source distributed computing platform;
selecting the document to be uploaded;
extracting a document content by performing a search in the document based on the parameters so configured before uploading the document;
analyzing the document content so extracted by applying one or more logic rules before storing the document content in a distributed file system; and
performing the operations in a parallel mode on the document content stored in the distributed file system based on the parameters configured, without moving the document content to other locations, wherein the configuring, the selecting, the extracting, the analyzing and the performing operations in a parallel mode are performed by a processor.
2. The method of claim 1, wherein one or more parameters further comprises of metadata, content type of the document, search attributes, rules or a combination thereof.
3. The method of claim 1, wherein extracting the document content is based on the search attributes configured.
4. The method of claim 1. wherein the open source distributed computing platform further includes Hadoop, the repository further includes Hbase and the distributed file system further includes Hadoop distributed file system (HDFS).
5. The method of claim 1, wherein the method further comprises validating the metadata based on the rules so configured.
6. The method of claim 1. wherein the method further comprises categorizing the document based on the content type of the document before storing the document in the distributed file system.
7. The method of claim 1, wherein the operations further comprises of uploading, search, retrieve and remove.

8. The method of claim 1. wherein the operations are performed based on the metadata mapped with the document.
9. The method of claim 1, wherein the logic rules further comprises of business rules and mathematical rules.
10. A system for data management in an open source distributed computing platform, the system comprising:
a processor; and
a memory coupled to the processor, wherein the processor is capable of executing a plurality of modules stored in the memory, and wherein the plurality of module comprising:
an input module constructed to configure one or more parameters to be used for performing one or more operations on one or more document, such that the parameters are further mapped with the document and are stored in a repository of the open source distributed computing platform;
a data uploading module configured to select the document to be uploaded;
an extraction module configured to extract document content by performing a search in the document based on the parameters so configured before uploading the document;
an analytical module configured to analyze the document content so extracted by applying one or more logic rules before storing the document content in a distributed file system; and
a processing module configured to perform operations in a parallel mode on the document content stored in the distributed file system based on the parameters configured, without moving the document content to other locations.
11. The system of claim 10, wherein one or more parameters further comprises of metadata, content type of the document, search attributes, rules or a combination thereof.
12. The system of claim 10. wherein the document content is extracted based on the search attributes configured.
13. The system of claim 10, wherein the open source distributed computing platform further includes Hadoop. the repository further includes Hbase and the distributed file system further includes Hadoop distributed file system (HDFS).

14. The system of claim 10, wherein the system further comprises of a validation module configured to validate the metadata based on the rules so configured.
15. The system of claim 10, wherein the system further comprises of a categorization module configured to categorize the document based on the content type of the document before storing the document in the distributed file system.
16. The systems of claim 10, wherein the operations further comprises of uploading, search, retrieve and remove.
17. The system of claim 10, wherein the operations are performed based on the metadata mapped with the document.
18. The system of claim 10, wherein the logic rules further comprises of business rules and mathematical rules.
19. A computer program product having embodied thereon a computer program for data management in an open source distributed computing platform, the computer program product comprising:
a program code for configuring one or more parameters to be used for performing one or more operations on one or more document, such that the parameters are further mapped with the document and are stored in a repository of the open source distributed computing platform;
a program code for selecting the document to be uploaded;
a program code for extracting a document content by performing a search in the document based on the parameters so configured before uploading the document;
a program code for analyzing the document content so extracted by applying one or more logic rules before storing the document content in a distributed file system; and
a program code for performing the operations in a parallel mode on the document content stored in the distributed file system based on the parameters configured, without moving the document content to other locations.