FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENT RULES, 2003
COMPLETE SPECIFICATION
(See Section 10 and Rule 13)
Title of invention:
A COLLABORATIVE FRAMEWORK FOR VISUALIZATION OF DISTRIBUTED STORAGE SYSTEM
Applicant
TATA Consultancy Services Limited A Company Incorporated in India under the Companies Act, 1956
Having address:
Nirmal Building, 9th Floor,
Nariman Point, Mumbai 400021,
Maharashtra, India
The following specification particularly describes the invention and the manner in which it is to be performed.

FIELD OF THE INVENTION
The present subject matter described herein relates to a method and framework for managing big datasets, and more particularly, relates to the method and the collaborative framework for visualization of a Big Data store.
BACKGROUND OF THE INVENTION
At present, immense amount of data get generated, processed, stored, distributed, analyzed everyday, which may comprise huge volume and variety. Hbase is an open-source, distributed database which hosts big data tables of billions of rows and millions of columns on top of cluster of machines. Hbase is a NoSQL database which stores data in a column-oriented format. It is different from the traditional relational database management systems (RDBMS) in a way that it does not support SQL as its access language.
RDBMS can scale well but only up to a certain limit. When the size of data increases the scalability reduces. This problem can be addressed by storing data (large) in Hbase which provides linear as well as modular scaling. Thus, Hbase provides both low response time and high throughput.
Although Hbase provides a data store, it lacks certain features with respect to data . management, data processing and user management. Hbase organizes data in sets containing key value pairs instead of relations. Consequently, it does not provide an effective way of querying data and is unable to handle complex queries.
Secondly, business users and administrators hailing from the background of traditional RDBMS find it difficult to work with the Hbase because of its non-relational or column-oriented format. Thus it lacks an interface for visualization of data.
Hence, there is a need for a framework which is capable of providing an effective data management and user management features. More particularly, there is a vital need of an

interface which may help the business and enterprise users in visualizing data stored in the Big Data storage systems.
OBJECTS OF THE INVENTION
It is the primary object of the invention to provide an integrated framework to facilitate a platform independent accessibility of data in a Big Data storage system.
It is another object of the invention to create an improved visualization of the Big Data storage system.
It is yet another object of the invention to transform a format of fetched data in real-time and storing the transformed data into one or more target data storage system.
It is yet another embodiment of the present invention to provide one or more components (query engine, CRUD engine etc) in order to optimize the Big Data storage system.
SUMMARY OF THE INVENTION
This summary is provided to introduce concepts related to a framework and method of facilitating a platform independent accessibility of data in Big Data storage system. This summary is not intended to identify essential features of the claimed subject matter nor is it intended for use in determining or limiting the scope of the claimed subject matter.
One of the preferred embodiment of the subject matter comprises a method comprising steps of authorizing at least one user by authenticating said user details for accessing the Big Data storage system in a network and fetching data in one or more manner desired by the user from one or more source data storage system to further populate one or more target data storage system. The method further comprises steps of processing the fetched data to provide an improved visualization of the Big Data storage system by performing one or more operations over data thus fetched. The processing further comprises transforming a format of fetched data in real-time and storing the transformed data in the target data storage system. The method further comprises steps of analyzing and

monitoring data thus transformed and stored in the target data storage system for its easy-retrieval.
The summary further provides an integrated framework to facilitate platform independent accessibility of data in a Big Data storage system by optimizing said Big Data storage system. The framework comprises a user interface configured to authorize at least one user by authenticating said user details for accessing the Big Data storage system in a network and a polling engine adapted to fetch data in one or more manner desired by the user from one or more source data storage system to further populate one or more target data storage system. The framework further comprises a processor configured to process data thus fetched and performs one or more operations over data for creating an improved visualization of the Big Data storage system. The processor further comprises a transformation module configured to transform a format of fetched data in real-time and storing the transformed data into one or more target data storage system. The framework further comprises an analytical module configured to analyze and monitor data thus transformed and stored in the target data storage system for its easy retrieval.
BRIEF DESCRIPTION OF THE DRWAINGS
Further objects, embodiments, features and advantages of the present invention will become more apparent and may be better understood when read together with the detailed description and the accompanied drawings. The components of the figures are not necessarily to scales, emphasis instead being placed on better illustration of the underlying principle of the subject matter. Different numeral references on figures designate corresponding elements throughout different views. However, the manner in which the above depicted features, aspects, and advantages of the present subject matter are accomplished, does not limit the scope of the subject matter, for the subject matter may admit to other equally effective embodiments.
Figure 1 illustrates the framework architecture in accordance with an embodiment of the invention.

Figure 1(a) further illustrates the collaborative approach of the integrated framework for fetching and processing data.
Figure 2 illustrates the process of the authorization and the authentication implemented by the user interface.
Figure 3 illustrates a flowchart in accordance with an alternate embodiment of the invention.
Figure 4 illustrates a flowchart in accordance with an alternate embodiment of the invention.
Figure 5 illustrates an exemplary flowchart in accordance with an alternate embodiment of the invention.
DETAILED DESCRIPTION
Some embodiments of this invention, illustrating its features, will now be discussed:
The words "comprising", "having", "containing", and "including", and other forms thereof, are intended to be equivalent in meaning and be open ended in that an item or items following any one of these words is not meant to be an exhaustive listing of such item or items, or meant to be limited to only the listed item or items.
It must also be noted that as used herein and in the appended claims, the singular forms "a", "an", and "the" include plural references unless the context clearly dictates otherwise. Although any systems, methods, apparatuses, and devices similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present invention, the preferred, systems and parts are now described. In the following description for the purpose of explanation and understanding reference has been made to numerous embodiments for which the intent is not to limit the scope of the invention.

One or more components of the invention are described as module for the understanding of the specification. For example, a module may include self-contained component in a hardware circuit comprising of logical gate, semiconductor device, integrated circuits or any other discrete component. The module may also be a part of any software programme executed by any hardware entity for example processor. The implementation of module as a software programme may include a set of logical instructions to be executed by the processor or any other hardware entity. Further a module may be incorporated with the set of instructions or a programme by means of an interface.
The disclosed embodiments are merely exemplary of the invention, which may be embodied in various forms.
The present invention relates to a framework and a method for facilitating a platform independent accessibility of data in a Big Data storage system. In the very first step, the user is authorized for an access to the Big Data storage system by authenticating the user details. Further, data is fetched from a source system which is populated in one or more target data storage system. The fetched data is further processed by transforming the format of data, to improve the visualization of the Big Data storage system. Further, the transformed data is monitored and analyzed for an easy retrieval.
In accordance with an embodiment, referring to figure 1, the framework (100) comprises a user interface (102) which is configured to authorize at least one user by authenticating said user's details and a polling engine (104) which is adapted to fetch data. The framework (100) further comprises a processor (106) which is configured to process the fetched data. The processor (106) further comprises a transformation module (108) configured to transform the format of the fetched data. The framework (100) further comprises an analytical module (110) which is configured to analyze and monitor the transformed data.
In accordance with an embodiment, still referring to figure 1, the user interface (102) is configured to authorize at least one user by authenticating said user details for accessing the Big Data storage system in a network. The user details may include but is not limited

to a user name, a password and an ip address. By way of specific example, let the available users may include: Admin Users, Full Access Users and Limited Users.
Admin users have the permission for creating, editing and deleting any user, ip address tables and cluster properties. The admin users cannot be deleted but their passwords can be edited.
Full Access users can create, delete and update any tables in the Hbase cluster. However, they cannot add or remove users and IP addresses.
Limited Access Users can only view and process data lying in Hbase and cannot manipulate them.
Referring to figure 2, the authentication of the user details is user name and password based as well as ip address based. The user is authorised access to the Big Data storage system only if the user name and password is valid and the ip address from which the user is accessing data storage system is in the allowed ip address list.
The framework (100) further comprises a polling engine (104) which is adapted to fetch data in one or more manner desired by the user from one or more source data storage system (112) to further populate one or more target data storage system (114).
Data may include but is not limited to a structural, a semi-structural and a non-structural data. The source data storage system may include but is not limited to an RDBMS (Relational Database Management System). The Big Data storage system and the target data storage system may include but is not limited to the Hbase and NoSQL databases like Cassandra, Hypertable, MongoDB.
For the purpose of understanding, let the target data storage system is Hbase:
The manner of fetching data may include but is not limited to fetching data in real-time and in batches. The fetching of data from the source data systems (112) may occur by configuring the parameters which may include but are not limited to:

• Time interval between two listener threads
• Number of source systems/target systems
• Source/Target system's connection string
• Mapping between datastore's and source/target system's columns
• Condition for polling
• Job type (real time or batch polling)
When the user specifies the poll as real time, data is fetched or pushed linearly from the specific source system (112). In the case of batch polling, a mapreduce job is executed that fetches or pushes data parallel from the source systems (112). By way of a specific example, in most of the cases of real time polling, the time difference between two listener threads is very less as compared to batch polling if data is fetched or pushed in bulk. In the case of conditional polling, the incoming or outgoing records have to satisfy the conditional criteria for being eligible for fetch or push. The conditions may be specified using SQL like statements native to the platform by the user. When the polling is initiated for the first time for the data store, the polling may be in the batch mode as bulk data is fetched or pushed in the first poll. Once the first poll is completed, the last record and the timestamp are updated for the runtime parameters. Subsequently, the poll will be from the timestamp updated in the runtime parameters till the current timestamp.
Figure 3 describes the steps implemented by the polling engine (104). In the first step (302), the polling engine (104) is initiated and it listens to the specified source data storage system (112) for fetching data. In the second step (304), the polling engine (104) checks if the user has specified any polling condition. On the satisfaction of the condition (as shown in step 306), if there is any new data found, the polling engine (104) ensures that the listening has stopped (as shown in step 308). After the polling is completed, the 'last record found' and the 'time stamp' are updated (as shown in step 310). The fetched data is stored on one or more target storage system (114).
Referring to figure 1(a), the framework (100) comprises a collaboration module (116) to fetch data from plurality of source storage system (112) and stores it into plurality of

target storage system (114) in a collaborative manner. This collaborative module (116) further supports the deployment of the framework in one or more manner.
In accordance with an embodiment, still referring to figure 1, the framework (100) further comprises a processor (106) which is configured to process data thus fetched and performs one or more operations over data for creating an improved visualization of the Big Data storage system. The processor (106) further comprises a transformation module (108) which is configured to transform a format of fetched data in real-time and storing the transformed data into one or more target data storage system. For example, the transformation module (108) may transform a column oriented data into a row oriented data.
Figure 4 describes the steps implemented by the transformation module (108). In the first step (402), the transformation module (108) checks for the mode selected by the user for the transformation. The transformation modes may include but is not limited to MapReduce mode and a general mode. In the MapReduce mode, as an input to the mapper, (row key, all the values) are fetched as (key, value) (as shown in step 404). Further, in the mapper, the column is extracted, updated and the column list is sorted (as shown in step 406). (Key, value) is emitted as (rowkey, all columns and values) to the reducer (as shown in step 408). In the next step (410), the reducer refers to the column list and generates the row with respective values and column positions.
Still referring to figure 4, in the general mode, initially all the columns (qualifiers) of the row are fetched (as shown in step 412). In the next step (414), if the column name exists then the column value is added for the respective map key and column (as shown in step 414). If the column name doesn't exist then the column name is added to the column list and the list is sorted and the position of the column is obtained (as shown in step 416). Further, all the previous data is moved by one column and a null value is added in each column for that given index. The value for the given key and column is added (as shown in step 418). In the next step (420), the transformation module (108) checks if next key

exists. If next key exists then the steps from 412 to 420 are repeated. If next key exist then all the respective values and row is rendered (as shown in step 424).
The transformation module (108) further comprises a paging module (not shown in the figure) which fetches data from the Big Data storage system in pages. These pages are asynchronously pulled from the Big Data storage system. This helps in visualizing data in such a way that there is minimum load on the client. The pages comprises various keys and their respective values. The size of the pages to be fetched can be configured by the end user based on the load his client can take.
The processor (106) further comprises a mapping module (not shown in figure) to map data fetched in batches to the target data storage system.
The processor (106) further comprises CRUD engine, a cluster management module and a predefined Map reduce library (not shown in figure) for supporting the optimization of the Big Data storage system.
By way of specific example, the integrated framework (100) is designed to provide Create-Retrieve-Update-Delete operations through the CRUD engine. The framework (100) also supports cluster management by means of a cluster management module (not shown in figure). The cluster management module (118) also manages cluster parameters, balancing the hbase cluster, explicit control on Write Ahead Log, memstore etc.
Still referring to figure 1, the framework (100) further comprises an analytical module (110) which is configured to analyze and monitor data thus transformed and stored in the target data storage system for its easy retrieval. The analytical module (110) further comprises a query engine in order to implement one or more query in real-time with respect to data thus transformed. The analytical module (110) further comprises a reporting module (not shown in figure) configured to generate one or more report template with respect to the query thus implemented.

The framework (100) further comprises a deployment module (not shown in figure) in order to support a hosted or a remote deployment of the integrated framework with one more data storage system. In hosted environment, the service for the workbench is started on the hbase cluster. The user interface (102) implementing authorization clears or rejects the user. In remote deployment, the user may connect to hbase using the REST client software. For this purpose the rest service is availed from the hbase side.
WORKING EXAMPLE OF THE PRESENT SUBJECT MATTER
The framework and method illustrated to facilitate a platform independent accessibility of data in Big Data storage system by optimizing said data storage system may be illustrated by working example stated in the following paragraph. A person skilled in the art would understand and appreciate that such an example is to merely illustrate various steps and features comprising the present subject matter and in no way limit the scope of the same from its realization in the broadest manner.
Let us consider the process of real-time weather reporting to understand how the wind speed affects the temperature on a real time basis.
Following parameters are captured for as weather data by weather station:
• Temperature
• Wind speed in knots
• Wind direction
• Humidity
• Atmospheric pressure
• Rainfall
• City
• Date and time
The above data is fetched from the source datastore using the polling engine (104). The polling engine (104) keeps fetching all the new incoming data. Finally the temperature v/s wind speed graph is plotted.

Column name Data type
TEMPARTURE INTEGER
WINDSPEED INTEGER
WINDJ3IRECT10N STRING
HUMIDITY INTEGER
ATMOSPHERICPRESSURE FLOAT
RAINFALL INTEGER
DATETIME DATE
CITY STRING
Table 1: Table structure in source weather datastore
Table 1 describes the table structure in the source weather datastore which may include but is not limited to a traditional RDBMS.

Column name Column family
Temp Cf
W Cf
Wd Cf
Humd Cf
Atmpres Cf
R Cf
Dat Cf

City
Cf
Table 2: Table structure in hbase
Table 2 describes the table structure of the target datastore which may include but is not limited to Hbase and NoSQL databases like Cassandra, Hypertable, MongoDB.
Key for the table is CITY with DATE_TIME appended without any delimiter.
Figure 5 describes the steps implemented by the reporting module to plot the temperature v/s wind speed graph. In the first step (502), the user logs into the framework. The user interface (102) authorizes the user by authenticating the user details (as shown in step 504). In the next step (506), the user selects the weather table in hbase. The next step (508) directs the user to the reporting module or the report designer. Further, the user specifies the qualifier (column) 'temp' and 'w' of the column family cf and selects the associated data type (as shown in step 510). The user chooses the type of the chart and assigns axes to 'temp' and 'w' In the next step (512), the report is generated.
Example 2
Let us also consider an example describing the functioning of the transformation module (108).

Key Column (qualifier) Value
1 employee_number 500
1 employee_name Debarshi
1 employeeaddress Mumbai
1 employeeemployer JDDB

2 empl oyee _n umber 501
2 employee name Shampa
2 employee address Mumbai
2 employeedepartment CEG
2 parentdepartment JDDB internal
2 department_number 1
Table 3
Table 3 describes the key, the column names and the corresponding values. The format of the above table structure is a column-oriented format. The transformation module (108) transforms the column-oriented format to a row-oriented format.
The transformation algorithm comprises two major components as described herein.
It comprises a column list, where newly found columns (qualifiers) are added when discovered by the algorithm during the iterations. At the end of the all iterations, the schema of the output record can be found in column list in sorted order. The Column list is updated and referred during the process of transformation.
Further, it comprises the mutation of row, whereby the out of transformation algorithm is a row-oriented record. In our algorithm, row is materialized intermediately. Thus, the output record keeps mutating till the last iteration when the required output is generated.
Consider the following example

{
{
"employee_number": "500, "employeename" : "Debarshi", "employeeaddress": "Mumbai", "employeeemployer" : "JDDB",
} ,
"2":
{
"empIoyee_number" : "501", "employeejiame" : "Shampa" , "employeeaddress" : "Mumbai", "employee department" : "CEG", "parent_department" : "JDDB internal" , "departmentnumber" : "1" } }
Here, the Parent value describes the key in the datastore whereas the inner notations represent column (qualifier) and the respective value (as illustrated in Table 3).
In the first iteration, all the columns for the given first key are extracted and are added to the Column List. It is to mention at this juncture that before the columns are added as the first iteration of the algorithm, the Column list is empty. When the iteration is initiated, the Column list is populated for the first key, which is '1' for the given example. Note that the Columns added to the column list are lexically (or based on the ASCII value) sorted, as depicted in Table 4.

Column list
emp!oyee_a ddress

employee. _employer
employee- name
employee number
Table 4
Once the Column list is populated, the corresponding row gets generated, as illustrated in Table 5.

Row employee_address employee_employer employee_name employeenumber
Key
1 Mumbai JDDB Debarshi 500
Table 5
In the second iteration, the transformation algorithm discovers new columns-employeedepartment, parentdepartment, departmentnumber and updates them in the list sorted, which is represented in Table 6.

Column list
department_number
employee_address
employee_department
employee_employer

employee-
name
employee- number
parent_department
Table 6
Also in this iteration, the rows start to mutate. As there are few new arriving columns, the mutation of rows looks like the following (Table 7):

Row department employee_ employee_ employee employe
Key _number address employer _name e_ number
1 Mumbai JDDB Debarshi 500
2 1
Table 7
Here while forming the second row, since the departmentnumber does not exist for key 1, an empty cell is inserted whereas a value is inserted in for key 2. Also, the department_number is added first because, it is at the top the Column list.
Similar mutation happens within this iteration, a depicted in Table 8 and Table 9.

Row Key department_ number employee_ address employee employer employee name employee number
I Mumbai JDDB Debarshi 500

2 ] Mumbai
Table 8

Ro departm emp!oyee_ employee_ employee employee employee_
w
Ke ent_ address departmen - - number
y number t employer name
j Mumbai JDDB Debarshi 500
2 1 Mumbai CEG
Table 9

Ro departm employe employee employee employee employe parent_d
w ent_ e_ e numb epartme
Ke
y number address departmen
t employer name er nt
1 Mumbai JDDB Debarshi 500
2 1 Mumbai CEG Shampa 501 JDDB internal
Table 10
Table 10 describes the final transformed structure of the column-oriented data into row-oriented data.
The present subject matter, therefore, provides a framework and method of facilitating a platform independent accessibility of data in Big Data storage system. It is to mention at

this juncture that although the present subject matter has been described in detail; those skilled in the art should understand that they can make various changes, substitutions and alteration herein, without departing from the crux of the subject matter in its broadest form.

WE CLAIM:
1. An integrated framework to facilitate platform independent accessibility of data
in a Big Data storage system, the framework comprising:
a user interface configured to authorize at least one user by authenticating user details for accessing the Big Data storage system in a network;
a polling engine adapted to fetch data in one or more manner desired by the user from one or more source data storage system to further populate one or more target data storage system;
a processor configured to process data thus fetched and perform one or more operations over data for creating an improved visualization of the Big Data storage system; the processor further comprising;
a transformation module configured to transform a format of fetched data in real time and storing the transformed data into one or more target data storage system; and
an analytical module configured to analyze and monitor data thus transformed and stored in the target data storage system for its easy retrieval.
2. The integrated framework as claimed in claim 1, wherein the data may include but is not limited to a structural, a semi-structural and a non-structural data.
3. The integrated framework as claimed in claim 1, wherein the framework further comprises a deployment module in order to support a hosted or a remote deployment of the integrated framework with one more data storage system.
4. The integrated framework as claimed in claim 1, wherein the Big Data storage system and the target data storage system may include but is not limited to Hbase and NoSQL databases like Cassandra, Hypertable, MongoDB.

5. The integrated framework as claimed in claim 1, wherein the framework further comprises a collaboration module to collaborate a plurality of users and/or storage systems over the network.
6. The integrated framework as claimed in claim 1, wherein the user details may include but is not limited to a user name, a password and an ip address.
7. The integrated framework as claimed in claim 1, wherein the manner of fetching data may include but is not limited to fetching data in real-time and in batches.
8. The integrated framework as claimed in claim 1, wherein the processor further comprises a mapping module to map data fetched in batches to the target data storage system.
9. The integrated framework as claimed in claim 1, wherein the transformation module further transforms a column oriented data into a row oriented data.
10. The integrated framework as claimed in claim 1, wherein the analytical module further comprises a query engine in order to implement one or more query in real-time with respect to data thus transformed.
11. The integrated framework as claimed in claim 1, wherein the analytical module further comprises a reporting module configured to generate one or more report template with respect to the query thus implemented.
12. The integrated framework as claimed in claim 1, wherein the processor further comprises CRUD engine, a cluster management module and a predefined Map reduce library for supporting the optimization of the Big Data storage system.

13. The integrated framework as claimed in claim 1, wherein the source data storage system may include but is not limited to an RDBMS (Relational Database Management System).
14. The integrated framework as claimed in claim 1, wherein the transformation module further comprises a paging module which fetches data from the Big Data storage system in pages.
15. A method to facilitate a platform independent accessibility of data in Big Data storage system, the method comprising steps of:
authorizing at least one user by authenticating user details for accessing the Big Data storage system in a network;
fetching data in one or more manner desired by the user from one or more source data storage system to further populate one or more target data storage system; processing the fetched data to provide an improved visualization of the Big Data storage system by performing one or more operations over data thus fetched; the processing further comprising steps of:
transforming a format of fetched data in real-time and storing the transformed data in the target data storage system; and
analyzing and monitoring data thus transformed and stored in the target data storage system for its easy-retrieval.
16. The method as claimed in claim 15, wherein the data may include but is not limited to a structural, a semi-structural and a non-structural data.
17. The method as claimed in claim 15, the method further comprises collaborating a plurality of users and/or storage systems over the network.

18. The method as claimed in claim 15, wherein the user details may include but is not limited to user name, password and ip address.
19. The method as claimed in claim 15, wherein the manner of fetching data may include but is not limited to fetching data in real-time and in batches.
20. The method as claimed in claim 15, wherein the processing further comprises mapping data fetched in batches to the target data storage system.
21. The method as claimed in claim 15, wherein data is transformed from a column oriented data into a row- oriented data.
22. The method as claimed in claim 15, wherein the analysis of the transformed data further comprises executing one or more query in real-time with respect to data thus transformed.
23. The method as claimed in claim 15, wherein the analysis of the transformed data further comprises generating one or more template report with respect to the query thus executed.