FIELD OF THE INVENTION
The present invention relates to a system and method that is inclined towards meaningful integration and retrieval of selective information from several disparate and standalone sources.
BACKGROUND OF THE INVENTION
Retrieval of information from vast amount of data sources (such as structured and unstructured datasets) primarily involves a searching step, where a user provides a query in the system. The query is considered as a collection of terms or words representing themes or topics, where each word is matched with the metadata available for the file.
Thus the retrieved datasets are matched based on the existence of some keyword from the query used in the search to the words found in the metadata information for the file. Metadata deals with information about the data in the form of title of the file, description of the file, column names or column descriptions for the column headings in the file, source of the file, publisher information etc. Searching on files containing structured or tabular data is generally performed using the metadata available for the file.
In current times, for matching the words present in the query, a direct matching system or method are widely used but has its limitations and does not match datasets where the terms used in the query do not match the terms present in the metadata for the file. Additionally, related terms do not match the words used to describe the various metadata for the file.
For instance, if the objective is to retrieve the selective information on "Air pollution data for the state of Karnataka" from the entire corpus of structured datasets, a dataset search might not be able to present the respective desired selection namely - the rows from the various air pollution datasets containing data for Karnataka, if the search is restricted to matching of terms from the query.

[0006] Multiple scenarios that can be encountered in this instance are: search retrieves files containing air pollution data for all the states of India since “Karnataka’ is not mentioned in any of the metadata information, search discards files satisfying the query since the metadata did not mention the terms air pollution’ or ‘Karnataka’.
[0007] Apart from the problem of direct matching of terms using metadata information, another problem is that terms do not represent the concepts. Air Pollution as a concept can be inferred from other related words such as particulate matter, suspended particles etc. that relate to the concept of “Air Pollution”. Finally, metadata information captures information relating to the file at the overall file level instead of the column or the row level. Hence dataset search purely based on matching query terms with metadata information has limitations when expecting selective retrieval of information based on a subset of columns or rows from the tabular data.
[0008] Certain systems or methods attempt to address this limitation using a taxonomy or hierarchy of terms such that information regarding related terms can be stored and maintained so as to make the search much more intensive and expressive.
[0009] But generating taxonomy approach is costlier as it requires a lot of manual operations and also the taxonomy approach results in insufficient accuracy in terms of resultant search, has limited expressiveness of representations of queries of users and a high cost is associated with the manual construction of linguistic resources and have limited adjustability. Whereas in the hierarchy of terms method there exists a chance of missed hit for keywords, also, some highly-complex queries require constant refinement in the hierarchy of terms method which makes this method more complicated in nature.

[0010] Many systems and methods are developed to retrieve the essential information from a vast amount of data such as the amazon data exchange system that is intended for the purpose of accessing and downloading data from any source but any kind of data integration is not possible in that system for improvising the search results.
[0011] Accordingly, there is a need to build knowledge repositories (also termed as knowledge bases or vocabularies) that capture essence from terms or a collection of terms by including their meanings in a machine readable format that can be queried and applied to dataset search.
[0012] The process of extracting meaning from these structured datasets using vocabularies in a manner such that machines and humans interpret it in a universal form is termed as “Semantics Extraction”. Additionally, apart from using only the metadata information from files such as title, description, source, publisher etc, there is a need to create a model for “Semantics Extraction” from each of the columns and rows and associate them with terms from respective vocabularies so as to enable selective retrieval of information from individual data sources.
[0013] Subsequently, the storage unit of the extracted metadata for the rows and columns of the tabular dataset should be able to scale to a large number of datasets existing in the public domain. This storage structure needs to ensure that the earlier versions of the data including their meanings can be retrieved.
[0014] Each dataset is associated with semantics or meaning which include spatial themes for location specific information, temporal themes for holding time periods and contextual themes (any topic that defines the data) at the file header level and at the row / column level. The semantics here refer to terms from well-defined and recognized knowledge bases / ontologies that lend meaning in a manner that both humans and machines can interpret. An ontology provides a unique universal identifier to concepts, things, places, persons or entities in a

manner that both humans and machines can interpret it in the same manner for what it represents.
[0015] US20100235354A1 discloses an apparatus and method for implementing a data integration and retrieval engine. A plurality of search tags/ themes associated with one or more search terms received from a client device associated with a user of a data integration and retrieval engine are determined. A plurality of datasets that are associated with information relevant to one or more of the search tags are identified. A time reference associated with the information associated with each of the users is determined. Datasets matching the criteria are retrieved with respect to the themes, at least in part, on the relevance of the information associated with each of the themes and the time reference associated with the information.
[0016] Various search engine systems or devices have been developed and widely used. But these systems or devices do not identify imperative (required or meaningful) data from disparate structured and unstructured datasets by semantically integrating data from multiple sources.
[0017] Due to the aforementioned drawbacks, there is a need to develop an effective data integration and retrieval system based on semantic associations that searches data by utilizing contextual clues for selective retrieval of the imperative data via integration and aggregation of data from multiple sources.
OBJECTS OF THE INVENTION
[0001] The primary object of the present invention is to overcome the disadvantages of the prior art.
[0002] An object of present invention is to provide a system that has the capability to identify imperative data from disparate structured and unstructured datasets.

[0003] Another object of the present invention is to provide a system to integrate and aggregate the imperative data with minimal human intervention.
[0004] Yet another object of the present invention is to provide a system that extracts meaning from the imperative data.
[0005] The foregoing and other objects of the present invention will become readily apparent upon further review of the following detailed description of the preferred embodiment as illustrated in the accompanying drawings.
SUMMARY OF THE INVENTION
[0006] The present invention is envisioned to a data integration and retrieval system and method based on semantic associations to facilitate a searching service by extracting meaning from the data available in public or private domain (in the form of structured data as comma separated values, JSONs, XMLs).
[0007] According to an embodiment of the present invention, a semantic based data integration and retrieval system comprises of an input module connected with the system that is accessed by user(s) for providing an input request for search via a search interface, a data collecting module linked with the system for collecting information from multiple data sources (i.e. public and private domains such as comma separated values, JSONs, XMLs), a pre-processing module associated with the collecting module for excluding null values and data cleansing from the information in order to get an imperative data, a data validation module in connection with the pre-processing module for confirming good quality of the imperative data via a validation routine protocol.
[0008] According to another embodiment of present invention, a system for semantic based data integration and retrieval comprises of a semantic processor

linked with the validation module for distinguishing relationships (i.e. spatial, temporal, contextual) in the imperative data and processing the relationships to compute meaning of the imperative data, a non-relational storage unit associated with the processor for storing the relationships, meaning and imperative data in an organized format (i.e. knowledge graph) for quick retrieval of data, and a data updating module linked with the storage unit for permitting the user(s) to expand, update the imperative data, a display unit for presenting the relationships, meaning and imperative data ideally in graphical format to the user(s).
[0009] According to an embodiment of present invention, a method for semantic data integration and retrieval comprises steps of specifying an input request by user(s) for searching via a search interface, extracting meaning of the request to collect information related to the input data from multiple data sources, retrieving an imperative data from the information by removing null value(s) from the information and other relevant data cleansing activities, validating the imperative data in order to confirm quality of the data, recognizing relationships in the data and mining out semantics of the data by processing the relationships, storing the semantics, relationships, imperative data in an organized format into a non-relational storage unit for fast retrieval, and displaying the data, relationships and meaning in a graphical format to the user as response to the request.
[0010] While the invention has been described and shown with particular reference to the preferred embodiment, it will be apparent that variations might be possible that would fall within the scope of the present invention.
BRIEF DESCRIPTION OF THE DRAWINGS
[0011] These and other features, aspects, and advantages of the present invention will become better understood with regard to the following description, appended claims, and accompanying drawings where:

Figure 1 illustrates a block diagram of system for semantic based data integration
and retrieval; and
Figure 2 illustrates a flow chart for semantic based data integration and retrieval.
DETAILED DESCRIPTION OF THE INVENTION
[0012] The following description includes the preferred best mode of one embodiment of the present invention. It will be clear from this description of the invention that the invention is not limited to these illustrated embodiments but that the invention also includes a variety of modifications and embodiments thereto. Therefore, the present description should be seen as illustrative and not limiting. While the invention is susceptible to various modifications and alternative constructions, it should be understood, that there is no intention to limit the invention to the specific form disclosed, but, on the contrary, the invention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the invention as defined in the claims.
[0013] In any embodiment described herein, the open-ended terms "comprising," "comprises,” and the like (which are synonymous with "including," "having” and "characterized by") may be replaced by the respective partially closed phrases "consisting essentially of," consists essentially of," and the like or the respective closed phrases "consisting of," "consists of, the like.
[0014] As used herein, the singular forms “a,” “an,” and “the” designate both the singular and the plural, unless expressly stated to designate the singular only.
[0015] The present invention relates to a semantic based data integration and retrieval that facilitates transforming the public data (structured data in the form of comma separated values, JSONs, XMLs) into accessible and understandable data with the help of semantic web technology.

[0016] The semantic data integration and retrieval system have multiple stages/services i.e. the primary service is first time semantic extraction employed by a data integration module, secondary service is periodic updates of semantic extraction that is done by a data updating module and lastly, servicing user request is processed by an input module.
[0017] Referring to figure 1, a block diagram of semantic data integration and retrieval system is illustrated. The system comprises of the data integration module that is installed in the system for identification of data sources of various structured datasets. After complete identification sources of the datasets are classified, then resource metadata is abstracted from the data source and raw dataset is extracted from the data source.
[0018] The datasets are sourced from public domains (e.g. Indian Open Government Portal) or from private sources that are authorized for access. The corpus of dataset sources is maintained as a list of web links / Uniform Resource Locator (URLs) and is continuously updated using most downloaded datasets present in the public domain or web scraping results. The web scraping is a technique of scanning the web for sources of structured datasets. The request for inclusion of a dataset also comes from specific user requests.
[0019] The data integration module is present in connection with a pre¬processing module. The pre-processing module classifies each data set as a source from where the raw dataset is downloaded without an API (Non-API Auto) or API based downloadable source (API based) or through manual downloads (Non-API MANUAL). API expanding to Application Programming Interface is a mechanism by which enterprises make their data available for downloads through program or code by providing an API key for each downloadable resource. Manual downloads are required for cases where the raw dataset is not accessible through a program or automatically, but requires a manual intervention wherein

the user is required to provide additional user information before downloading the dataset.
[0020] For identification of each classification, a source connector is used that extracts the metadata for the downloadable resource and performs the extraction or download of the dataset. The semantics extracted from the metadata at the dataset or file level is stored in a semantics enabled data structure (preferably in a knowledge graph). The knowledge graph stores the dataset Meta information such as its title, description, domain, source, publisher information and any spatial, temporal and contextual themes associated at the dataset level.
[0021] The pre-processor is associated with a data validation module that handle null values, complex header information, grouped rows and columns and data cleansing and data preparation. These are accomplished using a set of common validation routine protocol. The validation module is intended to provide certain well-defined guarantees for fitness, accuracy, and consistency for the dataset. The validation routine protocol comprises a set of executable rules and statements for validating the dataset. The statements used herein includes but not limited to format, date, domain, email, byte and time validators.
[0022] The datasets post validation are sent to an ingestion interface to extract semantics at the dataset content level from the rows and columns and discard the raw dataset post semantics. The ingestion interface uses the knowledge graph generated for the dataset meta vocabularies generated for the Indian context and other public vocabularies available in the open domain to extract meaning from the validated dataset and stores the semantics in a knowledge graph.
[0023] After successful validation a semantic processor that is linked with the validation module recognizes relationships in the dataset. The semantic processor has an ontology computing unit which has pre-defined vocabularies. The vocabularies are used to characterize the possible relationships in the dataset. Also

the computing unit provides a formal description of dataset as a set of concepts within the domain and relationships that hold the concepts inside the dataset.
[0024] To enable the descriptions, the computing unit specifies the instance of objects, classes, attributes and relationships as well as restrictions, rules and axioms. The relationships used herein are ideally spatial, temporal and contextual. The spatial denotes the geographic location, temporal indicates time information and contextual depicts content. For instance, if the user wants to find a dataset to determine an approximate or preferable time period when dengue fever is active. So, the information collected from the data sources has some geographic location to indicate which regions have high rate of dengue fever, time information for indicating at what time period there is a high rate of fever like in the month of June, July. The relationships are processed by the semantic processor for extracting the meaning of the dataset.
[0025] The actual content of the dataset is stored in a non-relational database (Wide column store) to facilitate fast retrieval of information based on a key. The key here depicts the various themes that have been extracted and stored in the knowledge graphs. The meanings, relationships and dataset are send to a storage unit that is associated with the processor for storing in an organized format. The storage unit used herein is ideally a non-relational storage unit. The non-relational storage unit does not use the tabular schema of rows and columns found in most traditional storage unit. Instead, non-relational databases use a storage model that is optimized for the specific requirements of the type of data being stored. For example, data may be stored as simple key/value pairs, as JSON documents, or as a graph consisting of edges and vertices.
[0026] The system also comprises of a display unit for presenting the relationships, meaning and dataset ideally in graphical format. The graphical format used herein includes bar graph but not limited to line graph, pie chart Mosaic or Mekko Charts. The display unit used here is ideally LCD monitor

display but the display unit is not limited to LED, cathode ray tube.
[0027] This combination of storing semantics and actual data from the source datasets creates a unique lethal mechanism that facilitates access to complete and selective data from multiple datasets based on the contextual clues provided in the search query. However, “Issue of Currency” of raw data in the data source needs to be addressed with the semantics extracted for the data through the semantic processor. “Issue of Currency” deals with the condition when the user updates the data in the source but the semantics extracted is outdated. Hence any process of retrieving data based on the user specified query will become invalid if an outdated semantics is used to service the user’s request. This is addressed by the data updating module.
[0028] The data updating module is linked with the storage unit for enabling the user(s) to update semantics in periodic interval. The data updating module ensures that the meaning so extracted and stored locally in the storage module is the latest meaning with the latest dataset published by the source. The data updating module entails by accessing each dataset link from the list of URLs, the resource metadata and raw dataset is extracted for each resource, a timestamp ( date-time when dataset was last updated by the publisher) is matched to the timestamp of the dataset last accessed by the semantic processor, appropriate actions based on the match results are performed.
[0029] If the timestamp of the extracted dataset during periodic updates is not different from the timestamp of the dataset during the latest update from the semantic processor, no action is performed, else the process of semantics extraction is repeated for the updated dataset.
[0030] The proposed system has the input module for servicing user’s data request. In this input module, user’s query is received in the form of a collection

of terms or words in English or any Indian language such as Hindi. The words or terms are processed through a natural language processing protocol to identify the primary intended or meaning of the request. The natural language processing protocol comprises a set of executables instructions for conducting a lexical analysis of the request i.e. breaks the words or terms into a series of tokens, by removing any whitespace or comments present in the terms such as “*, .,^,$,!”. These words are then processed to identify spatial and temporal clues. The terms are sent through an algorithm to identify terms from Indian context specific vocabularies to identify contextual themes. These themes are then matched with the knowledge graph edges connecting datasets to various themes to identify the datasets that matches the query.
[0031] Additionally, the terms from the user specified query is also matched with the knowledge graph edges holding semantics from the dataset content from its various rows and columns. This match results in the specific rows and columns from the respective datasets that match the user specified query and enables the selective integration of information from multiple datasets.
[0032] Lastly, if the dataset has been updated at the source from the last time its semantics was extracted, a user notification is sent out on the current latency of the data.
[0033] Referring to Figure 2, a flow chart is illustrated for the semantic data integration and retrieval method. The method comprises steps of identification of data sources (such as comma separated values, JSONs, XMLs) of various structured datasets. After complete identification, sources of the datasets are classified, then resource metadata is abstracted from the data source and raw dataset is extracted from the data source. Each data set is classified as a source from where the raw dataset is downloaded without an API (Non-API Auto) or API based downloadable source (API based) or through manual downloads (Non-API MANUAL). Null values, complex header information, grouped rows and

columns and data cleansing and data preparation are handled via a common validation routine protocol. Relationships are recognized in the dataset by analyzing the classified source. Semantics are extracted from the dataset by processing the relationships. The semantics, relationships of the dataset are stored in a knowledge graph. For semantic data retrieval, an input request is provided by user(s) for searching via a search interface. The input request may be provided in any language such as English or other Indian languages. The accurate meaning of the request is extracted by natural language processing of the request.
[0034] Although the field of the invention has been described herein with limited reference to specific embodiments, this description is not meant to be construed in a limiting sense. Various modifications of the disclosed embodiments, as well as alternate embodiments of the invention, will become apparent to persons skilled in the art upon reference to the description of the invention.

We Claim:
1. A semantic data integration and retrieval system, comprising;
i. an input module installed in said system that is accessed by user(s)
for providing an input request for search via a search interface; ii. a data integration module connected with said system for gathering
information related to said request from plurality of data sources; iii. a pre-processing module associated with said collecting module for
selective retrieval of imperative data from said information in order
to eliminate null value(s) from said information and perform
relevant data cleansing service(s); iv. a data validation module in connection with said pre-processing
module for ensuring quality of said data via a validation routine
protocol; v. a semantic processor associated with said validation module for
recognizing relationships in said data and processing said
relationships in order to extract meaning from said data; vi. a storage unit associated with said processor for storing said
relationships, meaning and data in an organized format for fast
retrieval of data; and vii. a data updating module linked with said storage unit for enabling
said user(s) to expand, update said imperative data.
2. The system as claimed in claim 1, wherein said system comprise of a display unit for presenting said relationships, meaning and imperative data ideally in graphical format.
3. The system as claimed in claim 1, wherein said input request is provided in a user desired language including English and other Indian languages.
4. The system as claimed in claim 1, wherein said input request is processed through a natural language processing protocol for gathering said information.

5. The system as claimed in claim 1, wherein said sources include public and private domains datasets such as comma separated values, JSONs, XMLs but not limited to surveys, data from private enterprise.
6. The system as claimed in claim 1, wherein said relationships include but not restricted to spatial, temporal, contextual.
7. The system as claimed in claim 1, wherein said storage unit is preferably a non-relational storage unit.
8. A semantic data integration and retrieval, method comprises the steps of :
a. specifying an input request for searching via a search interface;
b. extracting meaning of said request in order to gather information related to
said input data from plurality of data sources upon extraction;
c. eliminating null values from said information for selective retrieval of
imperative data;
d. validating said imperative data in order to ensure quality of said data;
e. recognizing relationships in said data by analysing the classified source;
f. extracting semantics of said data by processing said relationships;
g. storing said semantics, relationships and imperative data in an organized
format into a non-relational storage unit for fast retrieval; and
h. displaying said data, relationships and semantics in a graphical format.