Sign In to Follow Application
View All Documents & Correspondence

Loading Of Delta Data Into Hive Table

Abstract: A data loading system (104) for loading delta data into a Hive table includes a determination module (122) to determine one or more partition folders of the Hive table to be updated based on the delta data, where a partition folder is indicative of partitions in the Hive table, and where the delta data includes data to be loaded into the Hive table. The data loading system (104) further includes a replacement module (124) to generate one or more temporary folders based on merging the delta data and data in the Hive table, where each temporary folder corresponds to a partition folder. The replacement module (124) may then replace the one or more partition folders with the one or more temporary folders to load the delta data into the Hive table, where each partition folder is replaced with the corresponding temporary folder.

Get Free WhatsApp Updates!
Notices, Deadlines & Correspondence

Patent Information

Application #
Filing Date
24 March 2014
Publication Number
40/2015
Publication Type
INA
Invention Field
COMMUNICATION
Status
Email
iprdel@lakshmisri.com
Parent Application

Applicants

TATA CONSULTANCY SERVICES LIMITED
Nirmal Building, 9th Floor, Nariman Point, Mumbai, Maharashtra 400021

Inventors

1. MANAKKAL, Rony Pius
Tata Consultancy Services INFOPARK, KUSUMAGIRI POST, KAKKANAD, Kochi - 682030, Kerala

Specification

CLIAMS:1. A data loading system (104) for loading delta data into a Hive table, the data loading system (104) comprising:
a processor (114);
a determination module (122) coupled to the processor (114) to determine one or more partition folders of the Hive table to be updated based on the delta data, wherein a partition folder is indicative of partitions in the Hive table, and wherein the delta data includes data to be loaded into the Hive table; and
a replacement module (124) coupled to the processor (114) to:
generate one or more temporary folders based on merging the delta data and data in the Hive table, wherein each temporary folder corresponds to a partition folder; and
replace the one or more partition folders with the one or more temporary folders to load the delta data into the Hive table, wherein each partition folder is replaced with the corresponding temporary folder.
2. The data loading system (104) as claimed in claim 1, wherein the determination module (122) obtains, from a distributed file system (108), the delta data to be loaded into the Hive table.
3. The data loading system (104) as claimed in claim 2, wherein the distributed file system (108) is a Hadoop Distributed File System (HDFS).
4. The data loading system (104) as claimed in claim 1, wherein the partition folder is indicative of one or more partitions of the Hive table and a partition value corresponding to each partition.
5. The data loading system (104) as claimed in claim 1, wherein the Hive table is a partitioned Hive table having a plurality of partitions, and wherein partitioning is based on column fields of the Hive table.
6. The data loading system (104) as claimed in claim 1, wherein the delta data comprises information relating to at least one of a plurality of partitions of the Hive table to be modified, information of at least one row to modify the plurality of partitions, and an indicator which represents an operation to be performed on the at least one row, and wherein the operation is at least one of an update operation, a delete operation, and an insert operation.
7. A method for loading delta data into a Hive table, the method comprising:
obtaining, from a distributed file system (108), the delta data to be loaded into the Hive table, wherein the delta data includes information relating to a plurality of partitions of the Hive table to be modified and information of at least one row to modify the plurality of partitions;
determining one or more partition folders of the Hive table based on the delta data, wherein a partition folder is indicative of partitions in the Hive table;
merging the delta data and data in the Hive table to generate one or more temporary folders, wherein a temporary folder corresponds to a partition folder; and
replacing the one or more partition folders with the one or more corresponding temporary folders to load the delta data into the Hive table.

8. The method as claimed in claim 7, wherein the distributed file system (108) is a Hadoop Distributed File System (HDFS).
9. The method as claimed in claim 7, wherein the Hive table is a partitioned Hive table having a plurality of partitions, and wherein partitioning is based on column fields of the Hive table.
10. The method as claimed in claim 7, wherein the delta data further includes an indicator which represents an operation to be performed on the at least one row, such that each row is represented by a single indicator, and wherein the operation is at least one of an update operation, a delete operation, and an insert operation.
11. A non-transitory computer-readable medium having embodied thereon a computer program for executing a method comprising:
obtaining, from a distributed file system, the delta data to be loaded into the Hive table, wherein the delta data includes information relating to a plurality of partitions of the Hive table to be modified and information of at least one row to modify the plurality of partitions;
determining one or more partition folders of the Hive table based on the delta data, wherein a partition folder is indicative of partitions in the Hive table;
merging the delta data and data in the Hive table to generate one or more temporary folders, wherein a temporary folder corresponds to a partition folder; and
replacing the one or more partition folders with the one or more corresponding temporary folders to load the delta data into the Hive table.
,TagSPECI:As Attached

Documents

Application Documents

# Name Date
1 SPEC IN.pdf 2018-08-11
2 FORM 5.pdf 2018-08-11
3 FORM 3.pdf 2018-08-11
4 FIG IN.pdf 2018-08-11
5 ABSTRACT1.jpg 2018-08-11
6 989-MUM-2014-Power of Attorney-130215.pdf 2018-08-11
7 989-MUM-2014-FORM 18.pdf 2018-08-11
8 989-MUM-2014-FORM 1(10-9-2014).pdf 2018-08-11
9 989-MUM-2014-Correspondence-130215.pdf 2018-08-11
10 989-MUM-2014-CORRESPONDENCE(10-9-2014).pdf 2018-08-11
11 989-MUM-2014-FER.pdf 2019-10-04
12 989-MUM-2014-FORM-26 [17-03-2020(online)].pdf 2020-03-17
13 989-MUM-2014-OTHERS [26-03-2020(online)].pdf 2020-03-26
14 989-MUM-2014-FER_SER_REPLY [26-03-2020(online)].pdf 2020-03-26
15 989-MUM-2014-DRAWING [26-03-2020(online)].pdf 2020-03-26
16 989-MUM-2014-COMPLETE SPECIFICATION [26-03-2020(online)].pdf 2020-03-26
17 989-MUM-2014-CLAIMS [26-03-2020(online)].pdf 2020-03-26
18 989-MUM-2014-Correspondence to notify the Controller [08-10-2020(online)].pdf 2020-10-08
19 989-MUM-2014-Written submissions and relevant documents [27-10-2020(online)].pdf 2020-10-27
20 989-MUM-2014-US(14)-HearingNotice-(HearingDate-21-10-2020).pdf 2021-10-03
21 989-MUM-2014-US(14)-ExtendedHearingNotice-(HearingDate-08-07-2022).pdf 2022-06-08
22 989-MUM-2014-Correspondence to notify the Controller [23-06-2022(online)].pdf 2022-06-23

Search Strategy

1 10thfilesearchstrategy_04-10-2019.pdf