FIELD OF THE INVENTION
[0001] This invention pertains to the field of data deduplication, and more
particularly, to the field of deduplication of data chunks in client devices
interspersed in a networked environment.
BAKGROUND OF THE INVENTION
[0002] The proliferation of intranets, extranets and the internet has made
way for efficient networked environments based on Information &
Communication Technology which allow authorized persons and/ or entities to
work towards the development, completion and execution of projects,
irrespective of their geographical locations. In these networked environments,
usually, the projects to be executed are broken down into smaller aspects which
are then co – dependently and co – ordinatively, executed by the persons and/ or
the entities authorized thereof. Thus in this context, person skilled in the art
would appreciate that prompt and timely sharing of data/ information relevant to
the project sought to be developed, completed and executed (henceforth called
“electronic data file”) between the authorized persons and/ or the entities
4/46
becomes indispensable, because not doing so leads to failure of co-ordination
between the authorized persons and/ or the entities, which in turn, leads to
complete or partial failure of the project itself.
[0003] The prompt and timely transferring of the electronic data files is
achieved by creating, at the client device bearing the electronic data file to be
transferred, data chunks corresponding to the electronic data file to be transferred,
and transmitting the data chunks to those other client device(s) in the networked
environment wherefrom a request to receive the electronic data file to be
transferred, is made. Person skilled in the art would appreciate that creation and
transmission of data chunks is preferred over transmission of the electronic data
file in its entirety, because the later suffers from various problems that the former
successfully overcomes. The transferring of the electronic data file in its entirety
may suffer from problems such as lags in transfer, degradation of quality of
content, and distortion of content, and in certain exceptional cases, may also fall
victim to complete failure to transfer. Such exceptional cases may arise when the
client devices requesting to receive the electronic data files to be transferred have
a cap on the size of the electronic data files that they would be able to receive in
one instance of transfer, and the electronic data file to be transferred therein
exceed that size cap. For instance, client devices with FAT 32 file systems have a
size cap of 4GB and therefore any electronic data file whose size is in excess of
4GB, would not get transferred at such client devices having FAT 32 file
systems, in a single instance of transfer.
5/46
[0004] The creation and transferring of data chunks, despite being the
preferred way of transferring electronic data files, may suffer from the problem of
duplicate and/or redundant generation of data chunks which when stored at the
client devices where they are generated, may unnecessarily consume storage
space therein. For instance, if a request to receive an electronic data file
“ABC.txt” from a client device C1 is made by client devicesC2 and C4, all of
which are interspersed in the networked environment, then under normal
circumstances, two sets of identical data chunks of “ABC.txt” would be created
at the client device C1, which when stored therein would unnecessarily consume
the storage system. Person skilled in the art would understand that it would be
more space efficient to store one set of data chunks of the electronic data file
“ABC.txt” in the client device C1 and create their copies whenever a request for
the electronic data file to be transferred is received, instead of storing two sets of
data chunks of the file “ABC.txt” for client devices C2 and C4, each.
[0005] This invention, therefore, is directed to address the aforesaid
problem and offer a solution that easily and efficiently, deduplicates and/ or
eliminates the redundancy of the data chunks at the client devices where they are
generated.
6/46
SUMMARY OF THE INVENTION
[0006] The present invention is intended to address at least the above
mentioned problems and/or disadvantages and to provide a suitable solution.
Accordingly, an aspect of the present invention is to provide an expansion
cartridge attachable, externally, to client devices for deduplicating the data
chunks of the electronic data files to be transferred, stored therein, and a method
using the expansion cartridge attachable, externally, with the client device for
deduplicating the data chunks of the electronic data files to be transferred, stored
therein.
[0007] An aspect of the present invention, therefore, is an expansion
cartridge attachable, externally, to client devices carrying electronic data files to
be transferred, wherein the expansion cartridge is characterized by a file
management component, a chunk management component, a storage component,
and a mirroring component, wherein the expansion cartridge on being attached
with the client devices interfaces with a client side data historian and a client side
processor in the client device.
[0008] Yet another aspect of the present invention, therefore, is a method
for deduplicating, at the client device, the data chunks of the electronic data files
to be transferred, using the expansion cartridge wherein the method comprises the
steps of determining a Unique Identification String for the electronic data file to
7/46
be transferred, generating a preliminary set of metadata for the electronic data file
to be transferred, dividing the electronic data file to be transferred for generating
data chunks, generating a final set of metadata for each data chunk generated
thereof, and comparing the data chunks with the final sets of metadata generated
thereof with the data chunks with final sets of metadata already stored at the one
or more discrete tuples in the client side data historian, and, discarding or storing
the data chunks with the final sets of metadata generated thereof, based on the
comparing thereof.
[0009] Other aspects, advantages and salient features of the invention will
become apparent to those skilled in the art from the following detailed
description, which, taken in conjunction with the annexed drawings, discloses
exemplary embodiments of the invention.
BRIEF DESCRIPTION OF DRAWINGS
[0010] The above and other aspects, features and advantages of certain
exemplary embodiments of the present invention will be more apparent from the
detailed description that follows taken in conjunction with the accompanying
drawings, wherein:
8/46
[0011] Fig.1 illustrates a block diagram of a client device (100)
interspersed in the networked environment,
[0012] Fig.2 illustrates an expansion cartridge (200) in accordance with
the invention,
[0013] Fig.3 illustrates a table containing the final sets of metadata
generated for sample data chunks pertaining to an electronic data file to be
transferred, and
[0014] Fig.4 illustrates the problem of selective, acceptance or rejection,
of data chunks pertaining to an electronic data file to be transferred, at client
device (100), depending of the size of the data chunks and the number of data
chunks generated thereof.
DETAILED DESCRIPTION OF THE INVENTION
[0015] Advantages and features of the present invention and methods of
accomplishing the same may be understood more readily by referring to this
detailed description of preferred embodiments and the accompanying drawings.
The present invention may, however, be embodied in many different forms and
should not be construed as being limited to the embodiments set forth herein.
9/46
Rather, these embodiments should be considered for thorough and complete
disclosure of the invention and for fully conveying the concept of the invention to
those skilled in the art. For the sake of simplicity various aspects of the preferred
system and method embodiments are referred using numerals which refer to the
same aspects throughout the specification.
[0016] The terminology used herein is for the purpose of describing
particular embodiments only and is not intended to limit the invention. As used
herein, the singular forms “a”, “an” and “the” are intended to include the plural
forms as well, unless the context clearly indicates otherwise. It will further be
understood that the terms “comprises” and/or “comprising”, when used in this
specification, specify the presence of stated features, integers, steps, operations,
elements, and/or components, but do not preclude the presence or addition of
other features, integers, steps, operations, elements, and/or components.
[0017] Various techniques and mechanisms of the present invention will
sometimes be described in singular form for clarity. Such singular descriptions of
techniques and mechanisms may however include multiple iterations of that
technique and/ or multiple instantiations of that mechanism, respectively, unless
the context suggests otherwise. Furthermore, various techniques and mechanisms
of the present invention will sometimes describe a connection between two
entities. It should be noted that the connection between two entities does not
10/46
necessarily mean a direct, unimpeded connection, as a variety of other entities
may reside between the two entities. Further the connection between any two
entities disclosed in the invention should not only be construed as being
dedicated hard wired connections, but should also include virtual and/ or wireless
connections.
[0018] It will be understood that, although the terms first, second, third
etc., may be used herein to describe some elements, components, regions, layers
and/or sections, these elements, components, regions, layers and/or sections
should not be limited by these terms. Thus, a first element, component, region,
layer and/ or section disclosed in the description could be termed as a second
element, component, region, layer and/ or section, or vice versa, without
departing from the teachings of the present invention.
[0019] Unless otherwise defined, the terms (including technical and
scientific terms) used herein shall have the same meaning as commonly
understood by one of ordinary skill in the art. Further, the terms defined in
commonly used dictionaries should be interpreted and construed in a manner
consistent with their dictionary meaning, and should not be interpreted in an
idealized or overly formal sense unless expressly defined.
11/46
[0020] Fig.1 to Fig.4, discussed below are non limiting illustrations which
have been used to explain and describe the invention disclosed herein. Persons
skilled in the art will appreciate that the purpose of these figures is to provide
clarity to the concepts associated with the various technical embodiments of the
invention and not to limit the invention. These figures include without limitation,
block diagrams, flowcharts, schematics and/ or other simplistic representations
that explain the various aspects of the invention with ease and effectiveness.
[0021] Referring now to drawings, and first to Fig.1, a block diagram of a
client device (100) located in a networked environment, is designated by the
numeral (100) which comprises a client side data historian (125) based on
industrially accepted databases that store the electronic data files and/ or their
data chunks at discrete tuples which may access, operate, federate, combine,
sync, and link with the other discrete tuples therein, and a client side processor
(150) with specialized client side processing units including integrated system
(bus) controllers, memory management control units, floating point units,
graphics processing units, digital signal processing units, or a combination
thereof. Further, in accordance with this invention, the client devices (100) are
configured to embody, removably, an expansion cartridge (200) which, in
different embodiments, may either be inserted directly in the client devices (100)
or may be inserted over a removable drive bay mounted on the client devices
(100). Upon insertion of the expansion cartridge (200) in the client device (100),
the expansion cartridge (200) interfaces with the client side data historian (125)
12/46
and the client side processor (150), using interfacing options such as Small
Computer System Interface (SCSI), Fibre Channel (FC) interface, Ethernet
interface, Advanced Technology Attachment (ATA) interface, or a combination
thereof, on a per case basis.
[0022] Fig. 2 is an exemplary illustration of the expansion cartridge (200)
in accordance with the invention. As may be seen in the illustration, the
expansion cartridge (200) is a specially – designed plug in hardware, which
comprises male as well as female ports, slots, electrical contacts and arms,
electrical connectors and all such elements useful for compatibly, connecting, the
expansion cartridge (200) with the client devices (100). Person skilled in the art
would appreciate that the expansion cartridge (200), being a multi – functional
hardware device, would include inter alia a dedicated processing unit in
conjunction with active components such as transistors, diodes/ triodes, vacuum
tubes (valves), and tunnel diodes, or passive components such resistors,
capacitors, inductors and transformers, wherein, both, the active components and
the passive components, may be packaged discretely or in arrays, or may be
integrated inside of packages such as semiconductor integrated circuits, hybrid
integrated circuits, or thick film devices.
[0023] In accordance with the invention, upon the insertion of the
expansion cartridge (200) in the client device (100), the expansion cartridge
13/46
(200), firstly, connects with the client side processor (150) therein, to obtain the
electronic data file side, identified by the client side processor (150), for
transferring. On obtaining the electronic data file to be transferred, the expansion
cartridge (200) subjects the electronic data file to be transferred to its components
for further processment, which in accordance with the invention include a file
management component (220), a chunk management component (240), a storage
component (260) and a mirroring component (280). The expansion cartridge
(200), firstly, invokes the file management component (220) that acts upon the
electronic data file and executes a two – fold operation. Firstly, the file
management component (220) generates a Unique Identification String for the
electronic data file to be transferred, using a non – repetitive string generation
operation wherein the Unique Identification String generated thereof is consistent
with a fixed pattern (e.g. six-character alphanumeric strings starting with “$”
symbol). Next, the file management component (220) creates a preliminary set of
metadata for the electronic data file with Unique Identification String. As per a
preferred characterization, the preliminary set of metadata for a particular
electronic data file with particular Unique Identification String comprises:
(1) “SRC”, that is, the MAC/ Network Address of the client device (100)
where the particular electronic data file is present,
(2) “DSTN”, that is, the MAC/ Network Address of the client device (100)
making the request to receive the particular electronic data file,
14/46
(3) “UID”, that is, the Unique Identification String (UID) generated for the
particular electronic data file,
(4) “FNAME”, that is, the name of the particular electronic data file,
(5) “FSIZE”, that is, the size of the particular electronic data file, and
(6) “FFORMAT”, that is, the format of the particular electronic data file.
[0024] Accordingly, in an example, if the electronic data file “ABC.txt”
of 4 MB stored at the client device (100) C1 having MAC/ Network Address
00:A1:C7:18:F2:18, is requested by the client device (100) C4 having MAC/
Network Address 00:A1:C7:16:F9:01, then the file management component
(220) of C1 would firstly generate the Unique Identification String for
“ABC.txt”, say “$7H6DD5”, and thereafter create the preliminary set of metadata
for the electronic data file which would include -
“SRC” = 00:A1:C7:18:F2:18, “DSTN” = 00:A1:C7:16:F9:01, “UID” =
$7H6DD5,
“FNAME” = ABC.txt, “FSIZE” = 4MB, and “FFORMAT” = .txt (text).
[0025] Once the preliminary set of metadata is generated for the
electronic data file to be transferred, then all the operations at/ in relation to the
file management component (220) are deemed to be completed, and thereafter the
electronic data file to be transferred and its preliminary set of metadata are moved
15/46
to the chunk management component (240) which firstly divides the electronic
data file to be transferred to generate data chunks using chunking parameters that
include without limitation, the size/ size range of the data chunks, and the
chunking technique to be used, which in a preferred embodiment may be
obtained from the registered users of the networked environment.
[0026] To understand the functioning of the chunk management
component (240) in connection with the division of the electronic data file to be
transferred to generate data chunks, consider an example where a 100 MB image
file is received at the chunk management component (240) and the chunking
parameters for the image file set the chunking technique for the image file to
fixed- size chunking and the size of each data chunk to 1 MB then in such case,
the chunk management component (240) would create 100 data chunks of 1 MB
each. Likewise, in another example if a text file is received at the chunk
management component (240) and the chunking parameters for the text file set
the chunking technique to Basic sliding, then the chunk management component
(240) would introduce condition logics in the text file that mark chunk
boundaries therein, and then divide the text file at the said chunk boundaries to
obtain the requisite data chunks. Similarly, in another example if an AV (audio –
visual) file is received at the chunk management component (240) and the
chunking parameters for the AV file set the chunk technique to TTTD (Two
threshold two divisor) and the size range of the data chunks then the chunk
16/46
management component (240) would generate data chunks within the said size
range.
[0027] In accordance with the invention, while the chunk management
component (240) performs the operation of dividing the electronic data file and
generating data chunks, the chunk management component (240) also triggers
numerous intrinsic sub – operations, the first one being, recording the number at
which each data chunk is generated for the electronic data file to be transferred,
and the total number of data chunks generated thereof, and, attributing them as
the “CNO” and “TNO” of that data chunk, respectively. In accordance with the
invention, the chunk management component (240) performs this sub – operation
by initializing a counter variable at “0” and incrementing it by “1” every time a
data chunk is generated for the electronic data file, such that the value of the
counter variable after every increment is retained as the “CNO” of that data
chunk, and the finally incremented value of the counter variable after the
electronic data file to be transferred is divided completely, and all of its data
chunks are generated thereof, is retained at the “TNO” of that data chunk. Thus,
by the end of this sub – operation, the “CNO” and “TNO” values for all the data
chunks generated from the electronic data file to be transferred are obtained, and
the counter variable is reset to “0” so as to iteratively determine the values of
“CNO” and “TNO” for the data chunks generated from the next electronic data
file as well, in a similar fashion. As per an example of this sub - operation, if
electronic data file “ABC.txt” of 3MB is moved into the chunk management
17/46
component (240) to be divided into three data chunks then when the first data
chunk is generated, the value of the counter variable value is incremented from
“0” to “1” and the “CNO” value, therefore, of the first data chunk is retained as
“1”. Similarly when the second data chunk is generated, the value of the counter
variable value is incremented from “1” to “2” and the “CNO” value, therefore, of
the second data chunk is retained as “2”. Likewise, when the third data chunk is
generated, the value of the counter variable value is incremented from “2” to “3”
and the “CNO” value, therefore, of the third data chunk is retained as “3”.
Further, as per this sub - operation, once the data chunks are generated for the
electronic data file “ABC.txt”, the finally incremented value of the counter
variable (3, in this case) is retained as the “TNO” value for all the three data
chunks of the electronic data file “ABC.txt”. Once this sub – operation is
completed for the electronic data file “ABC.txt”, the counter variable value is
reset to “0” so that the values of “CNO” and “TNO” become calculable in a
similar manner for the next electronic data file moved into the chunk
management component (240).
[0028] A second intrinsic sub – operation that the chunk management
component (240) triggers in concurrence with the first intrinsic sub – operation is
that of recording the size of each data chunk generated for the electronic data file
to be transferred, and attributing the size as the “CSIZE” of that data chunk. In a
preferred embodiment, the chunk management component (240) achieves this for
18/46
each data chunk by measuring the byte size between the end limits of each data
chunk generated thereof.
[0029] A third intrinsic sub – operation that the chunk management
component (240) triggers in concurrence with the first intrinsic sub – operation
and the second intrinsic sub – operation is that of naming each data chunks
generated thereof as per at least one nomenclature rule, and attributing that string
as the “CNAME” of the data chunk named thereof. In a preferred embodiment,
this sub – operation may name the data chunks generated from the electronic data
file to be transferred as per the following naming formula -
wherein FNAME, CNO and FFORMAT,
represent the name of the electronic data file, the number at which the data chunk
for the electronic data file to be transferred is generated, and the format of the
electronic data file to be transferred, respectively. Accordingly, as per an example
of this sub - operation, if three data chunk pertaining to the electronic data file
“ABC.txt” are subjected to this sub – operation then the first data chunk which is
generated would called “ABC_1.txt”, the second data chunk which is generated
would be called “ABC_2.txt”, and the third data chunk which is generated would
be called “ABC_3.txt”, all of which would be saved as their respective
“CNAME” values.
19/46
[0030] Once the chunk management component (240) divides the
electronic data file to be transferred and generates the data chunks therefrom, in
conjunction with the first intrinsic sub – operation, the second intrinsic sub –
operation, and the third intrinsic sub – operation, the chunk management
component (240), generates a final set of metadata for each data chunk generated
thereof, by integrating the preliminary set of metadata of the electronic data file
to be transferred, generated at the file management component (220), with the
“CNO”, “TNO”, “CSIZE” and “CNAME” obtained per chunk, pursuant to the
first, second and third intrinsic operations, at the chunk management component
(240).
[0031] Fig.3 illustrates a table of the final set of metadata generated for
the electronic data file “ABC.txt” of 4 MB at the chunk management component
(240). In accordance with the illustration this generation of the final set of
metadata for “ABC.txt” is initiated by taking the preliminary set of metadata for
the electronic data file “ABC.txt”, generated at the file management component
(220), viz. –
“SRC” = 00:A1:C7:18:F2:18 (i.e. MAC address of the client device (100)
where ABC.txt is present)
“DSTN” = 00:A1:C7:16:F9:01, (i.e. MAC address of the client device
(100), where ABC.txt is to be transferred)
20/46
“UID” = $7H6DD5, (i.e. the UID created for ABC.txt)
“FNAME” = ABC.txt, (i.e. the name of the ABC.txt file)
“FSIZE” = 4MB, (i.e. the file size of ABC.txt)
“FFORMAT” = .txt (text), (i.e. the file format of ABC.txt)
and, integrating the values of the “CNO”, the “TNO”, the “CSIZE” and the
“CNAME” obtained for each data chunk at the chunk management component
(240), pursuant to the first intrinsic sub – operation, the second intrinsic sub –
operation, and the third intrinsic sub –operation described above. In accordance
with the illustration, the values of the “CNO”, the “CSIZE”, the “TNO” and the
“CNAME” details for each data chunks generated thereof is -
First Data chunk – “CNO” = 1, “TNO” = 4, “CSIZE” = 1 MB and “CNAME” =
ABC_1.txt
Second Data chunk – “CNO” = 2, “TNO” = 4, “CSIZE” = 1 MBand “CNAME”
= ABC_2.txt
Third Data chunk – “CNO” = 3, “TNO” = 4, “CSIZE” = 1 MBand “CNAME” =
ABC_3.txt,
Fourth Data chunk – “CNO” = 4, “TNO” = 4, “CSIZE” = 1 MBand “CNAME” =
ABC_4.txt
21/46
and therefore, the final set of metadata for each data chunk generated thereof
shall be –
First Data Chunk –
“SRC” = 00:A1:C7:18:F2:18, “DSTN” = 00:A1:C7:16:F9:01, “UID” =
$7H6DD5, “FNAME” = ABC.txt, “FSIZE” = 4MB, “FFORMAT” = .txt (text),
“CNO” = 1, “TNO” = 4 and “CNAME” = ABC_1.txt.
Second Data Chunk –
“SRC” = 00:A1:C7:18:F2:18, “DSTN” = 00:A1:C7:16:F9:01, “UID” =
$7H6DD5, “FNAME” = ABC.txt, “FSIZE” = 4MB, “FFORMAT” = .txt (text),
“CNO” = 2, “TNO” = 4 and “CNAME” = ABC_2.txt.
Third Data Chunk –
“SRC” = 00:A1:C7:18:F2:18, “DSTN” = 00:A1:C7:16:F9:01, “UID” =
$7H6DD5, “FNAME” = ABC.txt, “FSIZE” = 4MB, “FFORMAT” = .txt (text),
“CNO” = 3, “TNO” = 4 and “CNAME” = ABC_3.txt, and
Fourth Data Chunk –
22/46
“SRC” = 00:A1:C7:18:F2:18, “DSTN” = 00:A1:C7:16:F9:01, “UID” =
$7H6DD5, “FNAME” = ABC.txt, “FSIZE” = 4MB, “FFORMAT” = .txt (text),
“CNO” = 4, “TNO” = 4 and “CNAME” = ABC_4.txt.
[0032] Once the data chunks with final sets of metadata are generated
from the electronic data file to be transferred, at the chunk management
component (240), all activities at/ in relation to the chunk management
component (240) are deemed to be completed, and the data chunks with final sets
of metadata are forwarded to the storage component(260) which in the broadest
sense communicably connects with the client side data historian (125) to
selectively store, at discrete tuples therein, the data chunks with final sets of
metadata.
[0033] Person skilled in the art would appreciate that in accordance with
the invention, the purpose of having a client side data historian (125) that stores
data/ information at discrete tuples therein is to enable the storing of data chunks
with final sets of metadata at contiguous cells within the same tuple. This way all
the data chunks pertaining to a particular electronic data file would be stored
within the same tuple. Person skilled in the art would further appreciate that in
accordance with the invention, the tuples at the client side data historian (125)
would be dynamic so that the cells therein could be increased or decreased to
23/46
accommodate, exactly, the data chunks with the final sets of metadata for the
particular electronic data file.
[0034] Thus in conspectus of the forgoing, the storage component (260)
upon receiving the data chunks with the final sets of metadata from the chunk
management component (240), firstly compares the data chunks with final sets of
metadata received thereof against those data chunks already stored at the discrete
tuples at the client side data historian (125) to confirm whether they are already
present or not. In accordance with the invention, the storage component (260)
performs the comparison in two rounds. In the first round, the storage component
(260) reads the values of the “SRC”, “DSTN” and “UID” metadata from the final
set of metadata of any of the data chunks received thereof and matches them with
the “SRC”, “DSTN” and “UID” metadata from the final set of metadata of the
data chunks already stored at the discrete tuples in the client side data historian
(125) to determine whether or not there are any data chunks in the discrete tuples
in the client side data historian (125) that pertain to the electronic data file to be
transferred. If no match is found, then the storage component (260) deduces that
the data chunks with the final sets of metadata have been received for the first
time, and are therefore stored at an unoccupied discrete tuple therein. If however,
a match is found, then the storage component (260), probes further by comparing
the values of “TNO” and “CSIZE” of the data chunks with final sets of metadata
received at the client side data historian (125) against the values of “TNO” and
“CSIZE” of the data chunks which are present at that discrete tuple where the
24/46
values of the “SRC”, “DSTN” and “UID” metadata are found to be the same. If
match is found between the values of “TNO” and the “CSIZE” of all data chunks
with final sets of metadata compared thereof, then the storage component (260)
deduces that they are the same, and discards the data chunks with final sets of
metadata received at the client side data historian (125). If however, a match is
not found between the values of “TNO” and the “CSIZE” of all data chunks with
final sets of metadata compared thereof, then the storage component (260)
deduces that they are different, and stores them at an unoccupied discrete tuple in
the client side data historian (125).
[0035] Person skilled in the art would appreciate that while the matching
of the values of “SRC”, “DSTN” and “UID” in the final sets of metadata of the
data chunks received thereof with the values of “SRC”, “DSTN” and “UID” in
the final sets of metadata of the data chunks at the discrete tuples in the client
side data historian (125) is evidence enough to establish that the data chunks
pertain to the same electronic data file have been found, merely pursuing this
comparison to discard or store the data chunks with the final sets of metadata,
may lead to inefficiency as sometimes the size and number of the data chunks
pertaining to the electronic data file to be transferred may become pivotal in
determining whether the client device (100) requesting receipt of the data chunks
would be able to receive them or not.
25/46
[0036] Fig.4 illustrates a typical case of the scenario explained above. As
may be seen in the illustration when the electronic data file ABC.txt, 18 GB in
size, is divided into four data chunks of 4.5 GB each then all the data chunks
generate thereof may be rejected from entering a client device (100) having a
FAT 32 file system, as this file system places a cap of 4GB on the size of the
data/ information that the client device (100) may accept in one go. Likewise,
when the same electronic data file ABC.txt, is divided into nine data chunks of 2
GB each, then all the data chunks generated thereof would be accepted at the
same client device (100) as 2 GB is well within the size cap of 4 GB.
[0037] Thus as an outcome of the operation of comparison, and
subsequent storing and/ or discarding of the data chunks with final sets of
metadata, the storage component (260) does away with the problem of duplicate
and redundant storage of the data chunks at the discrete tuples at the client side
data historian (125), thereby optimizing storage space thereat.
[0038] Once the storage component (260) completes its operations, the
deduplication of the client side data historian (125) by the expansion cartridge
(200) is deemed to have been successfully accomplished. However, it is not just
an object of the invention to deduplicate the client side data historian (125) while
storing the data chunks with final sets of metadata therein, but to also maintain
the deduplicacy of the client side data historian (125) when the data chunks with
26/46
final sets of metadata are requested therefrom by other client devices (100) in the
networked environment. It may be the case that upon receiving the request for a
particular electronic data file to be transferred, the data chunks with the final sets
of metadata are removed and transferred from the client side data historian (125)
to the other client devices (100) requesting thereof, in which case all the data
chunks for the particular electronic data file would get lost from the client side
data historian (125) for future use and reference. In accordance with the
invention, this problem may be addressed by the mirroring component (280) that
connects with the client side data historian (125), and determines the discrete
tuple therein where the data chunks with the final sets of metadata requested
thereof are stored, and thereafter, instead of transferring the data chunks with
final sets of metadata stored therein, creates a copy of the data chunks with the
final sets of metadata stored at the discrete tuple determined thereof which is then
transmitted to the other client devices (100) in the networked environment
wherefrom the request of the data chunks with final sets of metadata is made.
This way the loss of data chunks is avoided and the integrity of the client side
data historian (125) is maintained.
[0039] In accordance with the invention, a method for deduplicating, at
the client device (100), the data chunks of the electronic data files to be
transferred, using the expansion cartridge (200) is also disclosed herein (300). As
per this method embodiment, firstly, the method triggers the step of determining
a Unique Identification String for the electronic data file to be transferred (310).
27/46
Thereafter, the method triggers the step of generating a preliminary set of
metadata for the electronic data file to be transferred (320). As per a preferred
embodiment, the generating the preliminary set of metadata for the electronic
data file to be transferred, consists of generating “SRC”, that is, the MAC/
Network address of the client device (100) where the electronic data file to be
transferred is present (321), generating “DSTN”, that is, the MAC/ Network
address of the client device (100) where the electronic data file to be transferred
is to be received (322), generating “UID”, that is, the Unique Identification String
for the electronic data file to be transferred (323), generating “FNAME”, that is,
the name of the electronic data file to be transferred (324), generating “FSIZE”,
that is, the size of the electronic data file to be transferred (325), and generating
“FFORMAT”, that is, the format of the electronic data file to be transferred
(326).
[0040] After generating the preliminary set of metadata for the electronic
data file to be transferred (320), the method triggers the step of dividing the
electronic data file to be transferred for generating data chunks (330). In
accordance with the invention, the step of dividing the electronic data file to be
transferred for generating data chunks (330), triggers during its execution, the sub
– steps of recording the number at which each data chunk is generated for the
electronic data file to be transferred, and, the total number data chunks generated
from the electronic data file to be transferred and attributing them as the “CNO”
and the “TNO” of that data chunk, respectively (333), recording the size of each
28/46
data chunk generated for the electronic data file to be transferred and attributing
the size as the “CSIZE” of that data chunk (336), and naming each data chunks
generated thereof as per at least one nomenclature rule and attributing that string
as the “CNAME” of the data chunk named thereof (339). Person skilled in the art
would appreciate that all of the forgoing sub – steps would be repeated until the
electronic data file to be transferred is divided, completely, and all data chunks
are generated thereof.
[0041] After dividing the electronic data file to generate data chunks
(330), the method triggers the next step of generating a final set of metadata for
each data chunk generated thereof (340), which in accordance with the invention
is achieved by, integrating the preliminary sets of metadata generated thereof, for
the electronic data file to be transferred, with the “CNO”, the “CSIZE”, the
“TNO” and the “CNAME” of each data chunk generated from the electronic data
file to be transferred.
[0042] After generating the final set of metadata for each data chunk
generated for the electronic data file to be transferred (340), the method triggers
the step of comparing the data chunks with the final sets of metadata generated
thereof with the data chunks with final sets of metadata already stored at the one
or more discrete tuples in the client side data historian (125), and, discarding or
storing the data chunks with the final sets of metadata generated thereof, based on
29/46
the comparing thereof (350). In accordance with the invention, this step is
achieved by further triggering numerous sub – steps, the first one being
comparing the values of the “SRC”, “DSTN” and “UID” from the final set of
metadata of any one of the data chunks with the final set of metadata generated
thereof, with the values of the “SRC”, “DSTN” and “UID” of any one of the data
chunks with the final set of metadata already stored at the one or more discrete
tuples in the client side data historian (125), (353).
[0043] Upon comparing (353) if a match is not found between the values
of the “SRC”, “DSTN” and “UID” from the final set of metadata of any one of
the data chunks generated thereof, and the values of the “SRC”, “DSTN” and
“UID” from the final sets of metadata of any of the data chunks stored at the one
or more discrete tuples in the client side data historian (125), the sub - step (353)
triggers storing the data chunks with the final sets of metadata received thereof at
an unoccupied discrete tuple in the client side data historian (125), (354). If
however, upon comparing (353), a match is found between the values of the
“SRC”, “DSTN” and “UID” from the final set of metadata of any one of the data
chunks generated thereof, and the values of the “SRC”, “DSTN” and “UID” from
the final sets of metadata of any of the data chunks stored at a specific discrete
tuple in the client side data historian (125), the sub - step (353) triggers reiterative
comparing of the “TNO” and the “CSIZE” from the final sets of metadata of each
data chunks generated thereof, with the “TNO” and the “CSIZE” from the final
30/46
sets of metadata of each data chunk stored at the specific discrete tuple in the
client side data historian (125), (356).
[0044] Upon comparison of the “TNO” and the “CSIZE” from the final
sets of metadata of each data chunks generated thereof, with the “TNO” and the
“CSIZE” from the final sets of metadata of each data chunk stored at the specific
discrete tuple in the client side data historian (125), (356), if a match is found
then the data chunks with the final sets of metadata received thereof are discarded
(357), and if a match is not found then the data chunks with the final sets of
metadata received thereof are stored at an unoccupied discrete tuple in the client
side data historian (125), (358).
[0045] Upon execution of the steps disclosed above, in connection with
the method (300), the deduplicating of the data chunks of the electronic data files
to be transferred, at the client device (100) shall successfully be accomplished.
However, in accordance with the invention, the method (300) is not just intended
for deduplicating the client side data historian (125) while storing the data chunks
with final sets of metadata therein, but also maintaining the deduplicacy of the
client side data historian (125) when the data chunks with final sets of metadata
are requested therefrom by other client devices (100) in the networked
environment. It may be the case that upon receiving the request for a particular
electronic data file to be transferred, the data chunks with the final sets of
31/46
metadata are removed and transferred from the client side data historian (125) to
the other client devices (100) requesting thereof, in which case all the data
chunks for the particular electronic data file would get lost from the client side
data historian (125) for future use and reference. Thus, in accordance with the
invention, the method (300) disclosed herein, may as per a preferred
embodiment, upon receiving a request from other client devices (100) in the
networked environment, for transferring a particular electronic data file thereat,
may comprise the additional steps of connecting with the client side data
historian (125), and determining the discrete tuple therein where the data chunks
with the final sets of metadata corresponding to the electronic data file requested
thereof are stored (360), and thereafter, instead of transferring the data chunks
with final sets of metadata stored therein, creating a copy of the data chunks with
the final sets of metadata stored at the discrete tuple determined thereof (370) and
transmitting the copy created thereof to the other client devices (100) in the
networked environment wherefrom the request of the data chunks with final sets
of metadata is made (380). This way the loss of data chunks is avoided and the
integrity of the client side data historian (125) is maintained.
[0046] While the present invention has been particularly shown and
described with reference to exemplary embodiments thereof, it will be understood
by those of ordinary skill in the art that various changes in form and details may
be made therein without departing from the spirit and scope of the present
invention as defined by the following claims.

I/ We claim:
1. An expansion cartridge (200) attachable, externally, to client devices
(100) carrying electronic data files to be transferred, wherein the
expansion cartridge (200) is characterized by –
a file management component (220),
a chunk management component (240),
a storage component (260), and
a mirroring component (280); and
wherein, the expansion cartridge (200) on being attached with the
client devices (100) interfaces with a client side data historian (125)
and a client side processor (150) in the client device (100) using
interfacing options, including without limitation, Small Computer
System Interfaces (SCSI), Fibre Channel (FC) Interface, Ethernet
Interface, Advanced Technology Attachment (ATA) Interface or a
combination thereof.
33/46
2. The expansion cartridge (200) as claimed in claim 1 wherein the file
management component (220) –
determines, a Unique Identification String for the electronic data
file to be transferred; and
generates, a preliminary set of metadata for the electronic data file
to be transferred, wherein the preliminary set of metadata consists
of –
“SRC”, that is, the MAC/ Network address of the client
device (100) where the electronic data file to be transferred
is present,
“DSTN”, that is, the MAC/ Network address of the client
device (100) requesting the receipt of the electronic data
file to be transferred,
“UID”, that is, the Unique Identification String for the
electronic data file to be transferred,
“FNAME”, that is, the name of the electronic data file to
be transferred,
34/46
“FSIZE”, that is, the size of the electronic data file to be
transferred, and
“FFORMAT”, that is, the format of the electronic data file
to be transferred.
3. The expansion cartridge (200) as claimed in claim 1 wherein the chunk
management component (240) divides the electronic data file to be
transferred to generate data chunks, and generates a final set of metadata
for each data chunk generated thereof.
4. The expansion cartridge (200) as claimed in claim 3 wherein the chunk
management component (240) while dividing the electronic data file to be
transferred to generate data chunks, further triggers, in conjunction, the
sub – operations of –
recording the number at which each data chunk is generated for
the electronic data file to be transferred, and the total number of
data chunks generated from the electronic data file to be
transferred, and attributing the numbers recorded thereof as the
“CNO” and the “TNO”, respectively, of that data chunk;
35/46
recording the size of each data chunk generated for the electronic
data file to be transferred, and attributing the size recorded thereof
as the “CSIZE” of that data chunk; and
naming each data chunks generated thereof as per at least one
nomenclature rule, and attributing the name as the “CNAME” of
that data chunk; and
wherein all the sub – operations are repeated until the electronic data
file to be transferred is completely divided and all data chunks are
generated thereof.
5. The expansion cartridge (200) as claimed in claim 3 wherein the chunk
management component (240) generates the final set of metadata for each
data chunk generated thereof by integrating the preliminary set of
metadata, generated for the electronic data file to be transferred, at the file
management component (220), with the “CNO”, the “TNO”, the “CSIZE”
and the “CNAME”, for each data chunks generated for the electronic data
file to be transferred, at the chunk management component (240).
6. The expansion cartridge (200) in claim 1 wherein the storage component
(260) –
36/46
receives the data chunks with the final sets of metadata for the
electronic data file to be transferred, from the chunk management
component (240); and
compares the data chunks with the final sets of metadata received
thereof with the data chunks with final sets of metadata already
stored at the one or more discrete tuples in the client side data
historian (125), by -
comparing the values of the “SRC”, “DSTN” and “UID”
from the final set of metadata of any one of the data
chunks with the final set of metadata generated thereof,
with the values of the “SRC”, “DSTN” and “UID” of any
one of the data chunks with the final set of metadata
already stored at the one or more discrete tuples in the
client side data historian (125),
storing the data chunks with the final sets of metadata
received thereof at an unoccupied discrete tuple in the
client side data historian (125), if a match is not found
between the values of the “SRC”, “DSTN” and “UID”
37/46
from the final set of metadata of any one of the data
chunks generated thereof, and the values of the “SRC”,
“DSTN” and “UID” from the final sets of metadata of any
of the data chunks stored at the one or more discrete tuples
in the client side data historian (125), upon comparing,
reiteratively comparing the “TNO” and the “CSIZE” from
the final sets of metadata of each data chunks generated
thereof, with the “TNO” and the “CSIZE” from the final
sets of metadata of each data chunk stored at that discrete
tuple in the client side data historian (125) where a match
is found between the values of the “SRC”, “DSTN” and
“UID” from the final set of metadata of any one of the data
chunks generated thereof, and the values of the “SRC”,
“DSTN” and “UID” from the final sets of metadata of any
of the data chunks stored at the one or more discrete tuples
in the client side data historian (125), upon comparing,
wherein on reiteratively comparing -
if a match is found between the “TNO” and the
“CSIZE” in the final sets of metadata of each data
chunks generated thereof, with the “TNO” and the
38/46
“CSIZE” in the final sets of metadata of each data
chunk stored at the discrete tuple, then the data
chunks with the final sets of metadata received
thereof are discarded, and
if a match is not found between the “TNO” and the
“CSIZE” in the final sets of metadata of each data
chunks generated thereof, with the “TNO” and the
“CSIZE” in the final sets of metadata of each data
chunk stored at the discrete tuple, then the data
chunks with the final sets of metadata received
thereof are stored at an unoccupied discrete tuple in
the client side data historian (125).
7. The expansion cartridge (200) as claimed in claim 1 wherein the
mirroring component (280) receives a request from other client devices
(100) in the networked environment to transfer the electronic data file to
be transferred, and thereafter -
connects with the client side data historian (125), and determines
the discrete tuple therein where the data chunks with the final sets
of metadata corresponding to the electronic data file requested
thereof, are stored;
39/46
creates a copy of the data chunks with the final sets of metadata
stored at the discrete tuple determined thereof; and
transmits the copy created thereof to the other client devices (100)
in the networked environment wherefrom the request of the data
chunks with final sets of metadata is made.
8. A method for deduplicating, at the client device (100), the data chunks of
the electronic data files to be transferred, using the expansion cartridge
(200), (300) wherein the method (300) comprises the steps of –
determining a Unique Identification String for the electronic data
file to be transferred (310);
generating a preliminary set of metadata for the electronic data file
to be transferred (320);
dividing the electronic data file to be transferred for generating
data chunks (330);
40/46
generating a final set of metadata for each data chunk generated
thereof (340); and
comparing the data chunks with the final sets of metadata
generated thereof with the data chunks with final sets of metadata
already stored at the one or more discrete tuples in the client side
data historian (125), and, discarding or storing the data chunks
with the final sets of metadata generated thereof, based on the
comparing thereof (350).
9. The method (300) as claimed in claim 8 wherein generating the
preliminary set of metadata for the electronic data file to be transferred
(320), consists of –
generating “SRC”, that is, the MAC/ Network address of the client
device (100) where the electronic data file to be transferred is
present (321);
generating “DSTN”, that is, the MAC/ Network address of the
client device (100) requesting the receiving of the electronic data
file to be transferred (322);
41/46
generating “UID”, that is, the Unique Identification String for the
electronic data file to be transferred (323);
generating “FNAME”, that is, the name of the electronic data file
to be transferred (324);
generating “FSIZE”, that is, the size of the electronic data file to
be transferred (325); and
generating “FFORMAT”, that is, the format of the electronic data
file to be transferred (326).
10. The method (300) as claimed in claim 8 wherein the dividing the
electronic data file to be transferred for generating data chunks, triggers
during its execution, the sub – steps of –
recording the number at which each data chunk is generated for
the electronic data file to be transferred, and the total number of
data chunks generated from the electronic data file to be
transferred, and, attributing the numbers recorded thereof as the
“CNO” and the “TNO”, respectively, of that data chunk (333);
42/46
recording the size of each data chunk generated for the electronic
data file to be transferred, and attributing the size as the “CSIZE”
of that data chunk (336); and
naming each data chunks generated thereof as per at least one
nomenclature rule, and attributing the name as the “CNAME” of
that data chunk (339); and
wherein all the sub – steps (333, 336, 339) are repeated until the
electronic data file to be transferred is divided, completely, and all
data chunks are generated thereof.
11. The method as claimed in claim 8, 9 and 10 wherein generating the final
set of metadata for each data chunk generated thereof (340) is achieved by
integrating the preliminary sets of metadata generated thereof, for the
electronic data file to be transferred, with the “CNO”, the “TNO”, the
“CSIZE” and the “CNAME”, of each data chunk generated from the
electronic data file to be transferred.
12. The method (300) as claimed 8 wherein the comparing the data chunks
with the final sets of metadata generated thereof with the data chunks with
final sets of metadata already stored at the one or more discrete tuples in
the client side data historian (125), and, discarding or storing the data
43/46
chunks with the final sets of metadata generated thereof based on the
comparing thereof (350), comprises -
comparing the values of the “SRC”, “DSTN” and “UID” from the
final set of metadata of any one of the data chunks with the final
set of metadata generated thereof, with the values of the “SRC”,
“DSTN” and “UID” of any one of the data chunks with the final
set of metadata already stored at the one or more discrete tuples in
the client side data historian (125), (353);
storing the data chunks with the final sets of metadata received
thereof at an unoccupied discrete tuple in the client side data
historian (125), if a match is not found between the values of the
“SRC”, “DSTN” and “UID” from the final set of metadata of any
one of the data chunks generated thereof, and the values of the
“SRC”, “DSTN” and “UID” from the final sets of metadata of any
of the data chunks stored at the one or more discrete tuples in the
client side data historian (125), upon comparing (353), (354); and
reiteratively comparing the “TNO” and the “CSIZE” from the
final sets of metadata of each data chunks generated thereof, with
the “TNO” and the “CSIZE” from the final sets of metadata of
44/46
each data chunk stored at that discrete tuple in the client side data
historian (125) where a match is found between the values of the
“SRC”, “DSTN” and “UID” from the final set of metadata of any
one of the data chunks generated thereof, and the values of the
“SRC”, “DSTN” and “UID” from the final sets of metadata of any
of the data chunks stored at the one or more discrete tuples in the
client side data historian (125), upon comparing (353), (356),
wherein on reiteratively comparing(356) -
if a match is found between the “TNO” and the “CSIZE”
in the final sets of metadata of each data chunks generated
thereof, with the “TNO” and the “CSIZE” in the final sets
of metadata of each data chunk stored at the discrete tuple,
then the data chunks with the final sets of metadata
received thereof are discarded (357), and
if a match is not found between the “TNO” and the
“CSIZE” in the final sets of metadata of each data chunks
generated thereof, with the “TNO” and the “CSIZE” in the
final sets of metadata of each data chunk stored at the
discrete tuple, then the data chunks with the final sets of
45/46
metadata received thereof are stored at an unoccupied
discrete tuple in the client side data historian (125), (358).
13. The method (300) as claimed in claim 8 wherein after the comparing the
data chunks with the final sets of metadata generated thereof with the data
chunks with final sets of metadata already stored at the one or more
discrete tuples in the client side data historian (125), and, discarding or
storing the data chunks with the final sets of metadata generated thereof
based on the comparing thereof (350), the method (300) may upon
receiving a request from other client devices (100) in the networked
environment to transfer the electronic data file to be transferred, may
trigger the additional sub – steps of –
connecting with the client side data historian (125), and
determining the discrete tuple therein where the data chunks with
the final sets of metadata corresponding to the electronic data file
requested thereof, are stored (360);
creating a copy of the data chunks with the final sets of metadata
stored at the discrete tuple determined thereof (370); and
transmitting the copy created thereof to the other client devices
(100) in the networked environment wherefrom the request of the
data chunks with final sets of metadata is made (380).