INCREMENTAL DOWNLOAD FRAMEWORK
BACKGROUND
[0001] Businesses, financial institutions, and the like generally deploy many systems and
sub-systems to store data. For example, a financial institution (e.g., a bank) may deploy multiple
systems that integrate together to achieve a function. The financial institution may have core
banking related operational data (e.g., customer data, account data, transaction data, etc.) stored
in plural tables. Third-party systems (e.g., document generation systems that generate account
statements to present to the customers) may require data from multiple tables in a production
environment combined together. Additionally, third-party systems often do not require access to
the entire data stored in the plural tables, but rather only require incremental data that has
changed since the last time the third-party system captured data from the plural tables.
[0002] Change data capture ("CDC") tools are available to extract incremental data at a table
level. For example, ORACLE™ offers a CDC tool that logs all changes to one or more tables in
an application dataset to corresponding tables in a staging dataset. However, no tools exist to
accurately extract change data at an entity level. In other words, no tools currently exist to extract
data from an application dataset that spans multiple tables and possibly multiple systems.
[0003] Additionally, plural third-party systems may each require data from different tables
and each of the systems may require the data to be formatted and sequenced differently. Current
systems do not offer a standard framework that allows for extraction of data formatted or
sequenced as required by the plural third-party systems. The absence of such a framework leads
to multiple applications performing similar tasks on the data set and hence redundant processing
which wastes resources.
[0004] Further, third-party systems may each access change data at different times during the
day. Current systems do not offer a standard framework that allows each party to access or
download only the change data that has been captured since the respective party last accessed or
downloaded change data.
[0005] Improved incremental download frameworks are desired.
BRIEF DESCRIPTION OF THE DRAWINGS
[0006] FIG. 1 shows an exemplary dataflow for an incremental download framework.
[0007] FIG. 2 shows an exemplary Change Table Manager command prompt useful for
providing an interface for a user to view and change environmental variables for a change data
capture tool.
[0008] FIG. 3 shows an exemplary download specification file.
[0009] FIG. 4 shows an exemplary process flow performed by an incremental download
generator.
[0010] FIG. 5 shows an exemplary computing device useful for performing processes
disclosed herein.
[0011] While systems and methods are described herein by way of example and
embodiments, those skilled in the art recognize that systems and methods for incremental
download frameworks are not limited to the embodiments or drawings described. It should be
understood that the drawings and description are not intended to be limiting to the particular
form disclosed. Rather, the intention is to cover all modifications, equivalents and alternatives
falling within the spirit and scope of the appended claims. Any headings used herein are for
organizational purposes only and are not meant to limit the scope of the description or the
claims. As used herein, the word "may" is used in a permissive sense (i.e., meaning having the
potential to) rather than the mandatory sense (i.e., meaning must). Similarly, the words
"include", "including", and "includes" mean including, but not limited to.
DETAILED DESCRIPTION
[0012] Disclosed embodiments provide a framework for incremental downloads of entity
data spanning multiple tables. For example, a financial institution may store data in many tables
including a customer data table, an account data table, a transaction data table, and the like. A
third-party system may require data relating to a logical entity "account transactions" that may
include fields spanning across plural tables (e.g., each of the customer data table, account data
table, and transaction data table) within the financial institution's application dataset.
Embodiments may allow a user to define the logical entity (e.g., a combination of fields spanning
one or more tables) and the format and sequence in which data relating to the logical entity is
required by the third-party system. The system may download and collate the specified fields
from plural tables in a change data dataset to an intermediate table then extract, format, and
sequence the fields as specified to generate a download file. The system may then make the
download file available to the third-party system to download.
[0013] Thus, embodiments may allow for incremental downloads involving derived fields
(e.g., incremental data spanning across multiple tables). Embodiments may provide little to no
impact on existing transactions and batch processes. Additionally, embodiments may provide
interfaces to allow third-party systems to easily define custom downloads including logical
entities spanning across multiple tables and allow the third-party systems to define the format
and sequence of the download. By providing a standard framework for incremental downloads,
redundant efforts may be reduced and computing resources conserved.
[0014] FIG. 1 illustrates an exemplary data flow 100 for an incremental download
framework. Application dataset 105 may be the production dataset of an institution (e.g., a bank)
including data spanning plural tables. Application dataset 105 may receive thousands or more
changes and insertions to the various tables every day. For example, customers (e.g., account
holders) may change their customer information, thereby causing changes in the customer data
table, may change their account information, thereby causing changes in the account data table,
and may be involved in financial transactions, thereby causing changes in the transaction data
table. A third-party system may be tasked with providing a daily report of account transactions.
Account transactions may include a logical entity incorporating data from each of the above
mentioned transactions (i.e., customer information change, account information change, and
financial transactions).
[0015] A change data capture application 110 may be configured to extract change data from
plural identified tables in the application dataset. For example, embodiments may use the
ORACLE™ CDC tool to asynchronously capture change data from plural tables of the
application dataset. A user may define various parameters for a CDC tool by setting variables or
providing parameters to the CDC tool. FIG. 2 shows an exemplary Change Table Manager
("CTMAN") command prompt (i.e., shell utility, console, etc.) useful for providing an interface
for a user to view and change environmental variables for the CDC tool.
[0016] A user may run the CTMAN command with several different switches (i.e., options)
to view and change environmental variables for the CDC tool. The command CTMAN may be
run with a - 1switch to generate a file "changeTableObject.lst" containing the schema name,
source table name, source table synonym, and unique index columns of the source tables for
which change tables exist. The command CTMAN may be run with the - s switch to add a single
change table. The other arguments available with the - s switch are schema name and synonym
name of the source table for which the change table is to be created. A change table may be
created following the naming convention "DNLD__CT". Once the change
table is created, the command prompt may display a related message on the screen and a success
log may be created.
[0017] A user may use the - F switch to add multiple change tables in a single command.
This option requires a file name as an argument. The file corresponding to the file name should
be a simple flat file containing the list of schema names and table synonym names for the source
tables on which change tables are required. The schema and synonym name should be pipe ('|')
delimited and there should be one line per change table in the file. A user may use the - r switch
to remove or drop a change table. This utility requires the schema name and synonym name of
the source table for which the corresponding change table needs to be dropped as an argument.
[0018] Of course, alternative embodiments may utilize alternative third party CDC tools or
may implement their own CDC tool. Embodiments may employ a CDC tool configured to
capture data logged after each data manipulation language ("DML") transaction. By capturing
data directly from each DML transaction minimal latency may be involved. The change data
may then be stored in a change data dataset 115 as shown in FIG. 1.
[0019] Change data dataset 115 may reside in the source database. For example, in the
Asynchronous Hotlog Mode of ORACLE™ CDC, the change data dataset may include plural
tables of change data corresponding to application dataset tables and both the change data dataset
and the application dataset may reside in the application database. The Asynchronous Hotlog
Mode only requires a single database, however because downstream processes may be
performed on the change data dataset, embodiments may utilize additional resources in the
application database, thereby slowing DML transactions in the application dataset.
[0020] By way of alternative example, in the Asynchronous Autolog Mode of ORACLE™
CDC, the change data dataset may include plural tables of change data corresponding to
application dataset tables and the change data dataset may reside in a staging database separate
from the application database. This may allow downstream resource usage to be limited to the
staging database, thereby avoiding use of resources on the application database that may have
slowed down DML transactions in the application dataset.
[0021] Embodiments may utilize CDC tools that capture change data after every DML
transaction is performed in the application dataset. This may be less resource intensive than
conventional timestamp CDC tools. Additionally, CDC tools that capture change data after every
DML transaction may increase data integrity, for example by avoiding instances where
simultaneous transactions in the application database have identical timestamps, thus potentially
confusing a conventional CDC tool. However, alternative embodiments may utilize timestamp
based CDC tools, version based CDC tools, status indicator based CDC tools, and the like.
[0022] Referring again to FIG. 1, an incremental download generator 120 may generate an
intermediate table 130 including data from change data dataset 115 according to a download
specification file 125. The download specification file 125 defines metadata information for
downloads from the change data dataset 115 to form the intermediate table 130. Embodiments
may allow a user to manually write a download specification file 125 or may provide a tool (e.g.,
a MICROSOFT™ EXCEL™ based tool) to assist a user to generate the download specification
file 125. A download specification tool may create a specification file while discarding potential
errors by validating factors such as field lengths, alphanumeric or special characters, order of
sections and fields with proper delimiters, total number of fields per section, and the like.
[0023] The download specification file 125 may be required to conform to a pre-specified
format comprising delimited (e.g., pipe delimited) columns. FIG. 3 illustrates an exemplary
download specification file. The section indicator column indicates the section which each
record in the various columns belongs to.
[0024] The HDR section may correspond to header information for the download, such as
the download name, download description, and whether the download is customizable or not.
The header information may be the first record in the download specification file and may appear
only once in the file. In some embodiments, the header information may be a mandatory section
of the specification file requiring character delimited information setting forth the download
name, download description, and custom required flag. As the name suggests, the download
name column may give the name of the download. The value may be limited to alphanumeric
characters (i.e., no spaces or symbols) and may be limited to a maximum length (e.g., no more
than 30 characters). However, exceptions to the rule may be made, for example an underscore
(i.e., '_') may be an allowed character despite a rule against symbols. The download description
column may provide a brief business description of the download. The download description
may allow for alphanumeric characters and spaces but may also be limited to a maximum length
(e.g., no more than 100 characters). The custom required flag may indicate whether the
download requires a field which cannot be directly retrieved from a table and instead has to be
derived using some logic or fields from multiple tables.
[0025] The FMT section may define different formats for the fields. Different interfaces
requiring similar data or similar sets of fields as part of a download often require different
formats or sequences of the data. The FMT section may define plural formats (e.g., formats 1,
2,..., 10 are shown in FIG. 3). The FMT section may be an optional section and the incremental
download generator 120 of FIG. 1 may be configured to download all fields in the order of their
occurrence in the download specification file as a default.
[0026] One or more FLD sections may define details of the fields to be downloaded. The
download specification may have as many FLD sections as the number of fields to be
downloaded from the change data dataset. Accordingly, embodiments may require at least one
FLD section in a download specification file. Embodiments may also define a maximum number
of FLD sections that a download specification document may contain, for example a 150 field
maximum. The FLD section may include a schema name, a synonym name, a table column, a
field name, a display format, a field type, and a field size. The following describes these portions
of the FLD section in greater detail.
[0027] The schema name in the FLD section may be the name of the owner schema for the
table which is to be enabled for download. In some instances, a user may want a NULL field at
certain positions in the downloaded record, for example to cater to third-party system
requirements to interpret each record in a certain format. In such cases, the schema name may be
given as a DUMMY and no other column in the FLD section may be filld in. Later, in the
download file (e.g., download file 150 of FIG. 1 discussed below), a blank space may be present
in the position where DUMMY was given in the download specification file.
[0028] The synonym name in the FLD section may store the table abbreviation for a table.
Embodiments may require a synonym name so that the incremental download generator may
recognize the correct table. The table column may identify the column which needs to be
downloaded. The table column may be required to exactly match the column name of the table in
the change data dataset.
[0029] The field name may be a user defined name for the field. For example, while "acid"
may be the application name for account identifier (i.e., the first two letters of each word),
external systems may not understand this nomenclature and may rather require the field name to
be "account_ID". The field name may define such a name for the field. There may be various
constraints on field names, for example they may not be allowed to contain spaces and may be
limited to a maximum number of characters (e.g., 30).
[0030] The display format field may be used to alter the formatting of column data for the
download file. For example, the date format in the application dataset and change data dataset
may have the format DD-MM-YYYY HH24: MI: SS, but a third party application might be
configured to process dates in another format. In such cases, the display format column may
define a format function for the field. However, embodiments may be limited to pre-defined or
user defined format functions applicable to a single table field.
[0031] The field type field may be provided when a display format is specified. This field
stores the return value type of the display format function. For example, values for this field may
be INT, FLOAT, LONG, DOUBLE, DATE, STRING, and CHAR. The field size field may be
given for STRING type fields to know the length in characters for the STRING which will be
returned.
[0032] The REL section may set forth the relationship between the tables involved in
creating a single entity. This section provides the information of the where clause in the query
based on which the selection of changed data after joining the tables happens. Embodiments may
limit the length of the relationship clause, for example the relationship clause may be limited to a
maximum of 4000 characters in length. Embodiments may allow a user to enter the relationship
clause on plural lines to increase readability of the download specification file, for example by
adding plural REL lines and continuing the query separated by a character (e.g., a pipe). For
single table downloads, the relationship clause may indicate " 1 = 1".
[0033] The KEY section may correspond to the fields which would form the unique entity.
The fields provided in this section, when combined as a group, will give the unique record for
the entity. The fields given in the KEY section may or may not be a downloadable field (i.e., a
field specified in a FLD section) and hence the column "Download Required Flag" may be
given. Table columns provided in the KEY section may be the key fields of the tables based on
which the join is created in the where clause.
[0034] The REF section may be an optional section used in case one or more tables involved
in the relationship clause contain no fields to be downloaded. In such an instance, the REF
section indicator corresponds to 'reference' section to provide the details of the tables being
referenced.
[0035] The following illustrates an exemplary incremental download specification file
defining an incremental term deposit account download including the fields foracid,
scheme_code, maturity date, maturity amount, and eff available amt. Note that
eff available amt is a field that cannot be directly derived from the database.
SAMPLE DOWNLOAD SPECIFICATION FILE
HDR|TERMDEP_DNLD|Deposit Accounts Download|10|Y
FLD| TBAADM|GAM|foracid|ACCT_ID
FLD|TBAADM|GAM|scheme_code|Product
FLD|TBAADM|TAM|maturity_date|MAT_DATE|TO_CHAR(MAT_DA
TE,'DD/MM/YYYY')|STRING| 10
FLD|TBAADM|TAM|maturity_amount|maturity_amount
REL|GAM.acid = TAM.acid and GAM.bankjd = TAM.bank id
REL|and GAM.pb_ps_code = 'S'
KEY|TBAADM|GAM|foracid|ACCT_ID|Y
The above exemplary incremental download specification does not include a field corresponding
to eff available amt because the field may involve some calculations and transformation of data
from other fields to derive its value. Rather, the Custom Required Flag in column three of the
header (i.e., HDR) line is set to Y (i.e., yes) thereby indicating that custom logic may be
implemented to derive one or more fields. The custom logic may provide additional flexibility
for including in a download file one or more fields derived from data stored in a plurality of
tables across an entity.
[0036] Referring again to FIG. 1, the incremental download generator 120 may receive the
download specification file 125. The incremental download generator 120 may then validate the
download specification file 125, generate one or more standard query language ("SQL") queries,
fetch data from the change data dataset 115, and store the data in intermediate table 130.
Exemplary functionality of incremental download generator 120 is described in greater detail
with reference to FIG. 4 which illustrates an exemplary process flow performed by the
incremental download generator.
[0037] At step 405, the incremental download generator may receive the download
specification file defining all fields to be captured from plural tables, the formatting of the fields,
the relationships between the tables, and the like. At step 410, the incremental download
generator may interpret all sections of the download specification file and determine whether the
specification file is valid. This step may include section level validation such as determining
whether the data in the download specification file complies with download specification file
requirements, for example the download specification file described above. If the specification
file validation determines that the format of the download specification file is invalid, at step 415
an error message may be returned to the user notifying them of one or more reasons why the
download specification file is invalid. The user may be permitted to transmit a revised download
specification file correcting any deficiencies, causing process flow 400 to start again at step 405.
If the specification file validation is successful, the incremental download generator may proceed
to step 420 and perform a table and field level validation.
[0038] The table and field level validation in step 420 may include determining whether all
tables and fields identified in the download specification file are present in the change data
dataset and that the incremental download generator has sufficient access to each table and field.
If one or more tables or fields are not present in the change data dataset, at step 425 the
incremental download generator may determine whether the fields are present in the application
dataset. If the fields or tables are present in the application dataset but not in the change data
dataset, the incremental download generator may transmit a request to the change data capture
application requesting that change data corresponding to the one or more additional tables or
fields be captured to the change data dataset at step 425. Alternatively, if the tables and fields
identified in the download specification file are also not in the application dataset, at step 415 an
error may be returned indicating so.
[0039] If the tables and fields are determined to be in the change data dataset, either by
successful validation at step 420 or by adding any necessary fields to the change data capture at
step 430, at step 435 the incremental download generator may generate one or more SQL
queries. For example, the incremental download generator may generate a SQL query to fetch
data from the change data dataset based on the download specification file and store the fetched
data in the intermediate table. The incremental download generator may also generate additional
SQL queries, for example queries to select specific fields in specific formats and sequences from
the intermediate table to generate flat download files (discussed in greater detail below in
relation to download file 150 of FIG. 1). The queries may additionally order and format fields as
specified in the download specification file.
[0040] At step 440, the incremental download generator may use a SQL query to fetch data
from the change data dataset corresponding to the download specification document and store
the data in an intermediate table. If an intermediate table already exists having the name
specified in the download specification file, the incremental download generator may append all
new change data to the end of the intermediate table. The intermediate table may be a superset of
all data from the change data dataset that may be required by downstream processes, for example
all data required for generating plural flat download files. The intermediate table may store data
that spans across multiple tables in the change data dataset collated to form a single logical
entity. The intermediate table may additionally store multiple copies of the same data fields in
varying formats and sequences as specified in the download specification file.
[0041] An exemplary intermediate table may have the following structure. All fields which
have been identified in the one or more KEY sections of the download specification file may
constitute the initial columns in the intermediate table. For example, if there are three fields in
the KEY section, the first three columns of the intermediate table may be those three fields. The
following columns of the intermediate table may include all fields which are part of the FLD
sections. These fields may be formatted and sequenced as specified in the download specification
file. Additionally, if the customization required flag is set to "yes" for the download, then two
additional columns, such as CUSTOM_FLD_l and CUSTOM_FLD_2 of length 1000 characters
each, may be part of the intermediate table definition. Finally, the last column of the intermediate
table may store a download time (e.g., DNLD TIME) which may be a field having a date type.
This column may be updated with the date and time when incremental data is extracted from the
change data dataset and the incremental data is inserted in the intermediate table. An index field,
for example named _IDX, may also be created for this field.
[0042] Referring again to FIG. 1, the intermediate table 130 may be stored in an application
database storing application dataset 105. This maybe true whether or not the change data dataset
115 is stored in the same database as the application dataset 105. In alternative embodiments, the
intermediate table 1 0 may be stored in an alternative database, for example a staging database,
or on a remote system or device. Whether the intermediate table 130 is stored in an application
database or a storing database may be determined, for example, based on the mode of a CDC
tool being used.
[0043] A download data to flat file ("DDFF") process 135 may run one or more queries (e.g.,
queries generated by the incremental download generator 120) to extract data from the
intermediate table 130 and to create a download file 150 for each query. Different download files
may be created for different downstream processes, systems, sub-systems, and the like that may
each require different data fields in different formats and sequences. As the name of DDFF
process 135 implies, each download file 140 may store incremental data for a logical entity as a
single flat file which may include data collated from multiple tables and derived fields based on
data stored in multiple tables.
[0044] DDFF process 135 may include an application programming interface ("API") to
allow subscribing processes and systems to download the download files 140. DDFF process 135
may keep track of the time when each subscriber downloads their download file 140 and may
purge the subscriber's respective download file 140 after the subscriber downloads their
download file 140. In other words, a schedule may be maintained that keeps track of when each
subscriber downloads a download file 140 and the system may purge the data in the data file 140
once the subscriber has pulled out or accessed that data. In this fashion, each download file 140
may provide its respective subscriber with only incremental data each time the subscriber
downloads their data. Thus, a subscriber may download their download file 140 on any schedule
and receive incremental data since their last download. Additionally, this may allow the
incremental download framework to provide incremental data without monitoring timestamps or
other scheduling information from each respective subscriber.
[0045] Further, all or portions of the data in intermediate table 130 may be purged once the
data has been downloaded by all necessary subscribers. In other word, data may be maintained in
intermediate table 130 only from the last time a subscriber downloaded the data. In this fashion,
the intermediate table 130 may only store incremental data that has not yet been consumed,
thereby minimizing resource consumption of the framework.
[0046] Data flow 100 may also include a cleanup process 155. Cleanup process 155 may be
configured to purge change data from the individual tables of the change data dataset 115 once
the data may no longer be needed by incremental download generator 120. Cleanup process 155
may, for example, be triggered by incremental download generator 120 to purge data received
from change data dataset 115 as soon as the data is stored in the intermediate table 130.
Alternatively, cleanup process 155 may run periodically to purge data that has been stored in
change data dataset 115 for at least a threshold time period (e.g., purge all data more than three
days old).
[0047] These embodiments may be implemented with software, for example modules
executed on computing devices such as computing device 510 of FIG. 5. Of course, modules
described herein illustrate various functionalities and do not limit the structure of any
embodiments. Rather the functionality of various modules may be divided differently and
performed by more or fewer modules according to various design considerations.
[0048] Computing device 510 has one or more processing device 5 11 designed to process
instructions, for example computer readable instructions (i.e., code) stored on a storage device
513. By processing instructions, processing device 511 may perform the steps and functions
disclosed herein. Storage device 513 may be any type of storage device (e.g., an optical storage
device, a magnetic storage device, a solid state storage device, etc.), for example a non-transitory
storage device. Alternatively, instructions may be stored in one or more remote storage devices,
for example storage devices accessed over a network or the internet. Computing device 510
additionally may have memory 512, an input controller 516, and an output controller 515. A bus
514 may operatively couple components of computing device 510, including processor 511,
memory 512, storage device 513, input controller 516, output controller 515, and any other
devices (e.g., network controllers, sound controllers, etc.). Output controller 515 may be
operatively coupled (e.g., via a wired or wireless connection) to a display device 520 (e.g., a
monitor, television, mobile device screen, touch-display, etc.) in such a fashion that output
controller 515 can transform the display on display device 520 (e.g., in response to modules
executed). Input controller 516 may be operatively coupled (e.g., via a wired or wireless
connection) to input device 530 (e.g., mouse, keyboard, touch-pad, scroll-ball, touch-display,
etc.) in such a fashion that input can be received from a user.
[0049] Of course, FIG. 5 illustrates computing device 510, display device 520, and input
device 530 as separate devices for ease of identification only. Computing device 510, display
device 520, and input device 530 may be separate devices (e.g., a personal computer connected
by wires to a monitor and mouse), may be integrated in a single device (e.g., a mobile device
with a touch-display, such as a smartphone or a tablet), or any combination of devices (e.g., a
computing device operatively coupled to a touch-screen display device, a plurality of computing
devices attached to a single display device and input device, etc.). Computing device 510 may be
one or more servers, for example a farm of networked servers, a clustered server environment, or
a cloud network of computing devices.
[0050] While exemplary descriptions of embodiments provided herein generally relate to
financial institutions, alternative embodiments may be useful to for extracting incremental data
from any system having logical entities including fields spanning plural tables.
[0051] Embodiments have been disclosed herein. However, various modifications can be
made without departing from the scope of the embodiments as defined by the appended claims
and legal equivalents.
What is Claimed is:
1. A computer-implemented method executed by one or more computing devices for
providing incremental downloads, the method comprising:
receiving, by at least one of the one or more computing devices, a download specification
file identifying a plurality of fields in a plurality of tables of a change data dataset to download, a
relationship between the plurality of tables, and parameters for a download file;
extracting, by at least one of the one or more computing devices, the change data
identified in the download specification file from the plurality of tables in the change data dataset
to an intermediate table;
storing, by at least one of the one or more computing devices, the plurality of fields from
the plurality of tables in the intermediate table; and
generating, by at least one of the one or more computing devices, a download file
including the change data identified in the download specification file formatted and sequenced
according to the set of parameters for the download file.
2. The method of claim 1, further comprising:
receiving a set of change data capture parameters;
capturing change data from an application dataset; and
storing the captured change data in the change data dataset.
3. The method of claim 2, wherein the application dataset and the change dataset reside in a
single database.
4. The method of claim 2, wherein the application dataset and the change dataset reside in
separate databases.
5. The method of claim 1, wherein the download file is a delimited flat file comprising
incremental data from the plural fields in the plural tables of the change data dataset.
6. The method of claim 5, wherein the download file includes at least one field derived from
data stored in fields from at least two of the plurality of tables.
7. The method of claim 1, wherein the step of storing the plurality of fields from the
plurality of tables in the intermediate table includes collating the fields according to the
relationship between the plurality of tables to form a single logical entity.
8. The method of claim 1, wherein the step of generating the download file includes purging
obsolete data from the intermediate table.
9. The method of claim 1, further comprising purging obsolete data from the change data
dataset.
10. The method of claim 1, further comprising providing a user interface configured to assist
a user to generate the download specification file.
11. A system for providing an incremental download framework comprising:
a memory; and
a processor operatively coupled to the memory, the processor configured to perform the
steps of:
receiving a download specification file identifying a plurality of fields in a
plurality of tables of a change data dataset to download, a relationship between the
plurality of tables, and parameters for a download file;
extracting the change data identified in the download specification file from the
plurality of tables in the change data dataset to an intermediate table;
storing the plurality of fields from the plurality of tables in the intermediate table;
and
generating a download file including the change data identified in the download
specification file formatted and sequenced according to the set of parameters for the
download file.
12. The system of claim 11, the processor further configured to perform the steps of:
receiving a set of change data capture parameters;
capturing change data from an application dataset; and
storing the captured change data in the change data dataset.
13. The system of claim 11, wherein the download file is a delimited flat file comprising
incremental data from the plural fields in the plural tables of the change data dataset,
wherein the download file includes at least one field derived from data stored in fields
from at least two of the plurality of tables, and
wherein the step of storing the plurality of fields from the plurality of tables in the
intermediate table includes collating the fields according to the relationship between the plurality
of tables to form a single logical entity.
14. The system of claim 11, the processor further configured to perform the step of purging
obsolete data from the change data dataset, wherein the step of generating the download file
includes purging obsolete data from the intermediate table.
15. The system of claim 11, the processor further configured to perform the step of providing
a user interface configured to assist a user to generate the download specification file.
16. Computer-readable code stored on a non-transitory computer-readable medium that,
when executed by a computing device, performs a method for providing an incremental
download framework, the method comprising:
receiving a download specification file identifying a plurality of fields in a plurality of
tables of a change data dataset to download, a relationship between the plurality of tables, and
parameters for a download file;
extracting the change data identified in the download specification file from the plurality
of tables in the change data dataset to an intermediate table;
storing the plurality of fields from the plurality of tables in the intermediate table; and
generating a download file including the change data identified in the download
specification file formatted and sequenced according to the set of parameters for the download
file.
17. The computer-readable medium of claim 16, the method further comprising:
receiving a set of change data capture parameters;
capturing change data from an application dataset; and
storing the captured change data in the change data dataset.
18. The computer-readable medium of claim 16, wherein the download file is a delimited flat
file comprising incremental data from the plural fields in the plural tables of the change data
dataset,
wherein the download file includes at least one field derived from data stored in fields
from at least two of the plurality of tables, and
wherein the step of storing the plurality of fields from the plurality of tables in the
intermediate table includes collating the fields according to the relationship between the plurality
of tables to form a single logical entity.
19. The computer-readable medium of claim 16, the method further comprising purging
obsolete data from the change data dataset, wherein the step of generating the download file
includes purging obsolete data from the intermediate table.
20. The computer-readable medium of claim 16, the method further comprising providing a
user interface configured to assist a user to generate the download specification file.