DESC:FIELD OF THE INVENTION

The present disclosure relates to a system and method for smart storage and retrieval of data in data flow/tasks. More specifically, the system is configured to reduce the redundancy of data storage required and ensure the security of stored data.

BACKGROUND OF THE INVENTION

Today all businesses are data-driven. The availability of appropriate data at the appropriate time leads to the appropriate decisions. Artificial Intelligence (AI) and Machine Learning (ML) based workloads require a massive amount of data to train and build the models. The data is required to be used continuously to keep these models running continuously.

AI depends on the capacity of processing and save big amounts of data. However, it increases the burden on data storage and processing power. Rubiscape is a user-centric platform. It is designed from the ground-up level. The platform entitles Extract, Transformation, and Load (ETL) and Extract, Load, and Transformation (ELT) types of processing. Data needed by the application is stored in the cache during the processing.

Big data and its storage reduce the speed and optimization of the processor and the computing machine. In current applications for data orchestration between tasks, data is directly streamed, or entire data is passed through the cache. The entire data in the cache passes to the next task, which increases data redundancy.

Because of the tremendous data load, only powerful computers will be able to perform the processes required. When big data is generated, the application’s data workloads and performance are affected. In many cases, it is required to store more data than the capacity available. The success of doing so can lead to an improvement in data retrieval performance. The idea is to use intermediate storage/cache during data processing.

In one prior art solution, the processing is done by passing on all the data to all the transactions. This increases the need for resources and processing time. The various transaction stages are denoted by T1, T2, and T3. It also shows Data Blocks denoted by D1, D2, D3, D4, D1’, D2’, and D3’. The input of Transaction T1 is D1, D2, D3, and D4. Transaction T1 produces the output data blocks as D1’, D2, D3, and D4. Where D1’ is the data updated by transaction T1. Here the blocks of data D2, D3, and D4 are unchanged. Conventionally, all these blocks of data are passed on to T2. This means the input of T2 is D1’, D2, D3, and D4. The output of T2 is D2’. Here the blocks of data D1’, D3, and D4 are unchanged. Conventionally all these blocks of data are passed on to T3. This means the input of T3 is D1’, D2’, D3, and D4. The output of T3 is D3’. Here the blocks of data D1’, D2’, and D4 are unchanged. In other words, some data blocks that are passed on are not processed by all the transactions. But still, those data blocks are passed on a stored again and again.

Some transactions maintain the data structure intact. Some transactions irreversibly change the data structure. For example, they delete duplicate data, or convert from one format to another, filtering the data, and so on. If these transactions can be identified, it will become easy to eliminate data duplication.

However, there are a few limitations to the conventional approaches i.e., 1) due to the size of the data, more resources are required to process and store the input and output data blocks at every stage, 2) The processing power required is also more, and hence overall process performance is reduced. This happens for every transaction, 3) If there are 50 transactions, this will have a compounded effect. The total wastage of resources, processing power, and time will depend on the amount of data processed at the respective transaction. In view of the foregoing discussion, it is clearly portrayed that there is a need to have a system and method for smart storage and retrieval of data in data flow/tasks.

SUMMARY OF THE INVENTION

The present disclosure seeks to provide a smart storage and retrieval system and a method for reducing the redundancy of data storage and managing the data complexity and processing speed.

In an embodiment, a system for smart storage and retrieval of data in data flow/tasks is disclosed. The system includes an input unit coupled to a pre-processor for maintaining a record of the sequence of transactions/tasks. The system further includes a cloud memory coupled to a central processor comprising instructions executable by the central processor, wherein the central processor is operable when executing the instructions to perform. The system further includes a carry-forward flag (CF) for differentiating the transactions/tasks, wherein the value of CF is set to 1 when a transaction processes the data such that the transaction is directly correlated to the input and the value of CF is set to 0 when a transaction processes the data such that the transaction cannot be directly correlated to the input. The system further includes a control unit for optimizing transactional data storage when the data structure is maintained intact for CF to be 1.

In an embodiment, a method for smart storage and retrieval of data in data flow/tasks is disclosed. The method includes maintaining a record of the sequence of transactions/tasks by a pre-processor using an input unit. The method further includes differentiating the transactions/tasks by a carry-forward flag (CF), wherein the value of CF is set to 1 when a transaction processes the data such that the transaction is directly correlated to the input and the value of CF is set to 0 when a transaction processes the data such that the transaction cannot be directly correlated to the input. The method further includes optimizing transactional data storage when the data structure is maintained intact for CF to be 1 using a control unit.

An object of the present disclosure is to reduce the redundancy of data storage required.

Another object of the present disclosure is to help in managing the data complexity and processing speed.

Yet another object of the present invention is to deliver an expeditious and cost-effective system that ensures the security of stored data.

To further clarify the advantages and features of the present disclosure, a more particular description of the invention will be rendered by reference to specific embodiments thereof, which is illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope. The invention will be described and explained with additional specificity and detail in the accompanying drawings.

BRIEF DESCRIPTION OF FIGURES

These and other features, aspects, and advantages of the present disclosure will become better understood when the following detailed description is read with reference to the accompanying drawings in which like characters represent like parts throughout the drawings, wherein:

Figure 1 illustrates a block diagram of a system for smart storage and retrieval of data in data flow/tasks in accordance with an embodiment of the present disclosure;
Figure 2 illustrates a flow chart of a method for smart storage and retrieval of data in data flow/tasks in accordance with an embodiment of the present disclosure;
Figure 3 illustrates a schematic diagram of transactional data storage optimization in accordance with an embodiment of the present disclosure;
Figure 4 illustrates a schematic diagram of beginning-of-process and end-of-process decryption & encryption in accordance with an embodiment of the present disclosure; and
Figure 5 illustrates a schematic diagram of beginning-of-transaction and end-of-transaction decryption & encryption in accordance with an embodiment of the present disclosure.

Further, skilled artisans will appreciate that elements in the drawings are illustrated for simplicity and may not have necessarily been drawn to scale. For example, the flow charts illustrate the method in terms of the most prominent steps involved to help to improve understanding of aspects of the present disclosure. Furthermore, in terms of the construction of the device, one or more components of the device may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details that are pertinent to understanding the embodiments of the present disclosure so as not to obscure the drawings with details that will be readily apparent to those of ordinary skill in the art having benefit of the description herein.

DETAILED DESCRIPTION:

For the purpose of promoting an understanding of the principles of the invention, reference will now be made to the embodiment illustrated in the drawings and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the invention is thereby intended, such alterations and further modifications in the illustrated system, and such further applications of the principles of the invention as illustrated therein being contemplated as would normally occur to one skilled in the art to which the invention relates.

It will be understood by those skilled in the art that the foregoing general description and the following detailed description are exemplary and explanatory of the invention and are not intended to be restrictive thereof.

Reference throughout this specification to “an aspect”, “another aspect” or similar language means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, appearances of the phrase “in an embodiment”, “in another embodiment” and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment.

The terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process or method that comprises a list of steps does not include only those steps but may include other steps not expressly listed or inherent to such process or method. Similarly, one or more devices or sub-systems or elements or structures or components proceeded by "comprises...a" does not, without more constraints, preclude the existence of other devices or other sub-systems or other elements or other structures or other components or additional devices or additional sub-systems or additional elements or additional structures or additional components.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The system, methods, and examples provided herein are illustrative only and not intended to be limiting.

Embodiments of the present disclosure will be described below in detail with reference to the accompanying drawings.

Referring to Figure 1, a block diagram of a system for smart storage and retrieval of data in data flow/tasks is illustrated in accordance with an embodiment of the present disclosure. The system 100 includes an input unit 102 coupled to a pre-processor 110 for maintaining a record of the sequence of transactions/tasks.

In an embodiment, a cloud memory 112 is coupled to a central processor 114 comprising instructions executable by the central processor, wherein the central processor is operable when executing the instructions to perform,

In an embodiment, a carry-forward flag (CF) 104 is connected to the input unit 102 for differentiating the transactions/tasks, wherein the value of CF 104 is set to 1 when a transaction processes the data such that the transaction is directly correlated to the input and the value of CF 104 is set to 0 when a transaction processes the data such that the transaction cannot be directly correlated to the input.

In an embodiment, a control unit 106 is connected to the carry-forward flag 104 for optimizing transactional data storage when the data structure is maintained intact for CF 104 to be 1.

In another embodiment, the transactions use column-wise data and one or more of the transactions maintain the data structure intact using the carry-forward flag (CF) 104.

In another embodiment, one or more of the transactions irreversibly change the data structure to delete duplicate data, or convert from one format to another, filtering the data, and so on, wherein the transactional data storage optimization Rubiscape platform 108 helps in managing the data complexity, and processing speed.

In another embodiment, the processing is performed by storing only modified data in the cache 116, wherein the process uses a recursive technique.

In another embodiment, the data blocks D1, D2, D3, and D4 are passed only to a first transaction stage (T1), wherein the output of the first transaction stage, wherein only D2 is passed to a second transaction stage (T2), and output is D2’ and input to a third transaction stage (T3) will be D3, and the output is D3’.

In another embodiment, the other blocks of data are passed only when the task requests, wherein before passing on the data, the value of CF 104 is checked to be 1, and thus, the need to pass, process, and store data that is not needed is eliminated.

In another embodiment, the data cannot be passed on as it cannot be correlated to the subsequent transaction when CF 104 is 0, wherein the types of transactions are known, and the original input data required by its subsequent transactions is separately passed on.

In another embodiment, the data is stored, passed on, and accessed in encrypted form only such that the security of the data is maintained, wherein the encryption and decryption happen at the beginning of the system if the system blocks are integrated, wherein the Rubiscape platform 108 ensures the security of stored data with the help of encryption.

In another embodiment, a distributed system is employed for performing the decryption, and encryption in-between the transactions, wherein the transactional data storage functionality of the Rubiscape reduces the complexity of the data and the functionality ensures secured data connectivity and reduces the redundancy of data.

Figure 2 illustrates a flow chart of a method for smart storage and retrieval of data in data flow/tasks in accordance with an embodiment of the present disclosure. At step 202, method 200 includes maintaining a record of the sequence of transactions/tasks by a pre-processor 110 using an input unit 102.

At step 204, method 200 includes differentiating the transactions/tasks by a carry-forward flag (CF) 104, wherein the value of CF 104 is set to 1 when a transaction processes the data such that the transaction is directly correlated to the input and the value of CF 104 is set to 0 when a transaction processes the data such that the transaction cannot be directly correlated to the input.

At step 206, method 200 includes optimizing transactional data storage when the data structure is maintained intact for CF 104 to be 1 using a control unit 106.

Figure 3 illustrates a schematic diagram of transactional data storage optimization in accordance with an embodiment of the present disclosure. The sequence of transactions/tasks is first noted down. Most transactions use column-wise data. The transactions use column-wise data and one or more of the transactions maintain the data structure intact. Some transactions irreversibly change the data structure. For example, they delete duplicate data, or convert from one format to another, filtering the data, and so on.

To differentiate these transactions, a flag 104 is set up. It is named the Carry-forward flag (CF) 104. When a transaction processes the data in such a way that it can be directly correlated to the input, the value of CF 104 is set to 1. When a transaction processes the data in such a way that it cannot be directly correlated to the input, the value of CF 104 is set to 0. Transactional Data Storage Optimization will only happen when the data structure is maintained intact (CF = 1).

The processing is done by storing only modified data in the cache 116 will be faster. The process uses a recursive technique.

Figure 3 shows that various transactions will only be passed the block of data it needs to process. All blocks of data will be passed only to T1. So, the output of T1 will be D1’. Only D2 will be passed to T2, and its output will be D2’. Similarly, the input to T3 will be D3, and its output will be D3’. Other data blocks will be passed only if the task requests it. Before passing on the data, the value of CF 104 is checked to be 1. Thus, the need to pass, process, and store data that is not needed is eliminated. Note that if CF 104 is 0, that data cannot be passed on as it cannot be correlated to the subsequent transaction. Since these types of transactions are known, the original input data required by its subsequent transactions can be separately passed on.

Since all the data is not passed on to every transaction, the overall speed of the process is considerably improved, and the resources needed are reduced.

Figure 4 illustrates a schematic diagram of beginning-of-process and end-of-process decryption & encryption in accordance with an embodiment of the present disclosure. The data is stored, passed on, and accessed in encrypted form only. Thus, the security of the data will be maintained.

Figure 5 illustrates a schematic diagram of beginning-of-transaction and end-of-transaction decryption & encryption in accordance with an embodiment of the present disclosure. The encryption and decryption will happen at the beginning of the system if the system blocks are integrated.
In a distributed system, decryption and encryption can be done in between the transactions. The transactional data storage functionality of the Rubiscape reduces the complexity of the data. The functionality ensures secured data connectivity and reduces the redundancy of data.
The developed system 100 relates to reducing the redundancy of data storage required. Rubiscape’s data processing is based on this Transactional Data Storage Optimization approach. It reduces the redundancy in the storage of data in the device. The help of Transactional Data Storage Optimization Rubiscape platform 108 helps in managing the data complexity and processing speed. With the help of encryption, it ensures the security of stored data.
As a result of this, even the devices will have lesser storage and computing power that can be used for Big Data/AI/ML applications.
The process reduces the resources needed to store data. So, the use of a cache 116 is optimized. Additional data can be passed on to the subsequent transactional stages when demanded required. The process increases processing speed and productivity. Conditions, where data is altered, are identified through the value of the carry-forward flag 104 (CF = 1). So, there is no confusion caused by requesting input data.
The functional units described in this specification have been labeled as devices. The functional units comprises input unit 102, carry-forward flag (CF) 104, control unit 106, Rubiscape platform 108, and pre-processor 110. A device may be implemented in programmable hardware devices such as processors, digital signal processors, central processing units, field programmable gate arrays, programmable array logic, programmable logic devices, cloud processing systems, or the like. The devices may also be implemented in software for execution by various types of processors. An identified device may include executable code and may, for instance, comprise one or more physical or logical blocks of computer instructions, which may, for instance, be organized as an object, procedure, function, or other construct. Nevertheless, the executable of an identified device need not be physically located together, but may comprise disparate instructions stored in different locations which, when joined logically together, comprise the device and achieve the stated purpose of the device.
Indeed, an executable code of a device or module could be a single instruction, or many instructions, and may even be distributed over several different code segments, among different applications, and across several memory devices. Similarly, operational data may be identified and illustrated herein within the device, and may be embodied in any suitable form and organized within any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed over different locations including over different storage devices, and may exist, at least partially, as electronic signals on a system or network.
In accordance with the exemplary embodiments, the disclosed computer programs or modules can be executed in many exemplary ways, such as an application that is resident in the memory of a device or as a hosted application that is being executed on a server and communicating with the device application or browser via a number of standard protocols, such as TCP/IP, HTTP, XML, SOAP, REST, JSON and other sufficient protocols. The disclosed computer programs can be written in exemplary programming languages that execute from memory on the device or from a hosted server, such as BASIC, COBOL, C, C++, Java, Pascal, or scripting languages such as JavaScript, Python, Ruby, PHP, Perl or other sufficient programming languages.
Some of the disclosed embodiments include or otherwise involve data transfer over a network, such as communicating various inputs or files over the network. The network may include, for example, one or more of the Internet, Wide Area Networks (WANs), Local Area Networks (LANs), analog or digital wired and wireless telephone networks (e.g., a PSTN, Integrated Services Digital Network (ISDN), a cellular network, and Digital Subscriber Line (xDSL)), radio, television, cable, satellite, and/or any other delivery or tunneling mechanism for carrying data. The network may include multiple networks or sub networks, each of which may include, for example, a wired or wireless data pathway. The network may include a circuit-switched voice network, a packet-switched data network, or any other network able to carry electronic communications. For example, the network may include networks based on the Internet protocol (IP) or asynchronous transfer mode (ATM), and may support voice using, for example, VoIP, Voice-over-ATM, or other comparable protocols used for voice data communications. In one implementation, the network includes a cellular telephone network configured to enable exchange of text or SMS messages.
Examples of the network include, but are not limited to, a personal area network (PAN), a storage area network (SAN), a home area network (HAN), a campus area network (CAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a virtual private network (VPN), an enterprise private network (EPN), Internet, a global area network (GAN), and so forth.
The drawings and the forgoing description give examples of embodiments. Those skilled in the art will appreciate that one or more of the described elements may well be combined into a single functional element. Alternatively, certain elements may be split into multiple functional elements. Elements from one embodiment may be added to another embodiment. For example, orders of processes described herein may be changed and are not limited to the manner described herein. Moreover, the actions of any flow diagram need not be implemented in the order shown; nor do all of the acts necessarily need to be performed. Also, those acts that are not dependent on other acts may be performed in parallel with the other acts. The scope of embodiments is by no means limited by these specific examples. Numerous variations, whether explicitly given in the specification or not, such as differences in structure, dimension, and use of material, are possible. The scope of embodiments is at least as broad as given by the following claims.

Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and any component(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature or component of any or all the claims. ,CLAIMS:1. A system for smart storage and retrieval of data in data flow/tasks, the system comprises:
an input unit (102) coupled to a pre-processor (110) for maintaining a record of the sequence of transactions/tasks;
a cloud memory (112) coupled to a central processor (114) comprising instructions executable by the central processor, wherein the central processor is operable when executing the instructions to perform;
a carry-forward flag (CF) (104) for differentiating the transactions/tasks, wherein the value of CF (104) is set to 1 when a transaction processes the data such that the transaction is directly correlated to the input and the value of CF (104) is set to 0 when a transaction processes the data such that the transaction cannot be directly correlated to the input; and
a control unit (106) for optimizing transactional data storage when the data structure is maintained intact for CF (104) to be 1.

2. The system as claimed in claim 1, wherein the transactions use column-wise data and one or more of the transactions maintain the data structure intact using the carry-forward flag (CF) (104).

3. The system as claimed in claim 1, wherein one or more of the transactions irreversibly change the data structure to delete duplicate data, or convert from one format to another, filtering the data, and so on, wherein transactional data storage optimization Rubiscape platform (108) helps in managing the data complexity, and processing speed.

4. The system as claimed in claim 3, wherein the processing is performed by storing only modified data in the cache (116), wherein the process uses a recursive technique.

5. The system as claimed in claim 1, wherein the data blocks D1, D2, D3, and D4 are passed only to a first transaction stage (T1), wherein the output of the first transaction stage, wherein only D2 is passed to a second transaction stage (T2), and output is D2’ and input to a third transaction stage (T3) is D3, and the output is D3’.

6. The system as claimed in claim 1, wherein the other blocks of data are passed only when the task requests, wherein before passing on the data, the value of CF (104) is checked to be 1, and thus, the need to pass, process, and store data that is not needed is eliminated.

7. The system as claimed in claim 6, wherein the data cannot be passed on as it cannot be correlated to the subsequent transaction when CF (104) is 0, wherein the types of transactions are known, the original input data required by its subsequent transactions is separately passed on.

8. The system as claimed in claim 6, wherein the data is stored, passed on, and accessed in encrypted form only such that the security of the data is maintained, wherein the encryption and decryption happen at the beginning of the system if the system blocks are integrated, wherein the Rubiscape platform (108) ensures the security of stored data with the help of encryption.

9. The system as claimed in claim 1, comprises a distributed system for performing the decryption, and encryption in between the transactions, wherein the transactional data storage functionality of the Rubiscape reduces the complexity of the data and the functionality ensures secured data connectivity and reduces the redundancy of data.

10. A method for smart storage and retrieval of data in data flow/tasks, the method comprises:
maintaining a record of the sequence of transactions/tasks by a pre-processor (110) using an input unit (102);
differentiating the transactions/tasks by a carry-forward flag (CF) (104), wherein the value of CF (104) is set to 1 when a transaction processes the data such that the transaction is directly correlated to the input and the value of CF (104) is set to 0 when a transaction processes the data such that the transaction cannot be directly correlated to the input; and
optimizing transactional data storage when the data structure is maintained intact for CF (104) to be 1 using a control unit (106).