DESC:
FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003

COMPLETE SPECIFICATION
(See section 10 and rule 13)
1. TITLE OF THE INVENTION
SYSTEM AND METHOD FOR FILTERING DATA
2. APPLICANT(S)
NAME NATIONALITY ADDRESS
JIO PLATFORMS LIMITED INDIAN OFFICE-101, SAFFRON, NR. CENTRE POINT, PANCHWATI 5 RASTA, AMBAWADI, AHMEDABAD 380006, GUJARAT, INDIA
3.PREAMBLE TO THE DESCRIPTION

THE FOLLOWING SPECIFICATION PARTICULARLY DESCRIBES THE NATURE OF THIS INVENTION AND THE MANNER IN WHICH IT IS TO BE PERFORMED.

FIELD OF THE INVENTION
[0001] The present invention relates to the field of wireless communication networks, more particularly relates to a system and a method for filtering data.
BACKGROUND OF THE INVENTION
[0002] In modern data analytics, data frames are one of the most common data structures used as they are a flexible and intuitive way of storing and working with data. Before conducting any data analysis, it is important to clean the data by identifying, fixing, and removing any inaccuracies or inconsistencies from a data set. This process forms the crucial part of data analysis. Some data rows may contain data with high values or low values which affects the Machine Learning (ML) training and better results are not obtained from the data. In conventional technologies, if there are repeated values in the data frame then it will remove those repeated values. Sometimes the repeated values can be there only in one or more columns and the rows that contain those repeated values are important for data analysis. Thus, in such conventional technologies, on that column value is repeated, those rows are also being removed which leads to loss of the data.
[0003] Therefore, there is a need for a system and a method that provides improvement in data accuracy and efficiency for filtering one or more rows for performing data analysis.
SUMMARY OF THE INVENTION
[0004] One or more embodiments of the present disclosure provide a method and a system for filtering data.
[0005] In one aspect of the present invention, the method for filtering the data is disclosed. The method includes the step of retrieving, by one or more processors, data from one or more data sources. The method includes the step of receiving, by the one or more processors, an input corresponding to one or more columns in a data frame, a column value and a value range from one of a User Interface (UI) and a User Equipment (UE). The method includes the step of comparing, by the one or more processors, column values of the one or more columns of the received data with the column values and the value range of the one or more columns as per the received input. The method includes the step of removing, by the one or more processors, one or more rows in the data frame if the column values of the one or more columns of the received data matches with the column values and the value range of the one or more columns as per the received input based on the comparison. The method includes the step of updating, by the one or more processors, the data frame on removing the one or more rows.
[0006] In one embodiment, the one or more data sources include at least, a file input, a source path, an input stream, a Hyper Text Transfer Protocol 2 (HTTP 2), a Hadoop Distributed File System (HDFS), and a Network Attached Storage (NAS).
[0007] In another embodiment, the retrieved data is stored in the data frame. The one or more removed rows are extracted from the data frame.
[0008] In yet another embodiment, the updated data frame is stored in a storage unit and used for Machine Learning (ML) training.
[0009] In yet another embodiment, the step of receiving, by the one or more processors, an input corresponding to one or more columns in a data frame, a column value and a value range from one of a User Interface (UI) and a User Equipment (UE), includes the step of creating a custom criteria via the UI by a user. The step of receiving, by the one or more processors, an input corresponding to one or more columns in a data frame, a column value and a value range from one of a User Interface (UI) and a User Equipment (UE), includes the step of applying complex logical operations to remove the data based on one or more columns.
[0010] In yet another embodiment, the column value and the value range are an integer value.
[0011] In yet another embodiment, the method further includes the step of maintaining a history associated with the one or more rows removal thereby allowing a user to revert removal of the one or more rows.
[0012] In another aspect of the present invention, the system for filtering of the data is disclosed. The system includes an integrated unit configured to retrieve data from one or more data sources. The system includes a selection unit configured to receive an input corresponding to one or more columns in a data frame, a column value and a value range from one of a User Interface (UI) and a User Equipment (UE). The system includes a pre-processing unit configured to compare column values of the one or more columns of the received data with the column values and the value range of the one or more columns as per the received input. The system includes a pre-processing unit configured to remove one or more rows in the data frame if the column values of the one or more columns of the received data matches with the column values and the value range of the one or more columns as per the received input based on the comparison. The system includes a data source unit configured to update, the data frame on removing the one or more rows.
[0013] In yet another embodiment, a User Equipment (UE) is disclosed. One or more primary processors are communicatively coupled to the one or more processors. The one or more primary processors coupled with a memory unit. The memory unit stores instructions which when executed by the one or more primary processors cause the UE to select an input corresponding to one or more columns in a data frame, a column value and a value range, and create a custom criteria via the User Interface (UI) by a user.
[0014] Other features and aspects of this invention will be apparent from the following description and the accompanying drawings. The features and advantages described in this summary and in the following detailed description are not all-inclusive, and particularly, many additional features and advantages will be apparent to one of ordinary skill in the relevant art, in view of the drawings, specification, and claims hereof. Moreover, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes and may not have been selected to delineate or circumscribe the inventive subject matter, resort to the claims being necessary to determine such inventive subject matter.
BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The accompanying drawings, which are incorporated herein, and constitute a part of this disclosure, illustrate exemplary embodiments of the disclosed methods and systems in which like reference numerals refer to the same parts throughout the different drawings. Components in the drawings are not necessarily to scale, emphasis instead being placed upon clearly illustrating the principles of the present disclosure. Some drawings may indicate the components using block diagrams and may not represent the internal circuitry of each component. It will be appreciated by those skilled in the art that disclosure of such drawings includes disclosure of electrical components, electronic components or circuitry commonly used to implement such components.
[0016] FIG. 1 is an exemplary block diagram of an environment for filtering data, according to one or more embodiments of the present disclosure;
[0017] FIG. 2 is an exemplary block diagram of a system for filtering the data, according to the one or more embodiments of the present disclosure;
[0018] FIG. 3 is a schematic representation of the system 120 in which various entities operations are explained, according to one or more embodiments of the present disclosure;
[0019] FIG. 4 is a block diagram of an architecture that can be implemented in the system of FIG.2, according to the one or more embodiments of the present disclosure;
[0020] FIG. 5 is a signal flow diagram illustrating for filtering the data, according to the one or more embodiments of the present disclosure; and
[0021] FIG. 6 is a flow diagram illustrating a method for filtering the data, according to the one or more embodiments of the present disclosure.
[0022] The foregoing shall be more apparent from the following detailed description of the invention.
DETAILED DESCRIPTION OF THE INVENTION
[0023] Some embodiments of the present disclosure, illustrating all its features, will now be discussed in detail. It must also be noted that as used herein and in the appended claims, the singular forms "a", "an" and "the" include plural references unless the context clearly dictates otherwise.
[0024] Various modifications to the embodiment will be readily apparent to those skilled in the art and the generic principles herein may be applied to other embodiments. However, one of ordinary skill in the art will readily recognize that the present disclosure including the definitions listed here below are not intended to be limited to the embodiments illustrated but is to be accorded the widest scope consistent with the principles and features described herein.
[0025] A person of ordinary skill in the art will readily ascertain that the illustrated steps detailed in the figures and here below are set out to explain the exemplary embodiments shown, and it should be anticipated that ongoing technological development will change the manner in which particular functions are performed. These examples are presented herein for purposes of illustration, and not limitation. Further, the boundaries of the functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternative boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed. Alternatives (including equivalents, extensions, variations, deviations, etc., of those described herein) will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein. Such alternatives fall within the scope and spirit of the disclosed embodiments.
[0026] The present disclosure provides a system and a method for filtering data based on a specified column value or range and updating the data frame with the filtered result. In an embodiment, the present disclosure provides a system that is configured to perform Machine Learning (ML) training for the given data extracted from a data source. The system allows filtering of the one or more rows based on a specified column value which is restricted to integers and updating the data frame with the filtered result. In this way, data cleaning is performed which generates high-quality, consistent data that is used for detailed analysis, which helps in achieving improvement in data accuracy and efficiency.
[0027] Referring to FIG. 1, FIG. 1 illustrates an exemplary block diagram of an environment 100 for filtering data, according to one or more embodiments of the present invention. The environment 100 includes a network 105, a User Equipment (UE) 110, a server 115, and a system 120. The UE 110 aids a user to interact with the system 120 for filtering the data. In an embodiment, the user is at least one of, a network operator, and a service provider. The filtering data refers to the process of selecting a subset of data from a larger dataset based on specified criteria or conditions. The filtered data allows the user to focus on relevant data, making it easier to analyze, visualize, or derive insights without being overwhelmed by irrelevant data.
[0028] For the purpose of description and explanation, the description will be explained with respect to the UE 110, or to be more specific will be explained with respect to a first UE 110a, a second UE 110b, and a third UE 110c, and should nowhere be construed as limiting the scope of the present disclosure. Each of the UE 110 from the first UE 110a, the second UE 110b, and the third UE 110c is configured to connect to the server 115 via the network 105. In an embodiment, each of the first UE 110a, the second UE 110b, and the third UE 110c is one of, but not limited to, any electrical, electronic, electro-mechanical or an equipment and a combination of one or more of the above devices such as smartphones, virtual reality (VR) devices, augmented reality (AR) devices, laptop, a general-purpose computer, desktop, personal digital assistant, tablet computer, mainframe computer, or any other computing device.
[0029] The network 105 includes, by way of example but not limitation, one or more of a wireless network, a wired network, an internet, an intranet, a public network, a private network, a packet-switched network, a circuit-switched network, an ad hoc network, an infrastructure network, a Public-Switched Telephone Network (PSTN), a cable network, a cellular network, a satellite network, a fiber optic network, or some combination thereof. The network 105 may include, but is not limited to, a Third Generation (3G), a Fourth Generation (4G), a Fifth Generation (5G), a Sixth Generation (6G), a New Radio (NR), a Narrow Band Internet of Things (NB-IoT), an Open Radio Access Network (O-RAN), and the like.
[0030] The server 115 may include by way of example but not limitation, one or more of a standalone server, a server blade, a server rack, a bank of servers, a server farm, hardware supporting a part of a cloud service or system, a home server, hardware running a virtualized server, one or more processors executing code to function as a server, one or more machines performing server-side functionality as described herein, at least a portion of any of the above, some combination thereof. In an embodiment, the entity may include, but is not limited to, a vendor, a network operator, a company, an organization, a university, a lab facility, a business enterprise, a defense facility, or any other facility that provides content.
[0031] The environment 100 further includes the system 120 communicably coupled to the server 115 and each of the first UE 110a, the second UE 110b, and the third UE 110c via the network 105. The system 120 is configured for filtering the data. The system 120 is adapted to be embedded within the server 115 or is embedded as the individual entity, as per multiple embodiments of the present invention.
[0032] Operational and construction features of the system 120 will be explained in detail with respect to the following figures.
[0033] FIG. 2 is an exemplary block diagram of a system 120 for filtering the data, according to one or more embodiments of the present disclosure.
[0034] The system 120 includes a processor 205, a memory 210, a user interface 215, and a storage unit 245. For the purpose of description and explanation, the description will be explained with respect to one or more processors 205, or to be more specific will be explained with respect to the processor 205 and should nowhere be construed as limiting the scope of the present disclosure. The one or more processor 205, hereinafter referred to as the processor 205 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, single board computers, and/or any devices that manipulate signals based on operational instructions.
[0035] As per the illustrated embodiment, the processor 205 is configured to fetch and execute computer-readable instructions stored in the memory 210. The memory 210 may be configured to store one or more computer-readable instructions or routines in a non-transitory computer-readable storage medium, which may be fetched and executed to create or share data packets over a network service. The memory 210 may include any non-transitory storage device including, for example, volatile memory such as RAM, or non-volatile memory such as EPROM, flash memory, and the like.
[0036] The User Interface (UI) 215 includes a variety of interfaces, for example, interfaces for a Graphical User Interface (GUI), a web user interface, a Command Line Interface (CLI), and the like. The user interface 215 facilitates communication of the system 120. In one embodiment, the user interface 215 provides a communication pathway for one or more components of the system 120. Examples of the one or more components include, but are not limited to, the UE 110, and the storage unit 245. The term “storage unit” and “database” are used interchangeably hereinafter, without limiting the scope of the disclosure.
[0037] The storage unit 245 is one of, but not limited to, a centralized database, a cloud-based database, a commercial database, an open-source database, a distributed database, an end-user database, a graphical database, a No-Structured Query Language (NoSQL) database, an object-oriented database, a personal database, an in-memory database, a document-based database, a time series database, a wide column database, a key value database, a search database, a cache databases, and so forth. The foregoing examples of storage unit 245 types are non-limiting and may not be mutually exclusive e.g., a database can be both commercial and cloud-based, or both relational and open-source, etc.
[0038] Further, the processor 205, in an embodiment, may be implemented as a combination of hardware and programming (for example, programmable instructions) to implement one or more functionalities of the processor 205. In the examples described herein, such combinations of hardware and programming may be implemented in several different ways. For example, the programming for the processor 205 may be processor-executable instructions stored on a non-transitory machine-readable storage medium and the hardware for processor 205 may comprise a processing resource (for example, one or more processors), to execute such instructions. In the present examples, the memory 210 may store instructions that, when executed by the processing resource, implement the processor 205. In such examples, the system 120 may comprise the memory 210 storing the instructions and the processing resource to execute the instructions, or the memory 210 may be separate but accessible to the system 120 and the processing resource. In other examples, the processor 205 may be implemented by electronic circuitry.
[0039] In order for the system 120 to filter the data, the processor 205 includes an integrated unit 220, a selection unit 225, a pre-processing unit 230, and a data source unit 235communicably coupled to each other. In an embodiment, operations and functionalities of the integrated unit 220, the selection unit 225, the pre-processing unit 230, and the data source unit 235can be used in combination or interchangeably.
[0040] The integrated unit 220 is configured to retrieve the data from one or more data sources. In an embodiment, the one or more data sources include at least a file input, a source path, an input stream, a Hyper Text Transfer Protocol 2 (HTTP 2), a Hadoop Distributed File System (HDFS), and a Network Attached Storage (NAS). The file input refers to reading data from the files stored locally or on the server 115. The files can be in different formats, including, but not limited to, Comma Separated Values (CSV), JavaScript Object Notation (JSON), extensible Markup Language (XML), or text files. In an exemplary embodiment, the data is stored in the CSV file, and the integrated unit 220 fetches the data for processing. The integrated unit 220 retrieves the data from the file and loads the data into the memory 210 for further processing.
[0041] The source path typically refers to the directory or network location where the data files are stored. The integrated unit 220 fetches the data by following the provided file path. In an exemplary embodiment for the source path, the system 120 stores images in a specific directory. The integrated unit 220 navigates to a designated source path and retrieves all files that match the required criteria (e.g., .jpg images). The input stream refers to continuous data that is read in real-time from a stream of data (e.g., data being transmitted over the network 105 or generated by sensors). In an exemplary embodiment, the data is being received from an Application Programming Interface (API) or a live data stream, and the integrated unit 220 fetches it in real-time. The HTTP 2 is a protocol used for communication over the web, which improves upon HTTP/1.1 by offering multiplexing and better performance for handling multiple requests. In an exemplary embodiment, the integrated unit 220 retrieves the data from the web server using the HTTP 2. The integrated unit 220 uses HTTP 2 to fetch the data from remote web servers or APIs.
[0042] The HDFS is a distributed file system used to store large datasets across multiple machines. The HDFS is commonly used in big data environments to store and retrieve large amounts of data. The integrated unit 220 connects to the HDFS to retrieve the files for processing. The NAS is a dedicated file storage system that provides Local Area Network (LAN) access to the data. The NAS allows multiple users or systems to access the data from a centralized storage device. The integrated unit 220 fetches the data from a NAS device over the network 105. In an exemplary embodiment, if the data is stored on the NAS, the integrated unit 220 fetches the data via network protocols. Upon retrieving the data from the one or more data sources, the retrieved data is stored in a data frame for further processing.
[0043] Upon retrieving the data from the one or more data sources, the retrieved data is stored in the data frame, which facilitates easy analysis and manipulation. The data frame is a structured format, typically resembling a table that includes rows and columns. The rows represent individual records, and the columns represent attributes or features of the data. Upon storing the retrieved data in the data frame, the selection unit 225 is configured to receive an input from the user via one of the UI 215 and the UE 110. In an embodiment, the received input includes, but not limited to, one or more columns in the data frame, a column value and a value range. The one or more columns in the data frame refers to the attributes or features within a structured dataset that the user can select for data analysis. The column value refers to the specific data entry within a particular column for a given row in the dataset. The value range refers to a specified interval of acceptable values for a given column. The value range helps to filter or analyse the data, which allows the users to focus on specific subsets of data that fall within the defined boundaries. In an embodiment, the column value and the value range are an integer value. In another embodiment, the column value and the value range are defined by the user.
[0044] Further, the selection unit 225 is configured to create a custom criteria via the UI 215 by the user. based on the applied complex logical operations. This flexibility extends to using user-defined functions for dynamic filtering, empowering users to specify the custom criteria based on complex calculations or conditions. The selection unit 225 is configured to apply complex logical operations to remove the data based on the one or more columns by using AI/ML algorithms to analyze data patterns. The one or more columns are combined by using various logical operations. In an embodiment, the logical operations include, but not limited to, AND, OR, XOR, NOT, NOR, and NAND logic, allowing for intricate data selection scenarios. In an exemplary embodiment, enabling filtering for employees who are either relatively recent hires with a salary exceeding a threshold value or older employees earning above a specified amount by selecting the one or more rows. The one or more rows includes, but not limited to, age details, salary details, and joining year. In one exemplary embodiment, the selection of the one or more rows includes age > 30 AND salary < 50,000, allowing the users to focus on individuals who meet both criteria. In another exemplary embodiment, the selection of the one or more rows includes age > 30 AND joining year > 2015 OR salary > 40,000.
[0045] Upon receiving the input corresponding to the one or more columns in the data frame, the pre-processing unit 230 is configured to compare the column values of the one or more columns of the received data with the column value and the value range of the one or more columns as per the received input. The pre-processing unit 230 analyses the column values of the one or more columns in the data frame. Upon analyzing the column values of the one or more columns, the pre-processing unit 230 evaluates each column values of the one or more columns with the column value and the value range of the one or more columns as per the received input. The pre-processing unit 230 compares the column value of the one or more columns with the user defined column value. If any of the column values match, then the corresponding rows are extracted, and the data frame is updated. In another embodiment, the pre-processing unit 230 compares the one or more columns with the value range of the one or more columns. The pre-processing unit 230 extracts the one or more rows, where the values in the one or more columns fall within the value range and updates the data frame accordingly.
[0046] Further, the pre-processing unit 230 performs the logical operations on the selected one or more columns to update the data frame by using the AI/ML algorithms to facilitate deeper analysis and tailored data management. The preprocessing stage includes deduplication (e.g., removing duplicate entries based on an ID column), handling missing values (e.g., filling missing entries with the median value using imputation algorithms such as K-Nearest Neighbours), converting data types (e.g., changing the column from string to datetime format), and detecting outliers (e.g., identifying values outside of 1.5 times the interquartile range using statistical methods such as Tukey method.
[0047] The pre-processing unit 230 automatically profiles the data using AI/ML techniques such as clustering algorithms (e.g., K-Means) and statistical analysis to generate comprehensive summary statistics during preprocessing, including measures such as mean, median, mode, and standard deviation, which provide a thorough understanding of the dataset. The user interface 215 displays the insights into data distribution that can be visualized through histograms or box plots (e.g., depicting the distribution of age), while skewness metrics help identify if the distribution is right-skewed or left-skewed, which is crucial for identifying potential biases or anomalies before further cleaning or filtering. The user can preview the filtered results in real-time and display a sample of the filtered one or more rows to provide immediate feedback on the impact of transformations.
[0048] Furthermore, the pre-processing unit 230 maintains a history associated with the one or more rows removal thereby allowing a user to revert removal of the one or more rows, enabling users to seamlessly revert to previous transformation steps, ensuring flexibility during the preprocessing phase. In an embodiment, the history is stored in the storage unit 245. The pre-processing unit 230 tracking the one or more rows removing operation by using advanced machine learning algorithms. In an embodiment, the advanced machine learning algorithms include, but not limited to, Long Short-Term Memory (LSTM) networks or recurrent neural networks (RNNs), thereby ensuring full auditability and transparency throughout the data transformation process.
[0049] Based on the comparison, the pre-processing unit 230 is configured to remove the one or more rows in the data frame if the column values of the one or more columns of the received data matches with the column value and the value range of the one or more columns as per the received input. The pre-processing unit 230 updates the data frame by selecting the one or more rows that matches with the column value and the value range of the one or more columns based on the comparisons made by the pre-processing unit 230. In an exemplary embodiment, in a network performance dataset, the input of the one or more columns refers as latency, the input for the column value refers as 30, and the input for the value range refers as a minimum value of 15 and a maximum value of 50. The pre-processing unit 230 evaluates each row in the one or more rows with the dataset. Consider, the latency for router1 is 20, the latency for router2 is 10, and the latency for router3 is 50. The latency for router1compares with the column value and the value range of the one or more columns, the latency for router1 match with the column value and the value range of the one or more columns. The latency for router2 compares with the column value and the value range of the one or more columns, the latency for router2 does not match the column value and the value range of the one or more columns. The one or more rows of the router2 will be removed, which does not match within the column value and the value range of the one or more columns. The latency for router3 compares with the column value and the value range of the one or more columns, the latency for router3 match with the column value and the value range of the one or more columns. The filtering unit 235 successfully selects and outputs the one or more rows of the router1 and the router 3, which meets the user-defined value for the latency. The functionality of the filtering unit 235 is essential for data analysis, allowing the users to concentrate on relevant data, facilitating better decision-making and performance evaluation.
[0050] Upon removing the one or more rows in the data frame, the data source unit 235 is configured to update the data frame on removing the one or more rows. The data source unit 235 is responsible for modifying the data frame after the filtering operation is performed. The data source unit 235 ensures that the data frame reflects the current state of the data based on the filtration of the one or more rows. After the filtering process, the data source unit 235 modifies the data frame to either retain only the removed one or more rows (which is relevant rows). In one embodiment, the data source unit 235 creates a new data frame that contains only these removed rows, effectively removing any rows that do not match with the column value and the value range of the one or more columns.
[0051] Upon updating the data frame on removing the one or more rows, the updated data frame is stored in the storage unit 245 and used for Machine Learning (ML) training unit 410 (as shown in FIG.4). Upon storing the updated data frame, also the raw data is retained for ML training. The updated data is performed to enhance the data by filtering the one or more, which helps Artificial Intelligence/Machine Learning (AI/ML) models to understand the structure and learn the patterns at a more granular level. The filtered one or more rows are provided to the AI/ML model training, which enhances the data clarity by removing the inconsistent or irrelevant data in the dataset. The updated data helps to generate more accurate and contextually appropriate responses by the AI/ML model.
[0052] By filtering the one or more rows, the system 120 removes the noise from extra rows and data. Data cleaning is performed using filtration of the one or more rows based on the column values of the one or more columns, which reduces the number of unique data present in the data source, removing the variations or inconsistent data. Further, the system 120 updates the data frame with the filtered result and produces high-quality, consistent data that is used for detailed analysis. By completion of the filtration of the one or more rows, the processor 205 improves data accuracy and efficiency, processing speed, and reduces requirement of memory space.
[0053] FIG. 3 is a schematic representation of the system 120 in which various entities operations are explained, according to one or more embodiments of the present disclosure. Referring to FIG. 3, it describes the system 120 for filtering the data. It is to be noted that the embodiment with respect to FIG. 3 will be explained with respect to the first UE 110a for the purpose of description and illustration and should nowhere be construed as limited to the scope of the present disclosure.
[0054] As mentioned earlier in FIG.1, In an embodiment, the first UE 110a may encompass electronic apparatuses. These devices are illustrative of, but not restricted to, personal computers, laptops, tablets, smartphones (including phones), or other devices enabled for web connectivity. The scope of the first UE 110a explicitly extends to a broad spectrum of electronic devices capable of executing computing operations and accessing networked resources, thereby providing users with a versatile range of functionalities for both personal and professional applications. This embodiment acknowledges the evolving nature of electronic devices and their integral role in facilitating access to digital services and platforms. In an embodiment, the first UE 110a can be associated with multiple users. Each UE 110 is communicatively coupled with the processor 205 via the network 105.
[0055] The first UE 110a includes one or more primary processors 305 communicably coupled to the one or more processors 205 of the system 120. The one or more primary processors 305 are coupled with a memory unit 310 storing instructions which are executed by the one or more primary processors 305. Execution of the stored instructions by the one or more primary processors 305 enables the first UE 110a to select the input corresponding to the one or more columns in the data frame, the column value and the value range, and create the custom criteria via the User Interface (UI) 215 by the user.
[0056] Furthermore, the one or more primary processors 305 within the UE 110 are uniquely configured to execute a series of steps as described herein. This configuration underscores the processor 205 capability to filter the data. The operational synergy between the one or more primary processors 305 and the additional processors, guided by the executable instructions stored in the memory unit 310, facilitates a seamless filtration of the data.
[0057] As mentioned earlier in FIG.2, the system 120 includes the one or more processors 205, the memory 210, and the user interface 215. The operations and functions of the one or more processors 205, the memory 210, and the user interface 215 are already explained in FIG. 2. For the sake of brevity, a similar description related to the working and operation of the system 120 as illustrated in FIG. 2 has been omitted to avoid repetition.
[0058] Further, the processor 205 includes the integrated unit 220, the selection unit 225, the pre-processing unit 230, and the data source unit 235. The operations and functions of the integrated unit 220, the selection unit 225, the pre-processing unit 230, and the data source unit 235 are already explained in FIG. 2. Hence, for the sake of brevity, a similar description related to the working and operation of the system 120 as illustrated in FIG. 2 has been omitted to avoid repetition. The limited description provided for the system 120 in FIG. 3, should be read with the description provided for the system 120 in the FIG. 2 above, and should not be construed as limiting the scope of the present disclosure.
[0059] FIG. 4 is a block diagram of an architecture 300 that can be implemented in the system of FIG.2, according to one or more embodiments of the present disclosure. The architecture 300 of the system 120 includes an integrated unit 220, a load balancer 405, and the processor 205. The processor 205 includes a selection unit 225, a pre-processing unit 230, a data source unit 235, and a machine learning training unit 410.
[0060] The architecture 300 of the system 120 is configured to interact with the integrated unit 220 and the load balancer 405. The integrated unit 220 is configured to access the relevant data in the network 105 and is capable of interacting with the server 115, the storage unit 245 to collect the data. In an embodiment, the integrated unit 220 includes, but not limited to, the one or more data sources, from where the data can be retrieved. In an embodiment, the data is retrieved as the file input, the source path, the input stream, the HTTP2, the DFS and the NAS.
[0061] The load balancer 405 includes distributing the one or more data sources request traffic across the one or more processors 205. The distribution of the one or more data source request traffic helps in managing and optimizing the workload, ensuring that no single processor is overwhelmed while improving overall system performance and reliability.
[0062] Upon retrieving the data from the one or more data sources, the retrieved data is stored in the data frame, which facilitates easy analysis and manipulation. Upon storing the retrieved data in the data frame, the selection unit 220 is configured to receive the input from the user via one of the UI 215 and the UE 110. In an embodiment, the received input includes, but not limited to, one or more columns in the data frame, the column value and the value range. In an embodiment, the column value and the value range are an integer value. In another embodiment, the column value and the value range are defined by the user.
[0063] Upon receiving the input corresponding to the one or more columns in the data frame, the pre-processing unit 230 is configured to compare the column values of the one or more columns of the received data with the column value and the value range of the one or more columns as per the received input. The pre-processing unit 230 analyses the column values of the one or more columns in the data frame. Upon analyzing the column values of the one or more columns, the pre-processing unit 230 evaluates each column value against the column value and the value range of the one or more columns as per the received input. The pre-processing unit 230 compares the column value of the one or more columns compared with the column value. If any of the column values match, then the corresponding rows are extracted, and the data frame is updated. In another embodiment, the pre-processing unit 230 compares the rows with the value range. The pre-processing unit 230 extracts one or more rows, where the values in one or more columns fall within the value range and updates the data frame accordingly.
[0064] Based on the comparison, the pre-processing unit 230 is configured to filter the one or more rows in the data frame if the column values of the one or more columns of the received data matches with the column value and the value range of the one or more columns as per the received input. The pre-processing unit 230 updates the data frame by selecting the one or more rows that matches with the column value and the value range of the one or more columns based on the comparisons.
[0065] Upon removing the one or more rows in the data frame, the data source unit 235 is configured to store the removed one or more rows and update the data frame. The data source is updated after the preprocessing, and the updated data frame is stored. The ML training unit 410 is configured to train ML model by using the filtered one or more rows from the data source and applies various machine learning algorithms to the preprocessed data to create predictive or analytical models. Based on the nature of the data, different ML algorithms include, but not limited to, linear regression, decision trees, neural networks, clustering algorithms, etc.
[0066] FIG. 5 is a signal flow diagram illustrating for filtering the data, according to one or more embodiments of the present disclosure.
[0067] At 505, Initially, the request is transmitted by the user via the UI 215 and the UE 110 for cleaning and filtering the data from the data frame by filtering the one or more rows.
[0068] At 510, upon receiving the request, the integrated unit 220 is configured to retrieve the data from the one or more data sources. In an embodiment, the one or more data sources include at least, the file input, the source path, the input stream, the HTTP 2, the HDFS, and the NAS. Upon retrieving the data, the retrieved data is stored in the data frame for further processing.
[0069] At 515, upon retrieving the data and stored in the data frame, the retrieved data is stored in the data frame, which facilitates easy analysis and manipulation. Upon storing the retrieved data in the data frame, the selection unit 225 is configured to receive the input from the user via one of the UI 215 and the UE 110. In an embodiment, the received input includes, but not limited to, one or more columns in the data frame, the column value and the value range. In an embodiment, the column value and the value range are an integer value. In another embodiment, the column value and the value range are defined by the user.
[0070] At 520, upon receiving the input corresponding to the one or more columns in the data frame, the pre-processing unit 230 is configured to compare the column values of the one or more columns of the received data with the column value and the value range of the one or more columns as per the received input. The pre-processing unit 230 analyses the column values of the one or more columns in the data frame. Upon analyzing the column values of the one or more columns, the pre-processing unit 230 evaluates each column value against the column value and the value range of the one or more columns as per the received input. The pre-processing unit 230 compares the column value of the one or more columns compared with the column value. If any of the column values match, then the corresponding rows are extracted, and the data frame is updated. In another embodiment, the pre-processing unit 230 compares the rows with the value range. The pre-processing unit 230 extracts one or more rows, where the values in one or more columns fall within the value range and updates the data frame accordingly.
[0071] Based on the comparison, the pre-processing unit 230 is configured to filter the one or more rows in the data frame if the column values of the one or more columns of the received data matches with the column value and the value range of the one or more columns as per the received input. The pre-processing unit 230 updates the data frame by selecting the one or more rows that matches with the column value and the value range of the one or more columns based on the comparisons.
[0072] At 525, upon removed the one or more rows in the data frame, the data source unit 235 is configured to store the removed one or more rows and update the data frame. The data source is updated after the preprocessing, and the updated data frame is stored. The ML training unit 410 is configured to train ML model by using the filtered one or more rows from the data source and applies various machine learning algorithms to the preprocessed data to create predictive or analytical models. Based on the nature of the data, different ML algorithms include, but are not limited to, linear regression, decision trees, neural networks, clustering algorithms, etc.
[0073] Upon pre-processing the filtered data, the pre-processed data is stored in the storage unit 245 and used for Machine Learning (ML) training unit 410 to train the data. The preprocessing is performed to enhance the data by filtering the one or more rows, which helps Artificial Intelligence/Machine Learning (AI/ML) models to understand the structure and learn the patterns at a more granular level. The filtered one or more rows are provided to the AI/ML model training, which ensures to identify the patterns and helps in generating more accurate and contextually appropriate responses.
[0074] FIG. 6 is a flow diagram illustrating a method 600 for filtering the data, according to one or more embodiments of the present disclosure.
[0075] At step 605, the method 600 includes the step of retrieving the data from the one or more data sources by the integrated unit 220. In an embodiment, the one or more data sources include at least, the file input, the source path, the input stream, the Hyper Text Transfer Protocol 2 (HTTP 2), the Distributed File System (DFS), and the Network Attached Storage (NAS). The file input refers to reading data from the files stored locally or on the server 115. The files can be in different formats, including, but not limited to, the Comma Separated Values (CSV), the JavaScript Object Notation (JSON), the extensible Markup Language (XML), or the text files. In an exemplary embodiment, the data is stored in the CSV file, and the integrated unit 220 fetches the data for processing. The integrated unit 220 retrieves the data from the file and loads it into the memory 210 for further processing.
[0076] At step 610, the method 600 includes the step of receiving the input from the user via one of the UI 215 and the UE 110 by the selection unit 225. In an embodiment, the received input includes, but not limited to, one or more columns in the data frame, the column value and the value range. The one or more columns in the data frame refers to the attributes or features within the structured dataset that the user can select for data analysis. The column value refers to the specific data entry within the particular column for the given row in the dataset. The value range refers to a specified interval of acceptable values for the given column. The value range helps to filter or analyse the data, which allows the users to focus on specific subsets of data that fall within the defined boundaries. In an embodiment, the column value and the value range are the integer value. In another embodiment, the column value and the value range are defined by the user.
[0077] At step 615, the method 600 includes the step of comparing the column values of the one or more columns of the received data with the column value and the value range of the one or more columns as per the received input by the pre-processing unit 230. The pre-processing unit 230 analyses the column values of the one or more columns in the data frame. Upon analyzing the column values of the one or more columns, the pre-processing unit 230 evaluates each column value against the column value and the value range of the one or more columns as per the received input. The pre-processing unit 230 compares the column value of the one or more columns compared with the column value. If any of the column values match, then the corresponding rows are extracted, and the data frame is updated. In another embodiment, the pre-processing unit 230 compares the rows with the value range. The pre-processing unit 230 extracts one or more rows, where the values in one or more columns fall within the value range and updates the data frame accordingly.
[0078] At step 615, the method 600 includes the step of removing the one or more rows in the data frame if the column values of the one or more columns of the received data matches with the column value and the value range of the one or more columns as per the received input by the pre-processing unit 230. The pre-processing unit 230 updates the data frame by selecting the one or more rows that matches with the column value and the value range of the one or more columns based on the comparisons made by the pre-processing unit 230.
[0079] Upon removing the one or more rows in the data frame, the data source unit 235 is configured to update the data frame on removing the one or more rows. The data source unit 235 is responsible for modifying the data frame after the filtering operation is performed. The data source unit 235 ensures that the data frame reflects the current state of the data based on the removed one or more rows. After the removing process, the data source unit 235 modifies the data frame to either retain only the removed one or more rows (which is relevant rows). In one embodiment, the data source unit 235 creates a new data frame that contains only these removed one or more rows, effectively removing any rows that do not match with the column value and the value range of the one or more columns.
[0080] Upon updating the data frame on removing the one or more rows, the updated data frame is stored in the storage unit 245 and used for Machine Learning (ML) training. Upon storing the updated data frame, also the raw data is retained for ML training. The updated data is performed to enhance the data by filtering the one or more, which helps Artificial Intelligence/Machine Learning (AI/ML) models to understand the structure and learn the patterns at a more granular level. The filtered one or more rows are provided to the AI/ML model training, which enhances the data clarity by removing the inconsistent or irrelevant data in the dataset. The updated data helps to generate more accurate and contextually appropriate responses by the AI/ML model.
[0081] By removing the one or more rows, the system 120 removes the noise from extra rows and data. Data cleaning is performed using filtration of the one or more rows based on the column values of the one or more columns, which reduces the number of unique data present in the data source, removing the variations or inconsistent data. Further, the system 120 updates the data frame with the filtered result and produces high-quality, consistent data that is used for detailed analysis. By completion of the removal of the one or more rows, the processor 205 improves data accuracy and efficiency, processing speed, and reduces requirement of memory space.
[0082] In another aspect of the embodiment, a non-transitory computer-readable medium having stored thereon computer-readable instructions that, when executed by a processor 205. The processor 205 is configured to retrieve data from one or more data sources. The processor 205 is configured to receive an input corresponding to one or more columns in a data frame, a column value and a value range from one of a User Interface (UI) and a User Equipment (UE). The processor 205 is configured to compare column values of the one or more columns of the received data with the column values and the value range of the one or more columns as per the received input. The processor 205 is configured to remove one or more rows in the data frame if the column values of the one or more columns of the received data matches with the column values and the value range of the one or more columns as per the received input based on the comparison. The processor 205 is configured to update the data frame on removing the one or more rows.
[0083] A person of ordinary skill in the art will readily ascertain that the illustrated embodiments and steps in description and drawings (FIGS.1-5) are set out to explain the exemplary embodiments shown, and it should be anticipated that ongoing technological development will change the manner in which particular functions are performed. These examples are presented herein for purposes of illustration, and not limitation. Further, the boundaries of the functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternative boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed. Alternatives (including equivalents, extensions, variations, deviations, etc., of those described herein) will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein. Such alternatives fall within the scope and spirit of the disclosed embodiments.
[0084] The present disclosure provides technical advancement for filtering the one or more rows and updating the data frame with the filtered data, which is applied to ML models for training data that aid in desired analysis. The trained data is used for efficient information retrieval and knowledge extraction as well as enhancement of the effectiveness of the system. The present disclosure acquires the relevant data which is further given for the ML training thus preventing loss of data, enhances the data clarity by deleting irrelevant data which helps the AI model to generate more accurate and contextually appropriate responses and leads to faster data analysis and processing.
[0085] The present invention offers multiple advantages over the prior art and the above listed are a few examples to emphasize on some of the advantageous features. The listed advantages are to be read in a non-limiting manner.

REFERENCE NUMERALS

[0086] Environment - 100
[0087] Network-105
[0088] User equipment- 110
[0089] Server - 115
[0090] System -120
[0091] Processor - 205
[0092] Memory - 210
[0093] User interface-215
[0094] Integrated unit – 220
[0095] Selection unit – 225
[0096] Pre-processing unit- 230
[0097] Data source unit- 235
[0098] Storage unit-245
[0099] Load balancer- 405
[00100] ML training unit- 410
,CLAIMS:CLAIMS
We Claim:
1. A method (500) of filtering of data, the method (500) comprising the steps of:
retrieving, by one or more processors (205), data from one or more data sources;
receiving, by the one or more processors (205), an input corresponding to one or more columns in a data frame, a column value and a value range from one of a User Interface (UI) (215) and a User Equipment (UE) (110);
comparing, by the one or more processors (205), column values of the one or more columns of the received data with the column values and the value range of the one or more columns as per the received input;
removing, by the one or more processors (205), one or more rows in the data frame if the column values of the one or more columns of the received data matches with the column values and the value range of the one or more columns as per the received input based on the comparison; and
updating, by the one or more processors (205), the data frame on removing the one or more rows.

2. The method (500) as claimed in claim 1, wherein the one or more data sources include at least, a file input, a source path, an input stream, a Hyper Text Transfer Protocol 2 (HTTP 2), a Distributed File System (DFS), and a Network Attached Storage (NAS).

3. The method (500) as claimed in claim 1, wherein the retrieved data is stored in the data frame, wherein on removing, the one or more removed rows are extracted from the data frame.

4. The method (500) as claimed in claim 1, wherein the updated data frame is stored in a storage unit (245) and used for Machine Learning (ML) training.

5. The method (500) as claimed in claim 1, wherein the step of receiving, by the one or more processors (205), an input corresponding to one or more columns in a data frame, a column value and a value range from one of a User Interface (UI) (215) and a User Equipment (UE) (110), comprising the step of:
creating a custom criteria via the UI (215) by a user; and
applying complex logical operations to remove the data based on one or more columns.

6. The method (500) as claimed in claim 1, wherein the column value and the value range are an integer value.

7. The method (500) as claimed in claim 1, wherein the method (500) further comprises the steps of:
maintaining a history associated with the one or more rows removal thereby allowing a user to revert removal of the one or more rows.

8. A system (120) for filtering of data, the system (120) comprises:
an integrated unit (220) configured to retrieve, data from one or more data sources;
a selection unit (225) configured to receive, an input corresponding to one or more columns in a data frame, a column value and a value range from one of a User Interface (UI) (215) and a User Equipment (UE) (110);
a pre-processing unit (230) configured to compare, column values of the one or more columns of the received data with the column value and the value range of the one or more columns as per the received input;
the pre-processing unit (230) configured to remove, one or more rows in the data frame if the column values of the one or more columns of the received data matches with the column values and the value range of the one or more columns as per the received input based on the comparison; and
a data source unit (235) configured to update, the data frame on removing the one or more rows.

9. The system (120) as claimed in claim 8, wherein the one or more data sources include at least, a file input, a source path, an input stream, a Hyper Text Transfer Protocol 2 (HTTP 2), a Distributed File System (DFS), and a Network Attached Storage (NAS).

10. The system (120) as claimed in claim 8, wherein the retrieved data is stored in the data frame, wherein on removing, the one or more removed rows are extracted from the data frame.

11. The system (120) as claimed in claim 8, wherein the updated data frame is stored in a storage unit (245) and used for Machine Learning (ML) training.

12. The system (120) as claimed in claim 8, wherein the selection unit (225) is further configured to:
create a custom criteria via the UI (215) by a user; and
apply complex logical operations to remove the data based on one or more columns.

13. The system (120) as claimed in claim 78 wherein the column value and the value range are an integer value.

14. The system (120) as claimed in claim 8, wherein the system (120) further comprises the:
maintaining a history associated with the one or more rows removal thereby allowing a user to revert removal of the one or more rows.

15. A User Equipment (UE) (110), comprising:
one or more primary processors (305) communicatively coupled to the one or more processors (205), the one or more primary processors (305) coupled with a memory unit (310), wherein said memory unit (310) stores instructions which when executed by the one or more primary processors (305) cause the UE (110) to:
select an input corresponding to one or more columns in a data frame, a column value and a value range; and
create a custom criteria via the User Interface (UI) (215) by a user.
wherein the one or more processors (205) are configured to perform the steps as claimed in claim 1.