Description: The present disclosure, which deals with distributed data processing systems, includes at least one component that deals with intelligence production and the detection of activity from occurrences in a distributed data processing system.
BACKGROUND OF THE INVENTION:
Since the dawn of time, computer network administrators have prioritised recognizing activities, both beneficial and detrimental. It is well-known in the public and commercial sectors of the computer network industry that users can communicate with one another through the use of desktop computers, laptop computers, tablets, smartphones, browsers, and other devices that are connected to a computer network through computers and servers connected to a computer network. Digital data, frequently in the form of data packets, are sent across a network by devices connected.
On the other hand, malicious acts can harm the network's software or hardware, as well as its users. Malicious acts include unauthorized access to network resources and data, followed by the unauthorized use of those resources and data. Malicious actions include the following: If a particular entity's use pattern is abnormal or otherwise different from the expected use pattern, network administrators will look for abnormal or otherwise different behaviour patterns. This could be a specific organization or subset of organizations, an individual user, an IP address, or a node or group of nodes in a network or any other entity they are familiar with.
Unauthorized access to a network is prevented by using security appliances, often seen in well-known systems. According to the appliance technique mentioned above, security appliances (which are often servers or PCs that have been configured to offer security) are placed at one or more locations throughout a network as part of the appliance method. Once installed, the device maintains track of all the data that flows through it while connected to the network. Among other things, the appliance may include functions such as unlawful access detection and illegal use of data in addition to virus detection and intrusion detection. In contrast to security hardware, which can be increased fast to handle temporary or permanent surges in network traffic, security software cannot be scaled as quickly. When network traffic increases, it is common for a security provider to be forced to install a new security appliance or execute an equally time-consuming equipment upgrade due to the increase in traffic.
Consequently, appliances often have only a limited level of network visibility since they are generally configured to monitor just data passing via the connection on which the appliance is installed. It is improbable that an appliance of this kind would be aware of actions taking place on other network segments watched by other appliances. So it won't have the capability of using the extra context information offered by actions happening on other network segments to identify a cleverly-designed piece of malware that would otherwise be impossible to detect from simply localized data.
Data network security is approached differently than in the past, thanks to installed software solutions rather than security hardware appliances. On terminal devices, such as PCs or laptops, these items, such as antivirus or anti-malware software, are most often installed (e.g., desktop and laptop computers, tablets, or smartphones). Installed products monitor the data travelling over the network between the terminal device and the central server for the presence of malware in either the incoming or departing data streams, as determined by the products' configuration. It is unfortunate when existing software solutions do not perform well in terms of scalability or network visibility compared to their potential. Because installed products are often located on terminal devices, their views of data on the network tend to be rather limited in scope. Furthermore, they are often installed on hardware that cannot be easily upgraded.
SUMMARY OF THE INVENTION:
Identifying activity on computer networks, both useful and detrimental, has been a top priority for computer network managers. Computers and servers connected to a computer network are widely used to interact with other users in the public and commercial sectors of the computer network business. Desktop computers, laptop computers, tablets, smartphones, browsers, mobile devices, and other devices are commonly used to interact with other users via computers and servers connected to a computer network. Devices that are linked send digital data, often found in data packets, from one device to another across a network.
If malicious actions are carried out on a network, the potential for damage to the network's software or hardware and its users cannot be overestimated. Unlawful access to network resources and data, followed by the unauthorized use of those resources and data, are all examples of malicious behaviours on networks. The following are examples of malicious behaviour: For any given entity, whether it's a single company or a group of companies, an individual user, a single IP address, an individual node or a group of nodes in a network, or any other entity with which the network administrator is familiar, the administrator will look for patterns of behaviour that are abnormal or otherwise differ from the expected use pattern for that particular entity.
Security appliances are often found in well-known systems, and they are designed to protect the network against unwanted access and usage of information. Appliances (often servers or PCs configured to provide security) are put at one or more locations across a network by the appliance strategy described above. As soon as it is installed, the device begins to keep track of all of the data that passes through it when connected to the internet. For example, the appliance may be able to execute activities such as unlawful access detection, unauthorised data usage detection, virus detection, and intrusion detection, among other things. In contrast to security hardware, which can be scaled up rapidly to address transient or permanent spikes in network traffic, security software cannot be ramped up as quickly. The rise in network traffic often results in a security provider being required to either install a new security appliance or perform an equally time-consuming equipment update due to the increased traffic. As a result, appliances often have only a limited amount of network visibility since they are typically designed to monitor just data passing over the connection on which the appliance is installed. Activities taking place on other network segments monitored by other appliances are unlikely to be seen by this sort of equipment in this situation. A cleverly-designed piece of malware that may be difficult to identify from simply localized data may be unable to use the extra context information offered by actions happening on other network segments due to this limitation.
The approach to data network security used by software solutions placed on a computer rather than security hardware appliances is distinct from the method taken by regular hardware appliances. Items like antivirus and anti-malware software, often found on terminal devices like PCs and laptops, are the most frequently found installed (e.g., desktop and laptop computers, tablets, or smartphones). According to the product's settings, installed products monitor data passing over the network between a terminal device and a central server for evidence of malware in either the incoming or outgoing data streams. The fact that present software solutions do not perform optimally in terms of scalability or network visibility compared to their potential is a sad fact of life. Most deployed items are installed on terminal devices on a local network. Consequently, they have a penchant for having limited perspectives of data on the network due to their position on the network. Furthermore, they are often put on hardware that is not readily upgradeable or replaceable.
BRIEF DESCRIPTION OF THE INVENTION:
An advanced system for abnormal activity detection in networked environments using multiple strategies and principles in ways that are both more insightful and scalable than the usual methods is described here as an example. The security platform uses a range of machine learning algorithms to undertake security evaluations, as will be detailed in further detail below. In addition, the platform is fueled by "big data." User behavioural analytics (UBA) and user/entity behavioural analytics (UEBA) are both capabilities of the security platform described here, and they may both be used to identify previously undisclosed security abnormalities and risks. In addition, the security platform may enable network security managers or analysts to respond and take the necessary action as soon as an anomaly or danger is discovered by providing analytical data with risk ratings and supporting proof. An integrated security platform can identify complex, hidden, or internal threats when paired with the above behavioural analytics approaches. For example, the behaviour analytics in this disclosure do not need any previous knowledge such as known signatures or rules to be applied to the data, which is a key element of this disclosure. The security platform may improve threat detection and response times by using a variety of danger indicators. Aside from providing proof in the context of a kill chain, the security platform also allows for targeted correction of any found anomaly or hazard.
Another benefit of a security platform is that it generates various rank-ordered lists of events that occur in the context of the kill chain, as previously described. Certain scenarios may benefit from deploying kill chains linked from an anomaly or a threat description to the relevant data acquired over time. A SIEM (Security Information and Event Management) solution such as the Splunk® App for Enterprise Security may be used to further scope and disrupt an attack and contain and recover from it.
An example of how the security platform provided here may be used in a general setting. The environment might represent one or more companies or organisations, and it could be used in various parts of the world. To connect, the Internet and one or more wired or wireless networks are employed (e.g. an Internet Protocol (IP)-based LAN, metropolitan area network (MAN), wide area network (WAN), or a Wireless LAN (WLAN) network such as Wireless Fidelity and/or GSM cellular networks). Computers, smartphones, servers, and laptop computers are all examples of computing equipment that may be used to access the information in the environment. One or more components in this context are connected through communication channels. Each computer system listed above may include one or more physical computers and/or additional processing devices. The system setup determines whether or not several wired or wireless networks are used to connect the various devices.
Security platforms can identify abnormalities and threats outside the organization's network, such as from a user, a device or an application. This is true whether the source of the anomalies and threats is within or outside the organization's network. Behaviour analytics, a security analytics strategy that can be used by enterprises of any size or skill level, may be used to discover and respond to unforeseen dangers using machine learning. Behavioural analytics include machine learning, behaviour modelling, peer group analysis, classification, statistical models, and graph analysis. To construct user and entity profiles for comparing and contrasting behaviours, the platform may use a range of methodologies, such as Markovian processing flows, inference and grouping procedures, and risk scoring systems, as explained in further detail below. Graphical user interface (GUI): The security platform's GUI may be used to map out the attack kill-chain in a visual way that security analysts can quickly and readily comprehend. This functionality is another useful aspect of the security platform.
The security platform can be implemented in several locations throughout a network. Private networks (e.g., a business intranet) may benefit from deploying a security platform at a key location (e.g., a router or a gateway linked to an administrator's computer console) to monitor and/or regulate the network traffic inside the private intranet. Using a cloud-based application, an organisation may be able to install at least a piece of the security platform, such as on the cloud-based servers, to save money and resources. Additionally, the security platform may be placed on a private network while still collecting and monitoring events on cloud-based servers as an extra or alternative option. The security platform may be able to monitor a combination of intranet and cloud-based network traffic depending on the installation. On this page, you'll find more details on using the security platform and its many features.
Using the security platform described here, you can detect, react to, and automate activities in a contemporary network environment across the whole lifetime of sophisticated security threats, known and undiscovered. Security-related challenges in today's network environments may be addressed comprehensively by this approach.
The security architecture presented here is exceptionally successful for dealing with large volumes of data, particularly machine data, from diverse sources. If you're looking for a data source that can provide information in various ways, this is the one for you (e.g., gigabytes of data per second or more). To extract information from large volumes of unstructured or structured data, machine learning/data science methodologies are used in diverse applications. This is a continuation and expansion of data mining, sometimes referred to as knowledge discovery and data mining in certain quarters (KDD).
The security platform may be cloud-based and use big data technologies to evaluate a massive amount of fast information and be highly scalable, depending on the setup. Cloud-based security platforms may be offered to consumers as a subscription service, depending on implementation. For instance, a cloud-based security platform may be accessible in certain cases (PaaS). Using cloud computing services such as PaaS, clients may easily design, operate, and manage Web applications without dealing with the burden of setting up and maintaining their infrastructure. There are a variety of PaaS services to choose from. PaaS may be supplied in at least two ways: A public cloud service from a provider, in which the consumer controls software deployment and configuration settings, while the provider provides networks, servers, storage devices and other services to host the consumer's application; and (ii) as software installed in private data centres or public infrastructure and managed by internal information technology (IT) departments.
In certain instances, machine learning approaches may detect a security threat signature or activity without the need for previous information. The following is an example of a scenario: Even if a security threat has never been encountered before or a signature for the threat has never been identified, the event data collected may reveal it.
In the various embodiments described in this paper, it is possible to detect behaviours indicative of security risks. On the other hand, the security platform and methodology presented here may be used to detect any unusual or aberrant activity involving data access, data transfer, network access, and network use, regardless of whether a security is involved.
We use the term "event data" to refer to machine data linked to network activity about a certain entity (for example, a single or multiple users, one or more network nodes, one or more network segments, one or more applications, and so on.) Depending on the implementation, the incoming event data is evaluated in two separate paths: I, a real-time processing route, and (ii) a batch processing route. If the evaluation of event data in these two data pathways can be done simultaneously, that would be ideal. For the purpose of detecting anomalies and threats as soon as they occur, this path is set up to continuously monitor and analyse incoming event data (such as an unbounded data stream). For the system to perform in real-time, the assessment must be based mostly or exclusively on current event data produced by or received from the data source (s). When the technology is used for real-time processing, data that has been stored in relation to prior events is not included. Third-party data may be excluded from the assessment in the real-time processing channel in a second embodiment of the system. If the real-time approach doesn't include the data you need, you may utilise batch processing to examine it.
According to this context, "event" and "event data" are sometimes used interchangeably to describe the collection of machine data that represents or corresponds to some network action. But the underlying action might also be referred to as an "event."
This definition also includes the concept of "anomaly," which is defined as an observable departure from an expected pattern of behaviour on an entity, regardless of whether the departure is deemed dangerous. Any time an abnormality happens, it's a hint that something isn't quite right and that more investigation is in need. Realities that may be seen or sensed are known as anomalies. A threat indicator or a threat may be identified by evaluating one or more abnormalities in combination. When a group of anomalies and/or threat indicators are taken together, they form a threat. As the severity of the danger signals and threats increases, so do the worrying events. In terms of scale, it is feasible to analyse hundreds of millions of incoming event data packets from different data sources to provide 100 anomalies, which may then be analysed further to offer ten threat indicators, which can then be further reviewed to produce one or two threats. That data scalability is possible explains why the security platform can offer anomaly and threat detection in real-time.
The amount of historical data and/or third-party data that can be used to create and improve the machine learning models that will be used to perform the evaluation is potentially much larger than the amount of real-time data stream that will be used to perform the evaluation can be used for creation and improvement of the data. Since the real evaluation of historical data is more time-consuming, it is more difficult. In other system implementations, the real-time processing channel does not use either or both of the historical data or third-party data as inputs. As an alternative, data from archives or other sources may be used as inputs. Even said, the majority of real-time processing route evaluation data still comes from incoming event data in other implementations. This is a drawback. An urgent need for fast event data analysis has resulted in obtaining actionable threat information as quickly as possible to prevent impending harm.
If danger or abnormality is identified in real-time, how would you respond? Using the real-time processing approach, an action such as blocking an incursion, shutting down network access, locking out users, preventing information theft or transfer, shutting down computer software and hardware functions, and so on, maybe automatically triggered in this situation. There are various implementations in which the discovered anomalies and dangers may be reported for decision making to a network operator (e.g., a network security administrator or analyst). Instead of or in addition to automatically taking action in response to anomalies and threats, feedback data from the user's choices (e.g., whether the found anomalies and threats are accurately diagnosed or if they are false positives) may be utilised to update and enhance the models.
Owing to real-time processing route response time limits, the batch processing path can't reveal more subtle abnormalities and risks than the real-time processing path can due to the responsive time constraints of the batch processing path. This enables the detection of more subtle abnormalities and threats than the real-time processing approach. It's possible to do batch processing in combination with real-time processing, or according to a predetermined schedule, among other possibilities.
Historical data may include data from several real-time evaluators located all around the network, each with its own unique data set. The historical data might come from a variety of periods and locales. Depending on the implementation, some implementations only analyse the incoming event data in the real-time processing path, while others store a complete version of the data as historical data; because of this, historical data may include event data that has more characteristics than the abbreviated events supplied for assessment in one or more of these implementations, depending on the implementation.
Batch analyzer anomalies, threat indicators, and threats may be automatically acted upon, as in the real-time data route. Human operators may be given the option of taking action or not. Batch analyzer results could be challenged by the operator, who could either confirm or deny them. In such a situation, the security platform may utilise this activity as a source of feedback for improving its evaluation of data that has been handled later.

Claims: 1. First event data indicative of computer network activity of an entity participating in or interacting with a computer network are received by the computer system. A first automated process in the computer system constructs based on the first event data and representative of a first particular type of computer network activity, an initial baseline of the entity's first variable behaviour. Automatic processes in the computer system are used to determine whether or not the extra computer network activity is a network security threat or an anomaly associated with the entity, based on at least one of the first variable behaviour baseline and second variable behaviour baseline.
2. The first event data and the second event data are formed completely of machine data similar to that of claim 1.
3. The first and second event data are time-stamped machine data, as stated in Claim 1.
4. According to claim 4, a computer network where the entity is a device linked to the network.