1. A method for detecting anomalous behavior patterns in a
network, wherein the method is implemented by at least one
processor (114) executing program instructions stored in a
memory(116), the method comprising:
extracting, by the processor(114), a plurality of feature values associated with predetermined features from a first collection of one or more datasets associated with a plurality of anomaly classes, wherein the predetermined features are individual measurable characteristics of network behavior and user behavior in the presence or absence of an anomaly;
determining, by the processor(114), a generic pattern of behavior associated with the plurality of anomaly classes based on the extracted feature values, wherein the generic pattern is representative of behavior which substantially simulates feature values on attack by any of the plurality of anomaly classes;
updating, by the processor(114), the determined generic pattern based on the analysis of performance of the determined generic pattern based on a second collection of one or more datasets associated with the plurality of anomaly classes and normal behavior classes; and
detecting, by the processor(114), anomalous behavior pattern in a real-time traffic based on the updated generic pattern.
2. The method as claimed in claim 1, wherein, a notification indicating abnormal traffic is generated if the feature values associated with the real-time traffic is abnormal.
3. The method as claimed in claim 1, wherein the updated generic pattern is further updated based on cumulative rewards generated based on each correct notification.
4. The method as claimed in claim 1, wherein extracting the plurality of feature values from the first collection of one or

more datasets comprises refining the first collection of one or more datasets using a first set of rules and using at least one of: a probability distribution technique and traffic pattern recognition techniques on the refined first collection of one or more datasets to extract the plurality of feature values, wherein the first set of rules comprises:
categorizing a plurality of files included in the first collection of one or more datasets into network traffic files and user behavior files, wherein the network traffic files include data associated with network anomaly class and normal traffic class, and the user behavior files include data associated with user behavior anomalies and normal user behavior;
segmenting each of the categorized datasets, wherein the network traffic files and the user behavior files are segmented into small chunks network packets and user logs respectively based on time of collection of data within the files using Long-Short-Term memory (LSTM) technique;
removing improper datasets from the segmented datasets, wherein each of the network traffic file and each of the user behavior file is processed to remove improper network packets and improper user logs respectively; and
arranging the network packets and the user logs in a chronological order.
5. The method as claimed in claim 1, wherein the plurality of feature values associated with the predetermined features are extracted from refined network packets, using a probability distribution technique, the probability distribution technique selected from a group of Levy, Cauchy, Pareto, Weibull and Lognormal.
6. The method as claimed in claim 1, wherein determining the generic pattern of behavior comprises:
labeling the extracted plurality of feature values for each of a plurality of network traffic files and user behavior

files included in the first collection of one or more datasets as abnormal or normal using supervised learning;
feeding each file with labeled features as an input to a deep Q learning technique;
identifying the fed files as normal or abnormal based on the feature values by performing a random identification action on the fed files using the deep Q learning technique, wherein the random identification action is evaluated based on the labeling, wherein a positive reward is provided if the random action is same as the label and a negative reward is provided if the random action is not same as the label, and computing a total cumulative reward based on the total number of correct actions for the plurality of files of respective anomaly classes in the first collection of one or more datasets; and
evaluating individual patterns for respective anomaly classes based on identification of fed files, and determining the generic pattern of behavior associated with the plurality of anomaly classes from the individual pattern of anomaly classes and feature values associated with respective anomaly classes, the generic pattern is fixed as a boundary for each of the plurality of anomaly classes.
7. The method as claimed in claim 1, wherein updating and analysis of the determined generic pattern is performed using a second set of rules, wherein the second set of rules comprises:
refining the second collection of one or more datasets, wherein a plurality of files included in the second collection of one or more datasets are categorized into network traffic files and user behavior files, segmented into network packets and user logs, and processed to remove improper network packets and user logs, wherein the processed network packets and user logs are arranged in a chronological order respectively;

extracting a plurality of feature values associated with the predetermined features from the plurality of files included in the second collection of one or more datasets;
identifying the feature values associated with each file included in the second collection of one or more datasets as normal or abnormal by analyzing the plurality of extracted feature values based on the determined generic pattern using reinforcement learning, wherein each file included in the second collection of one or more datasets is fed as an input to a deep Q learning technique;
evaluating each identification action based on the determined generic pattern and generating rewards for each correct identification action; and
updating the generic pattern based on cumulative rewards generated based on the total number of correct actions for the plurality of files of respective anomaly classes in the second collection of one or more datasets.
8. The method as claimed in claim 1, wherein anomalous
behavior pattern in the network is detected by analyzing the
real-time traffic using a third set of rules, wherein the third
set of rules comprises:
refining the incoming real-time traffic, wherein a plurality of files included in the real-time traffic are categorized into network traffic files and user behavior files, segmented into network packets and user logs, and processed to remove improper network packets and user logs, wherein the processed network packets and user logs are arranged in a chronological order respectively;
extracting a plurality feature values associated with the predetermined features from each of the plurality of files included in the incoming real-time traffic;
identifying the respective feature values of the refined files as normal or abnormal by analyzing the extracted plurality

of feature values based on the updated generic pattern using reinforcement learning, wherein each of refined files included in the incoming real-time traffic is fed as an input to a deep Q learning technique; and
evaluating each identification action based on the updated generic pattern and generating rewards for each correct identification action.
9. A system (108) for detecting anomalous behavior patterns in a network, the system (108) comprising:
a memory (116) storing program instructions; a processor (114) configured to execute program instructions stored in the memory (116); and an anomaly detection engine (112) in communication with the processor and configured to:
extract a plurality of feature values associated with predetermined features from a first collection of one or more datasets associated with a plurality of anomaly classes, wherein the predetermined features are individual measurable characteristics of network behavior and user behavior in the presence or absence of an anomaly;
determine a generic pattern of behavior associated with the plurality of anomaly classes based on the extracted feature values, wherein the generic pattern is representative of behavior which substantially simulates feature values on attack by any of the plurality of anomaly classes;
update the determined generic pattern based on the analysis of performance of the determined generic pattern based on a second collection of one or more datasets associated with the plurality of anomaly classes and normal behavior classes; and
detect anomalous behavior pattern in a real-time traffic based on the updated generic pattern.

10. The system (108) as claimed in claim 9, wherein, a notification indicating abnormal traffic is generated if the feature values associated with the real-time traffic is abnormal.
11. The system (108) as claimed in claim 9, wherein the updated generic pattern is further updated based on cumulative rewards generated based on each correct notification.
12. The system (108) as claimed in claim 9, wherein extracting the plurality of feature values from the first collection of one or more datasets comprises refining the first collection of one or more datasets using a first set of rules and using at least one of: a probability distribution technique and traffic pattern recognition techniques on the refined first collection of one or more datasets to extract the plurality of feature values.
13. The system (108) as claimed in claim 9, wherein the anomaly detection engine (112) comprises a data segmentation unit (120) in communication with the processor (114), said data segmentation unit (120) configured to refine the first collection of datasets, second collection of one or more datasets and the incoming real-time traffic using a first set of rules, wherein the first set of rules comprises
a. categorizing a plurality of files included in the first
collection of one or more datasets into network traffic files and
user behavior files, wherein the network traffic files include
data associated with network anomaly class and normal traffic
class, and the user behavior files include data associated with
user behavior anomalies and normal user behavior;
b. segmenting each of the categorized datasets, wherein the
network traffic files and the user behavior files are segmented
into small chunks network packets and user logs respectively
based on time of collection of data within the files using Long-
Short-Term memory (LSTM) technique;

c. removing improper datasets from the segmented datasets,
wherein each of the network traffic file and each of the user
behavior file is processed to remove improper network packets and
improper user logs respectively;
d. arranging the network packets and the user logs in a
chronological order; and
e. repeating steps a-d for the second collection of one or
more datasets and the incoming real-time traffic.
14. The system (108) as claimed in claim 9, wherein the anomaly detection engine (112) comprises a feature extraction unit (122) in communication with the processor (114), said feature extraction unit (122) configured to extract the plurality of feature values associated with network anomaly classes from refined network packets using a probability distribution technique and a traffic pattern recognition techniques, wherein the network traffic files associated with the refined network packets are fitted into one or more probability distribution techniques, and the parameters of the distribution are used as distribution features.
15. The system (108) as claimed in claim 9, wherein the anomaly detection engine comprises an identification unit in communication with the processor, said identification unit configured to determine the generic pattern of behavior by:
labeling the extracted plurality of feature values for each of a plurality of network traffic files and user behavior files included in the first collection of one or more datasets as abnormal or normal using supervised learning;
feeding each file with labeled features as an input to a deep Q learning technique;
identifying the fed files as normal or abnormal based on the feature values by performing a random identification action on the fed files using the deep Q learning technique, wherein the random identification action is evaluated based on the labeling,

wherein a positive reward is provided if the random action is same as the label and a negative reward is provided if the random action is not same as the label, and computing a total cumulative reward based on the total number of correct actions for the plurality of files of respective anomaly classes in the first collection of one or more datasets; and
evaluating individual patterns for respective anomaly classes based on identification of fed files, and determining the generic pattern of behavior associated with the plurality of anomaly classes from the individual pattern of anomaly classes and feature values associated with respective anomaly classes, the generic pattern is fixed as a boundary for each of the plurality of anomaly classes.
16. The system (108) as claimed in claim 9, wherein the anomaly detection engine (112) comprises an identification unit (124) in communication with the processor, said identification unit (124) configured to analyze performance of the determined generic pattern and update said generic pattern using a second set of rules, wherein the second set of rules comprises:
refining the second collection of one or more datasets, wherein a plurality of files included in the second collection of one or more datasets are categorized into network traffic files and user behavior files, segmented into network packets and user logs, and processed to remove improper network packets and user logs, wherein the processed network packets and user logs are arranged in a chronological order respectively;
extracting a plurality of feature values associated with the predetermined features from the plurality of files included in the second collection of one or more datasets;
identifying the feature values associated with each file included in the second collection of one or more datasets as normal or abnormal by analyzing the plurality of extracted features based on the determined generic pattern using reinforcement learning, wherein each file included in the second

collection of one or more datasets is fed as an input to a deep Q learning technique;
evaluating each identification action based on the determined generic pattern and generating rewards for each correct identification action; and
updating the generic pattern based on cumulative rewards generated based on the total number of correct actions for the plurality of files of respective anomaly classes in the second collection of one or more datasets.
17. The system (108) as claimed in claim 9, wherein the anomaly detection engine (112) comprises an identification unit (124) in communication with the processor, said identification unit (124) configured to detect anomalous behavior pattern in the network by analyzing real-time traffic using a third set of rules, wherein the third set of rules comprises:
refining the incoming real-time traffic, wherein a plurality of files included in the real-time traffic are categorized into network traffic files and user behavior files, segmented into network packets and user logs, and processed to remove improper network packets and user logs, wherein the processed network packets and user logs are arranged in a chronological order respectively;
extracting a plurality feature values associated with the predetermined features from each of the plurality of files included in the incoming real-time traffic;
identifying the respective feature values of the refined files as normal or abnormal by analyzing the extracted plurality of feature values based on the updated generic pattern using reinforcement learning, wherein each of refined files included in the incoming real-time traffic is fed as an input to a deep Q learning technique; and

evaluating each identification action based on the updated generic pattern and generating rewards for each correct identification action.