A Security Operations Center (SOC) can be defined as an organized and highly skilled team that uses advanced computer forensics tools to prevent, detect and respond to cybersecurity incidents of an organization. The fundamental aspects of an effective SOC is related to the ability to examine and analyze the vast number of data flows and to correlate several other types of events from a cybersecurity perception. The supervision and categorization of network flow is an essential process not only for the scheduling, management, and regulation of the network’s services, but also for attacks identification and for the consequent forensics’ investigations. A serious potential disadvantage of the traditional software solutions used today for computer network monitoring, and specifically for the instances of effective categorization of the encrypted or obfuscated network flow, which enforces the rebuilding of messages packets in sophisticated underlying protocols, is the requirements of computational resources. In addition, an additional significant inability of these software packages is they create high false positive rates because they are deprived of accurate predicting mechanisms.
For all the reasons above, in most cases, the traditional software fails completely to recognize unidentified vulnerabilities and zero-day exploitations. This paper proposes a novel intelligence driven Network Flow Forensics Framework (NF3) which uses low utilization of computing power and resources, for the Next Generation Cognitive Computing SOC (NGC2SOC) that rely solely on advanced fully automated intelligence methods. It is an effective and accurate Ensemble Machine Learning forensics tool to Network Traffic Analysis, Demystification of Malware Traffic and Encrypted Traffic Identification.
Cyber Warfare is the current single greatest emerging threat to National Security. Network security has become an essential component of any computer network. As computer networks and systems become ever more fundamental to modern society, concerns about security has become increasingly important. There are a multitude of different applications open source and proprietary available for the protection +-system administrator, to decide on the most suitable format for their purpose requires knowledge of the available safety measures, their features and how they affect the quality of service, as well as the kind of data they will be allowing through un flagged. A majority of methods currently used to ensure the quality of a networks service are signature based. From this information, and details on the specifics of popular applications and their implementation methods, we have carried through the ideas, incorporating our own opinions, to formulate suggestions on how this could be done on a general level. The main objective was to design and develop an Intrusion Detection System. While the minor objectives were to; Design a port scanner to determine potential threats and mitigation techniques to withstand these attacks. Implement the system on a host and Run and test the designed IDS. In this project we set out to develop a Honey Pot IDS System. It would make it easy to listen on a range of ports and emulate a network protocol to track and identify any individuals trying to connect to your system. This IDS will use the following design approaches: Event correlation, Log analysis, Alerting, and policy enforcement. Intrusion Detection Systems (IDSs) attempt to identify unauthorized use, misuse, and abuse of computer systems. In response to the growth in the use and development of IDSs, we have developed a methodology for testing IDSs. The methodology consists of techniques from the field of software testing which we have adapted for the specific purpose of testing IDSs. In this paper, we identify a set of general IDS performance objectives which is the basis for the methodology. We present the details of the methodology, including strategies for test-case selection and specific testing procedures. We include quantitative results from testing experiments on the Network Security Monitor (NSM), an IDS developed at UC Davis. We present an overview of the software platform that we have used to create user-simulation scripts for testing experiments. The platform consists of the UNIX tool expect and enhancements that we have developed, including mechanisms for concurrent scripts and a record-and-replay feature. We also provide background information on intrusions and IDSs to motivate our work.
Toward Continuous Cybersecurity with Network Automation
Network security is a dynamic art, with dangers appearing as fast as black hats can exploit vulnerabilities. While there are basic “golden rules” which can make life difficult for the bad guys, it remains a challenge to keep networks secure. John Chambers, Executive Chairman of Cisco, famously said “there are two types of companies: those that have been hacked, and those who don’t know they have been hacked”. The question for most organizations isn’t if they’re going to be breached, but how quickly they can isolate and mitigate the threat. In this paper, we’ll examine best practices for effective cybersecurity – from both a proactive (access hardening) and reactive (threat isolation and mitigation) perspective. We’ll address how network automation can help minimize cyberattacks by closing vulnerability gaps and how it can improve incident response times in the event of a cyberthreat. Finally, we’ll lay a vision for continuous network security, to explore how machine-to-machine automation may deliver an auto-securing and self-healing network.
Go to www.esgjrconsultinginc.com
IRJET- Local Security Enhancement and Intrusion Prevention in Android Dev...
This document proposes a local security enhancement and intrusion prevention system for Android devices. It summarizes existing host-based intrusion detection systems and behavior-based intrusion prevention systems for Android smartphones. The proposed system uses net flow based clustering to identify anomalies and correlates with host-based features to detect malware intrusions. The goal is to provide versatile security for Android smartphones by detecting a wide range of attacks, including denial of service attacks and probing. The system aims to detect new attacks as well.
Network infrastructures have played important part in most daily communications for business industries,
social networking, government sectors and etc. Despites the advantages that came from such
functionalities, security threats have become a daily struggle. One major security threat is hacking.
Consequently, security experts and researchers have suggested possible security solutions such as
Firewalls, Intrusion Detection Systems (IDS), Intrusion Detection and Prevention Systems (IDP) and
Honeynet. Yet, none of these solutions have proven their ability to completely address hacking. The reason
behind that, there is a few researches that examine the behavior of hackers. This paper formally and
practically examines in details the behavior of hackers and their targeted environments. Moreover, this
paper formally examines the properties of one essential pre-hacking step called scanning and highlights its
importance in developing hacking strategies. Also, it illustrates the properties of hacking that is common in
most hacking strategies to assist security experts and researchers towards minimizing the risk of hack.
Fundamentals of information systems security ( pdf drive ) chapter 1
This document discusses the growth of the internet and increased connectivity of devices beyond just computers. It notes that as internet usage has increased, issues of privacy, data security, and protecting sensitive information have become more important for both personal and business use. The document provides an overview of common security concepts and terms to help understand how to prevent cyberattacks and secure sensitive data. It also includes a table summarizing several high-profile data breaches between 2013-2015 at companies like Target, Anthem, and Sony Pictures that compromised personal and financial information for millions of customers.
This document provides guidance for lawyers on data security issues and how to help clients meet data security standards. It discusses how lack of security knowledge is common among both personal and enterprise computer users. Various threats like viruses, worms, Trojans, bots, and spyware/adware are described. Examples of data security risks include loss of portable devices containing personal information, insecure home networks that employees access for work, and insecure disposal of physical documents and digital media. The document advises evaluating security controls and investing in tools to detect breaches and audit compliance.
Towards the security issues in Mobile Ad Hoc Networks
Mobile Ad hoc Networks (MANETs) are wireless networks consisted of mobile free nodes that can move anywhere at any time without the need to any fixed infrastructure or any centralized administration. In this category of networks existing nodes must rely on each other to play the role of routers or switches instead of using central ones. The self-organized nature of such environments made MANETs vulnerable against many security threats. As a result, providing security requirements in MANETs is one of the most interesting challenges in such a network. In this group of networks, the use of cryptographic solutions is one of the most interesting security issues. The importance of this scientific area in MANETs is more drastic by considering that mentioned schemes must be lightweight enough to be appropriate for resource constrained platforms in such environment. This paper has tried to represent the position of cryptographic issues in MANETs. Moreover, security issues in mobile Ad hoc networks beside of different classes of public key cryptosystems have been introduced.
Malicious Code Intrusion Detection using Machine Learning and Indicators of C...
Malicious activities (malcodes) are self replicating
malware and a major security threat in a network environment.
Timely detection and system alert flags are very essential to
prevent rapid malcodes spreading in the network. The difficulty
in detecting malcodes is that they evolve over time. Despite the fact
that signature-based tools, are generally used to secure systems,
signature-based malcode detectors neglect to recognize muddled
and beforehand concealed malcode executables. Automatic signature
generation systems has likewise been use to address the issue
of malcodes, yet there are many works required for good detection.
Base on the behavior way of malcodes, a behavior approach is
required for such detection. Specifically, we require a dynamic
investigation and behavior Rule Base system that distinguishes
malcodes without erroneously block legitimate traffic or increase
false alarms. This paper proposed and discussed the approach
using Machine learning and Indicators of Compromise (IOC) to
analyze intrusion in a network, to identify the cause of the attack
and to provide future detection. This paper proposed the use of
behaviour malware analysis framework to analyze intrusion data,
apply clustering algorithm on the analyzed data and generate IOC
from the clustered data for IOCRule, which will be implemented
into Snort Intrusion Detection System (IDS) for malicious code
detection.
Supply chain attacks target software developers and suppliers by infecting legitimate applications to distribute malware. Attackers can compromise developer Git accounts to inject malware into repositories that get delivered to clients. They can also introduce vulnerable modules that aren't properly tested. This can lead to financial and personal data theft for customers of affected e-commerce sites, and legal issues for site owners and software vendors due to data breaches and loss of trust. Detecting malware involves scanning modules, servers, and developer systems using tools like YARA, LMD, and SYNK at various stages of the software development and delivery process.
Intelligent Network Surveillance Technology for APT Attack Detections
Recently, long-term, advanced cyber-attacks targeting a specific enterprise or organization have been occurring again. These attacks occur over a long period and bypass detection by security systems unlike the existing attack pattern. For such reason, they create problems such as delayed real-time response and detection after damages have already been incurred. This paper introduces the design of technology that applies real-time network traffic monitoring to detect unknown functional cyber-attack on the network. Specifically, the algorithm was verified and evaluated in terms of performance in an actual commercial environment. Cyber-attack detection performance is expected to be improved by enhancing the algorithm and processing large volumes of traffic
Darktrace enterprise immune system whitepaper_digital
- Darktrace takes a fresh approach to cyber defense using advanced machine learning and mathematics rather than traditional perimeter-based security.
- Traditional security models that try to distinguish insiders from outsiders no longer work in today's globally connected networks, as threats are already inside networks and boundaries are impossible to define.
- An "immune system" approach that monitors subtle internal changes and behaviors is needed to detect emerging threats, rather than defining "bad" and trying to keep threats out. This embraces probability and understands what is happening inside complex information systems.
IJERA (International journal of Engineering Research and Applications) is International online, ... peer reviewed journal. For more detail or submit your article, please visit www.ijera.com
Report: Study and Implementation of Advance Intrusion Detection and Preventio...Deepak Mishra
This document discusses building an intrusion detection system that combines network-based and log-based detection. It proposes using the Security Onion distribution and its included tools like Snort, Sguil, Squert and OSSEC. It describes configuring Security Onion sensors to monitor network traffic and logs, storing alerts in databases, and using the management consoles to analyze alerts. The goal is to create a comprehensive security monitoring platform through centralized log management and correlation of network and host-based events.
1. Cyber Ethics and Cyber Crime
2. Security in Social Media & Risk of Child Internet
3. Social media in Schools and photo privacy
4. Risk of OSNs and Security, Privacy of Facebook
5. Risk and Security of Social Networking site Facebook and Twitter
6. Risk analysis of Government and Online Transaction
Cyber Warfare is the current single greatest emerging threat to National Security. Network security has become an essential component of any computer network. As computer networks and systems become ever more fundamental to modern society, concerns about security has become increasingly important. There are a multitude of different applications open source and proprietary available for the protection +-system administrator, to decide on the most suitable format for their purpose requires knowledge of the available safety measures, their features and how they affect the quality of service, as well as the kind of data they will be allowing through un flagged. A majority of methods currently used to ensure the quality of a networks service are signature based. From this information, and details on the specifics of popular applications and their implementation methods, we have carried through the ideas, incorporating our own opinions, to formulate suggestions on how this could be done on a general level. The main objective was to design and develop an Intrusion Detection System. While the minor objectives were to; Design a port scanner to determine potential threats and mitigation techniques to withstand these attacks. Implement the system on a host and Run and test the designed IDS. In this project we set out to develop a Honey Pot IDS System. It would make it easy to listen on a range of ports and emulate a network protocol to track and identify any individuals trying to connect to your system. This IDS will use the following design approaches: Event correlation, Log analysis, Alerting, and policy enforcement. Intrusion Detection Systems (IDSs) attempt to identify unauthorized use, misuse, and abuse of computer systems. In response to the growth in the use and development of IDSs, we have developed a methodology for testing IDSs. The methodology consists of techniques from the field of software testing which we have adapted for the specific purpose of testing IDSs. In this paper, we identify a set of general IDS performance objectives which is the basis for the methodology. We present the details of the methodology, including strategies for test-case selection and specific testing procedures. We include quantitative results from testing experiments on the Network Security Monitor (NSM), an IDS developed at UC Davis. We present an overview of the software platform that we have used to create user-simulation scripts for testing experiments. The platform consists of the UNIX tool expect and enhancements that we have developed, including mechanisms for concurrent scripts and a record-and-replay feature. We also provide background information on intrusions and IDSs to motivate our work.
Network security is a dynamic art, with dangers appearing as fast as black hats can exploit vulnerabilities. While there are basic “golden rules” which can make life difficult for the bad guys, it remains a challenge to keep networks secure. John Chambers, Executive Chairman of Cisco, famously said “there are two types of companies: those that have been hacked, and those who don’t know they have been hacked”. The question for most organizations isn’t if they’re going to be breached, but how quickly they can isolate and mitigate the threat. In this paper, we’ll examine best practices for effective cybersecurity – from both a proactive (access hardening) and reactive (threat isolation and mitigation) perspective. We’ll address how network automation can help minimize cyberattacks by closing vulnerability gaps and how it can improve incident response times in the event of a cyberthreat. Finally, we’ll lay a vision for continuous network security, to explore how machine-to-machine automation may deliver an auto-securing and self-healing network.
Go to www.esgjrconsultinginc.com
IRJET- Local Security Enhancement and Intrusion Prevention in Android Dev...IRJET Journal
This document proposes a local security enhancement and intrusion prevention system for Android devices. It summarizes existing host-based intrusion detection systems and behavior-based intrusion prevention systems for Android smartphones. The proposed system uses net flow based clustering to identify anomalies and correlates with host-based features to detect malware intrusions. The goal is to provide versatile security for Android smartphones by detecting a wide range of attacks, including denial of service attacks and probing. The system aims to detect new attacks as well.
Network infrastructures have played important part in most daily communications for business industries,
social networking, government sectors and etc. Despites the advantages that came from such
functionalities, security threats have become a daily struggle. One major security threat is hacking.
Consequently, security experts and researchers have suggested possible security solutions such as
Firewalls, Intrusion Detection Systems (IDS), Intrusion Detection and Prevention Systems (IDP) and
Honeynet. Yet, none of these solutions have proven their ability to completely address hacking. The reason
behind that, there is a few researches that examine the behavior of hackers. This paper formally and
practically examines in details the behavior of hackers and their targeted environments. Moreover, this
paper formally examines the properties of one essential pre-hacking step called scanning and highlights its
importance in developing hacking strategies. Also, it illustrates the properties of hacking that is common in
most hacking strategies to assist security experts and researchers towards minimizing the risk of hack.
Fundamentals of information systems security ( pdf drive ) chapter 1newbie2019
This document discusses the growth of the internet and increased connectivity of devices beyond just computers. It notes that as internet usage has increased, issues of privacy, data security, and protecting sensitive information have become more important for both personal and business use. The document provides an overview of common security concepts and terms to help understand how to prevent cyberattacks and secure sensitive data. It also includes a table summarizing several high-profile data breaches between 2013-2015 at companies like Target, Anthem, and Sony Pictures that compromised personal and financial information for millions of customers.
This document provides guidance for lawyers on data security issues and how to help clients meet data security standards. It discusses how lack of security knowledge is common among both personal and enterprise computer users. Various threats like viruses, worms, Trojans, bots, and spyware/adware are described. Examples of data security risks include loss of portable devices containing personal information, insecure home networks that employees access for work, and insecure disposal of physical documents and digital media. The document advises evaluating security controls and investing in tools to detect breaches and audit compliance.
Mobile Ad hoc Networks (MANETs) are wireless networks consisted of mobile free nodes that can move anywhere at any time without the need to any fixed infrastructure or any centralized administration. In this category of networks existing nodes must rely on each other to play the role of routers or switches instead of using central ones. The self-organized nature of such environments made MANETs vulnerable against many security threats. As a result, providing security requirements in MANETs is one of the most interesting challenges in such a network. In this group of networks, the use of cryptographic solutions is one of the most interesting security issues. The importance of this scientific area in MANETs is more drastic by considering that mentioned schemes must be lightweight enough to be appropriate for resource constrained platforms in such environment. This paper has tried to represent the position of cryptographic issues in MANETs. Moreover, security issues in mobile Ad hoc networks beside of different classes of public key cryptosystems have been introduced.
Malicious activities (malcodes) are self replicating
malware and a major security threat in a network environment.
Timely detection and system alert flags are very essential to
prevent rapid malcodes spreading in the network. The difficulty
in detecting malcodes is that they evolve over time. Despite the fact
that signature-based tools, are generally used to secure systems,
signature-based malcode detectors neglect to recognize muddled
and beforehand concealed malcode executables. Automatic signature
generation systems has likewise been use to address the issue
of malcodes, yet there are many works required for good detection.
Base on the behavior way of malcodes, a behavior approach is
required for such detection. Specifically, we require a dynamic
investigation and behavior Rule Base system that distinguishes
malcodes without erroneously block legitimate traffic or increase
false alarms. This paper proposed and discussed the approach
using Machine learning and Indicators of Compromise (IOC) to
analyze intrusion in a network, to identify the cause of the attack
and to provide future detection. This paper proposed the use of
behaviour malware analysis framework to analyze intrusion data,
apply clustering algorithm on the analyzed data and generate IOC
from the clustered data for IOCRule, which will be implemented
into Snort Intrusion Detection System (IDS) for malicious code
detection.
Supply chain attacks target software developers and suppliers by infecting legitimate applications to distribute malware. Attackers can compromise developer Git accounts to inject malware into repositories that get delivered to clients. They can also introduce vulnerable modules that aren't properly tested. This can lead to financial and personal data theft for customers of affected e-commerce sites, and legal issues for site owners and software vendors due to data breaches and loss of trust. Detecting malware involves scanning modules, servers, and developer systems using tools like YARA, LMD, and SYNK at various stages of the software development and delivery process.
Intelligent Network Surveillance Technology for APT Attack DetectionsAM Publications,India
Recently, long-term, advanced cyber-attacks targeting a specific enterprise or organization have been occurring again. These attacks occur over a long period and bypass detection by security systems unlike the existing attack pattern. For such reason, they create problems such as delayed real-time response and detection after damages have already been incurred. This paper introduces the design of technology that applies real-time network traffic monitoring to detect unknown functional cyber-attack on the network. Specifically, the algorithm was verified and evaluated in terms of performance in an actual commercial environment. Cyber-attack detection performance is expected to be improved by enhancing the algorithm and processing large volumes of traffic
Darktrace enterprise immune system whitepaper_digitalCMR WORLD TECH
- Darktrace takes a fresh approach to cyber defense using advanced machine learning and mathematics rather than traditional perimeter-based security.
- Traditional security models that try to distinguish insiders from outsiders no longer work in today's globally connected networks, as threats are already inside networks and boundaries are impossible to define.
- An "immune system" approach that monitors subtle internal changes and behaviors is needed to detect emerging threats, rather than defining "bad" and trying to keep threats out. This embraces probability and understands what is happening inside complex information systems.
IJERA (International journal of Engineering Research and Applications) is International online, ... peer reviewed journal. For more detail or submit your article, please visit www.ijera.com
Intrusion Detection System (IDS) is meant to be a software application which monitors the network or system activities and finds if any malicious operations occur. Tremendous growth and usage of internet raises concerns about how to protect and communicate the digital information in a safe manner. Nowadays, hackers use different types of attacks for getting the valuable information. Many intrusion detection techniques, methods and algorithms help to detect these attacks. This main objective of this paper
is to provide a complete study about the definition of intrusion detection, history, life cycle, types of intrusion detection methods, types of attacks, different tools and techniques, research needs, challenges and
applications.
“AI techniques in cyber-security applications”. Flammini lnu susec19Francesco Flammini
The document discusses using artificial intelligence techniques like Bayesian networks and event trees for cybersecurity applications. It describes how these techniques can help address issues with security operations centers being overwhelmed by too much information from various sensors and systems. Bayesian networks and event trees can help fuse data from different sources to detect threats more effectively. The document provides examples of how Bayesian networks can be built using historical threat data and customized for specific organizations. It also discusses how these models can be updated dynamically based on real-time data from systems.
IRJET- Security Risk Assessment on Social Media using Artificial Intellig...IRJET Journal
1. The document proposes using artificial intelligence to assess security risks on social media by detecting suspicious activity and malicious URLs.
2. It discusses drawbacks of existing intrusion detection systems, including complexity and vulnerabilities.
3. The proposed system would use AI techniques to automate intrusion detection, identify unknown threats, and learn over time to handle large volumes of data.
AGILIS: an on-line map reduce environment for collaborative securityRoberto Baldoni
The document discusses using a collaborative approach and distributed event processing platform called Agilis to detect stealthy port scans across multiple organizations. It describes how a stealthy scan works and how collaborating organizations can share network traffic data in a "semantic room" to identify scanners that target only a small number of ports at each location. The Agilis platform is able to process large amounts of real-time data in parallel to detect such attacks with low latency even when the workload varies over time. A demonstration of the system detected a stealthy scan within 700 seconds using traffic from 8 machines simulated to represent different collaborators.
Cybercrime is increasing at a faster pace and sometimes causes billions of dollars of business- losses so
investigating attackers after commitment is of utmost importance and become one of the main concerns of
network managers. Network forensics as the process of Collecting, identifying, extracting and analyzing
data and systematically monitoring traffic of network is one of the main requirements in detection and
tracking of criminals. In this paper, we propose an architecture for network forensic system. Our proposed
architecture consists of five main components: collection and indexing, database management, analysis
component, SOC communication component and the database.
The main difference between our proposed architecture and other systems is in analysis component. This
component is composed of four parts: Analysis and investigation subsystem, Reporting subsystem, Alert
and visualization subsystem and the malware analysis subsystem. The most important differentiating
factors of the proposed system with existing systems are: clustering and ranking of malware, dynamic
analysis of malware, collecting and analysis of network flows and anomalous behavior analysis.
Intrusion Detection Systems By Anamoly-Based Using Neural NetworkIOSR Journals
To improve network security different steps has been taken as size and importance of the network has
increases day by day. Then chances of a network attacks increases Network is mainly attacked by some
intrusions that are identified by network intrusion detection system. These intrusions are mainly present in data
packets and each packet has to scan for its detection. This paper works to develop a intrusion detection system
which utilizes the identity and signature of the intrusion for identifying different kinds of intrusions. As network
intrusion detection system need to be efficient enough that chance of false alarm generation should be less,
which means identifying as a intrusion but actually it is not an intrusion. Result obtained after analyzing this
system is quite good enough that nearly 90% of true alarms are generated. It detect intrusion for various
services like Dos, SSH, etc by neural network
Pre-filters in-transit malware packets detection in the networkTELKOMNIKA JOURNAL
Conventional malware detection systems cannot detect most of the new malware in the network
without the availability of their signatures. In order to solve this problem, this paper proposes a technique
to detect both metamorphic (mutated malware) and general (non-mutated) malware in the network using a
combination of known malware sub-signature and machine learning classification. This network-based
malware detection is achieved through a middle path for efficient processing of non-malware packets.
The proposed technique has been tested and verified using multiple data sets (metamorphic malware,
non-mutated malware, and UTM real traffic), this technique can detect most of malware packets in
the network-based before they reached the host better than the previous works which detect malware in
host-based. Experimental results showed that the proposed technique can speed up the transmission of
more than 98% normal packets without sending them to the slow path, and more than 97% of malware
packets are detected and dropped in the middle path. Furthermore, more than 75% of metamorphic
malware packets in the test dataset could be detected. The proposed technique is 37 times faster than
existing technique.
A New Way of Identifying DOS Attack Using Multivariate Correlation Analysisijceronline
This document summarizes a research paper that proposes a new method for identifying denial of service (DoS) attacks using multivariate correlation analysis (MCA). The method involves three main steps: 1) generating basic features from network traffic, 2) using MCA to extract correlations between features and generate triangle area maps, and 3) using an anomaly-based detection mechanism to distinguish attacks from normal traffic based on differences from pre-generated normal profiles. The researchers evaluate their method on the KDD Cup 99 dataset and achieve moderate detection performance. However, they identify issues related to differences in feature scales that reduce detection of some attacks. They propose using statistical normalization to address this.
Team research paper and project on network vulnerabilities with multiple attacks and defesnses:
Cybersecurity
-For this project, our class was paired with teams to attempt to find vulnerabilities in other teams networks and to successfully beach their network.
-My role in this group was to help breach other team vulnerabilities through different attacks like responder attacks, honeypots, etc.
-The main challenges of this project were trying to find the vulnerabilities successfully, as the whole team had troubles with each of our different attacks and defenses.
-We learned how to use cybersecurity tools to help find vulnerabilities in networks and how to protect against them better. For example, in the honeypot we used we deployed it to port 80, when the attacker tried to access our fake server we were notified. We also deployed palto alto firewall to create our private and secure network. For an attack, we also used password crackers like john the ripper. This project taught us how to breach networks as a team.
Network Forensics is scientifically proven technique to accumulate, perceive, identify, examine, associate, analyse and document digital evidence from multiple systems for the purpose of uncovering the fact of attacks and other problem incident as well as performing the action to recover from the attack. Many systems are proposed for designing the network forensic systems. In this paper we have prepared comparative analysis of various models based on different techniques.
COPYRIGHTThis thesis is copyright materials protected under the .docxvoversbyobersby
COPYRIGHT
This thesis is copyright materials protected under the Berne Convection, the copyright Act 1999 and other international and national enactments in that behalf, on intellectual property. It may not be reproduced by any means in full or in part except for short extracts in fair dealing so for research or private study, critical scholarly review or discourse with acknowledgment, with written permission of the Dean School of Graduate Studies on behalf of both the author and XXX XXX University.ABSTRACT
With Fast growing internet world the risk of intrusion has also increased, as a result Intrusion Detection System (IDS) is the admired key research field. IDS are used to identify any suspicious activity or patterns in the network or machine, which endeavors the security features or compromise the machine. IDS majorly use all the features of the data. It is a keen observation that all the features are not of equal relevance for the detection of attacks. Moreover every feature does not contribute in enhancing the system performance significantly. The main aim of the work done is to develop an efficient denial of service network intrusion classification model. The specific objectives included: to analyse existing literature in intrusion detection systems; what are the techniques used to model IDS, types of network attacks, performance of various machine learning tools, how are network intrusion detection systems assessed; to find out top network traffic attributes that can be used to model denial of service intrusion detection; to develop a machine learning model for detection of denial of service network intrusion.Methods: The research design was experimental and data was collected by simulation using NSL-KDD dataset. By implementing Correlation Feature Selection (CFS) mechanism using three search algorithms, a smallest set of features is selected with all the features that are selected very frequently. Findings: The smallest subset of features chosen is the most nominal among all the feature subset found. Further, the performances using Artificial neural networks(ANN), decision trees, Support Vector Machines (SVM) and K-Nearest Neighbour (KNN) classifiers is compared for 7 subsets found by filter model and 41 attributes. Results: The outcome indicates a remarkable improvement in the performance metrics used for comparison of the two classifiers. The results show that using 17/18 selected features improves DOS types classification accuracies as compared to using the 41 features in the NSL-KDD dataset. It was further observed that using an ensemble of three classifiers with decision fusion performs better as compared to using a single classifier for DOS type’s classification. Among machine learning tools experimented, ANN achieved best classification accuracies followed by SVM and DT. KNN registered the lowest classification accuracies. Application: The proposed work with such an improved detection rate and lesser classification time and lar.
THE METHOD OF DETECTING ONLINE PASSWORD ATTACKS BASED ON HIGH-LEVEL PROTOCOL ...IJCNCJournal
Although there have been many solutions applied, the safety challenges related to the password security mechanism are not reduced. The reason for this is that while the means and tools to support password attacks are becoming more and more abundant, the number of transaction systems through the Internet is increasing, and new services systems appear. For example, IoT also uses password-based authentication.
In this context, consolidating password-based authentication mechanisms is critical, but monitoring measures for timely detection of attacks also play an important role in this battle. The password attack detection solutions being used need to be supplemented and improved to meet the new situation. In this
paper we propose a solution that automatically detects online password attacks in a way that is based solely on the network, using unsupervised learning techniques and protected application orientation. Our solution therefore minimizes dependence on the factors encountered by host-based or supervised learning solutions. The certainty of the solution comes from using the results of in-depth analysis of attack
characteristics to build the detection capacity of the mechanism. The solution was implemented experimentally on the real system and gave positive results.
The Next Generation Cognitive Security Operations Center: Adaptive Analytic L...Konstantinos Demertzis
The document discusses a proposed Next Generation Cognitive Computing Security Operations Center (NGC2SOC) that uses a novel intelligence driven cognitive computing framework called the λ-Architecture Network Flow Forensics Framework (λ-NF3). The λ-NF3 implements a Lambda machine learning architecture to analyze both batch and streaming network data using two computational intelligence algorithms - an Extreme Learning Machine neural network and a Self-Adjusting Memory k-Nearest Neighbors classifier. It aims to provide fully automated network traffic analysis, malware detection, and encrypted traffic identification for efficient defense against adversarial attacks without relying on human expertise.
Passive monitoring to build Situational AwarenessDavid Sweigert
Passive network monitoring techniques can provide valuable situational awareness for network security professionals. The document describes techniques for passively discovering information about nodes on a network, including operating systems, roles, services, and configurations. This contextual information helps analysts by reducing false positives and focusing resources. The passive approach does not disrupt networks and can operate continuously, in contrast to active scanning tools. A network monitoring prototype is being developed to test these passive discovery techniques.
Collecting and analyzing network-based evidenceCSITiaesprime
Since nearly the beginning of the Internet, malware has been a significant deterrent to productivity for end users, both personal and business related. Due to the pervasiveness of digital technologies in all aspects of human lives, it is increasingly unlikely that a digital device is involved as goal, medium or simply ‘witness’ of a criminal event. Forensic investigations include collection, recovery, analysis, and presentation of information stored on network devices and related to network crimes. These activities often involve wide range of analysis tools and application of different methods. This work presents methods that helps digital investigators to correlate and present information acquired from forensic data, with the aim to get a more valuable reconstructions of events or action to reach case conclusions. Main aim of network forensic is to gather evidence. Additionally, the evidence obtained during the investigation must be produced through a rigorous investigation procedure in a legal context.
Hyperparameters optimization XGBoost for network intrusion detection using CS...IAESIJAI
With the introduction of high-speed internet access, the demand for security and dependable networks has grown. In recent years, network attacks have gotten more complex and intense, making security a vital component of organizational information systems. Network intrusion detection systems (NIDS) have become an essential detection technology to protect data integrity and system availability against such attacks. NIDS is one of the most well-known areas of machine learning software in the security field, with machine learning algorithms constantly being developed to improve performance. This research focuses on detecting abnormalities in societal infiltration using the hyperparameters optimization XGBoost (HO-XGB) algorithm with the Communications Security Establishment-The Canadian Institute for Cybersecurity-Intrusion Detection System2018 (CSE-CICIDS2018) dataset to get the best potential results. When compared to typical machine learning methods published in the literature, HO-XGB outperforms them. The study shows that XGBoost outperforms other detection algorithms. We refined the HO-XGB model's hyperparameters, which included learning_rate, subsample, max_leaves, max_depth, gamma, colsample_bytree, min_child_weight, n_estimators, max_depth, and reg_alpha. The experimental findings reveal that HO-XGB1 outperforms multiple parameter settings for intrusion detection, effectively optimizing XGBoost's hyperparameters.
NETWORK INTRUSION DETECTION AND COUNTERMEASURE SELECTION IN VIRTUAL NETWORK (...ijsptm
Intrusion in a network or a system is a problem today as the trend of successful network attacks continue to
rise. Intruders can explore vulnerabilities of a network system to gain access in order to deploy some virus
or malware such as Denial of Service (DOS) attack. In this work, a frequency-based Intrusion Detection
System (IDS) is proposed to detect DOS attack. The frequency data is extracted from the time-series data
created by the traffic flow using Discrete Fourier Transform (DFT). An algorithm is developed for
anomaly-based intrusion detection with fewer false alarms which further detect known and unknown attack
signature in a network. The frequency of the traffic data of the virus or malware would be inconsistent with
the frequency of the legitimate traffic data. A Centralized Traffic Analyzer Intrusion Detection System
called CTA-IDS is introduced to further detect inside attackers in a network. The strategy is effective in
detecting abnormal content in the traffic data during information passing from one node to another and
also detects known attack signature and unknown attack. This approach is tested by running the artificial
network intrusion data in simulated networks using the Network Simulator2 (NS2) software.
Network Intrusion Detection And Countermeasure Selection In Virtual Network (...ClaraZara1
Intrusion in a network or a system is a problem today as the trend of successful network attacks continue to rise. Intruders can explore vulnerabilities of a network system to gain access in order to deploy some virus or malware such as Denial of Service (DOS) attack. In this work, a frequency-based Intrusion Detection System (IDS) is proposed to detect DOS attack. The frequency data is extracted from the time-series data created by the traffic flow using Discrete Fourier Transform (DFT). An algorithm is developed for anomaly-based intrusion detection with fewer false alarms which further detect known and unknown attack signature in a network. The frequency of the traffic data of the virus or malware would be inconsistent with the frequency of the legitimate traffic data. A Centralized Traffic Analyzer Intrusion Detection System called CTA-IDS is introduced to further detect inside attackers in a network. The strategy is effective in detecting abnormal content in the traffic data during information passing from one node to another and also detects known attack signature and unknown attack. This approach is tested by running the artificial network intrusion data in simulated networks using the Network Simulator2 (NS2) software.
Identifying Malicious Data in Social MediaIRJET Journal
This document discusses two approaches for identifying malicious data in social media: Shannon entropy and power law distribution. The Shannon entropy approach calculates the entropy of features like source/destination IP addresses and port numbers to detect anomalous network traffic patterns. The power law distribution approach models malware propagation across networks and finds that malware distribution transitions from exponential to power law over time. Experimental results on social media datasets found the Shannon entropy approach could detect malware based on the number of applications installed, while power law distribution identified good and malicious files shared between users. Both techniques aim to improve detection of malicious content shared over social networks.
Optimized Intrusion Detection System using Deep Learning Algorithmijtsrd
A method and a system for the detection of an intrusion in a computer network compare the network traffic of the computer network at multiple different points in the network. In an uncompromised network the network traffic monitored at these two different points in the network should be identical. A network intrusion detection system is mostly place at strategic points in a network, so that it can monitor the traffic traveling to or from different devices on that network. The existing Software Defined Network SDN proposes the separation of forward and control planes by introducing a new independent plane called network controller. Machine learning is an artificial intelligence approach that focuses on acquiring knowledge from raw data and, based at least in part on the identified flow, selectively causing the packet, or a packet descriptor associated with the packet. The performance is evaluated using the network analysis metrics such as key generation delay, key sharing delay and the hash code generation time for both SDN and the proposed machine learning SDN. Prof P. Damodharan | K. Veena | Dr N. Suguna "Optimized Intrusion Detection System using Deep Learning Algorithm" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-3 | Issue-2 , February 2019, URL: https://www.ijtsrd.com/papers/ijtsrd21447.pdf
Paper URL: https://www.ijtsrd.com/engineering/other/21447/optimized-intrusion-detection-system-using-deep-learning-algorithm/prof-p-damodharan
Secure intrusion detection and countermeasure selection in virtual system usi...eSAT Publishing House
IJRET : International Journal of Research in Engineering and Technology is an international peer reviewed, online journal published by eSAT Publishing House for the enhancement of research in various disciplines of Engineering and Technology. The aim and scope of the journal is to provide an academic medium and an important reference for the advancement and dissemination of research results that support high-level learning, teaching and research in the fields of Engineering and Technology. We bring together Scientists, Academician, Field Engineers, Scholars and Students of related fields of Engineering and Technology
This document provides summaries of 7 IEEE papers from 2012 related to software projects in various domains such as Java, J2ME, J2EE, .NET, MATLAB and NS2. The papers discuss topics such as password security, data provenance, trust-aware routing in wireless sensor networks, content distribution via network coding, detecting insider threats, secure message passing interfaces, and the security of an anonymity system with traceability.
Optimised malware detection in digital forensicsIJNSA Journal
On the Internet, malware is one of the most serious threats to system security. Most complex issues and
problems on any systems are caused by malware and spam. Networks and systems can be accessed and
compromised by malware known as botnets, which compromise other systems through a coordinated
attack. Such malware uses anti-forensic techniques to avoid detection and investigation. To prevent systems
from the malicious activity of this malware, a new framework is required that aims to develop an optimised
technique for malware detection. Hence, this paper demonstrates new approaches to perform malware
analysis in forensic investigations and discusses how such a framework may be developed.
Similar to The Next Generation Cognitive Security Operations Center: Network Flow Forensics Using Cybersecurity Intelligence (20)
Commentary: Aedes albopictus and Aedes japonicus—two invasive mosquito specie...Konstantinos Demertzis
In this interesting and original study, the authors present an ensemble Machine Learning (ML) model for the prediction of the habitats’ suitability, which is affected by the complex interactions between living conditions and survival-spreading climate factors. The research focuses in two of the most dangerous invasive mosquito species in Europe with the requirements’ identification in temperature and rainfall conditions. Though it is an interesting approach, the ensemble ML model is not presented and discussed in sufficient detail and thus its performance and value as a tool for modeling the distribution of invasive species cannot be adequately evaluated.
A Dynamic Intelligent Policies Analysis Mechanism for Personal Data Processin...Konstantinos Demertzis
The document describes an Intelligent Policies Analysis Mechanism (IPAM) that is part of the ADVOCATE framework. IPAM uses machine learning methods like Fuzzy Cognitive Maps and Extreme Learning Machines to identify potentially conflicting rules or consents from a user that could lead to personal data collection and profiling without consent. The framework aims to help users maintain control over their personal data as required by GDPR regulations. IPAM simulates how smart devices collect personal data and identifies rules that may enable profiling, training on example data to learn how to detect such instances.
GeoAI: A Model-Agnostic Meta-Ensemble Zero-Shot Learning Method for Hyperspec...Konstantinos Demertzis
The document discusses a new meta-ensemble zero-shot learning method called MAME-ZsL for hyperspectral image analysis and classification. MAME-ZsL overcomes the difficulties of traditional deep learning methods that require large labeled datasets and long training times. It reduces computational costs, avoids overfitting, and achieves high classification accuracy even when testing classes were not present during training. The method is a novel optimization-based meta-ensemble architecture that facilitates learning representations from limited labeled examples to enable one-shot and zero-shot learning.
Modeling and Forecasting the COVID-19 Temporal Spread in Greece: An Explorato...Konstantinos Demertzis
This document presents a novel method for modeling and forecasting the temporal spread of COVID-19 in Greece based on complex network analysis. The method develops a spline regression model where the knot vector is determined by community detection in the network representing the time series. The model provides a reliable framework for forecasting that can help inform decision making and management of health resources for fighting COVID-19 in Greece. The analysis finds that Greece's infection curve experienced 5 stages of declining dynamics and showed signs of saturation after 33 days, suggesting Greece's response has been effective at keeping cases and deaths relatively low.
Extreme deep learning in biosecurity the case of machine hearing for marine s...Konstantinos Demertzis
Biosafety is defined as a set of preventive measures aimed at
reducing the risk of infectious diseases’ spread to crops and
animals, by providing quarantine pesticides. Prolonged and
sustained overheating of the sea, creates significant habitat losses,
resulting in the proliferation and spread of invasive species, which invade foreign areas typically seeking colder climate. This is one of the most important modern threats to marine biosafety. The research effort presented herein, proposes an innovative approach
for Marine Species Identification, by employing an advanced
intelligent Machine Hearing Framework (MHF). The final target is the identification of invasive alien species (IAS) based on the
sounds they produce. This classification attempt, can provide
significant aid towards the protection of biodiversity, and can
achieve overall regional biosecurity. Hearing recognition is
performed by using the Online Sequential Multilayer Graph
Regularized Extreme Learning Machine Autoencoder
(MIGRATE_ELM). The MIGRATE_ELM uses an innovative Deep Learning algorithm (DELE) that is applied for the first time for the above purpose. The assignment of the corresponding class ‘native’ or ‘invasive’ in its locality, is carried out by an equally innovative approach entitled ‘Geo Location Country Based Service’ that has been proposed by our research team.
The internet has revolutionized the way we live our lives – enabling us to read the news, enjoy entertainment, carry out research, book our holidays, buy and sell, shop, network, learn, bank and carry out many other everyday tasks. However, there are a number of risks associated with going online. Hackers are still on the lookout for personal information they can use to access your credit card and bank information.
Η χαλαζόπτωση αποτελεί έναν από τους σοβαρότερους κινδύνους της γεωργικής παραγωγής. Οι άμεσες συνέπειες της παρατηρούνται στη διάλυση της επιδερμίδας και τον τραυματισμό ή και την πτώση των ανθέων, καρπών, φύλλων και βλαστών, ενώ επιπρόσθετα τα πληγέντα φυτά παρουσιάζουν μεγαλύτερη ευαισθησία σε μυκητολογικές ασθένειες και σε εντομολογικές προσβολές. Σκοπός Η έγκαιρη αξιολόγηση σε ημερήσια βάση, όσον αφορά στο αν θα προκύψει φυσική καταστροφή λόγω χαλαζόπτωσης, μπορεί συμβάλλει καθοριστικά στην προστασία του γεωργικού κεφαλαίου της χώρας, αφού θα ενδυναμώσει σημαντικά τους μηχανισμούς πολιτικής προστασίας και θα δημιουργήσει τις κατάλληλες συνθήκες για βιώσιμη ανάπτυξη και οικονομική ευημερία. Υλικό Για τον έγκαιρο και έγκυρο χαρακτηρισμό (πρόβλεψη) μιας ημέρας ως ημέρα χαλαζόπτωσης, δημιουργήθηκε ένα νευρωνικό δίκτυο εμπρόσθιας τροφοδοσίας, το οποίο είναι ικανό να προβλέψει την χαλαζόπτωση. Για την εκπαίδευση και αξιολόγηση του συστήματος, χρησιμοποιήθηκαν τα ιστορικά δεδομένα χαλαζόπτωσης καθώς και τα μετεωρολογικά δεδομένα των τελευταίων 18 ετών της Κεντρικής Μακεδονίας. Μέθοδος Η σχεδίαση και ανάπτυξη του προτεινόμενου συστήματος πραγματοποιήθηκε με τη χρήση τεχνητών νευρωνικών δικτύων τα οποία έχουν την δυνατότητα να μοντελοποιήσουν πολύπλοκα μη γραμμικά προβλήματα ταξινόμησης (classification) εκμεταλλευόμενα την εγγενή ικανότητα μάθησης των τεχνητών νευρώνων. Η προσέγγιση επιλέχθηκε μετά από εξαντλητικές δοκιμές και συγκρίσεις διαφορετικών αλγοριθμικών μεθόδων μηχανικής μάθησης. Αποτελέσματα Τα αποτελέσματα της έρευνας είναι ιδιαίτερα ενθαρρυντικά καθώς η πρόβλεψη της χαλαζόπτωσης επιτυγχάνεται με ποσοστό ακρίβειας (Accuracy) 91,5%. Το γεγονός της ύπαρξης πολλών δεδομένων που αφορούν σε μεγάλο πλήθος εμπλεκομένων παραμέτρων, συνέβαλε σημαντικά στην επιτυχία της συγκεκριμένης μεθόδου. Συμπεράσματα Η εργασία προτείνει ένα σύστημα Μηχανικής Μάθησης με δυνατότητα ταξινόμησης των περιπτώσεων ως ημέρες χαλαζόπτωσης ή όχι. Το κυριότερο είναι ότι αυτό γίνεται εύκολα, γρήγορα και με μεγάλη ακρίβεια. Η αξιοπιστία και η βέλτιστη απόδοση του προτεινόμενου συστήματος με νέα δεδομένα που δεν είχαν καμία σχέση με τα δεδομένα εκπαίδευσης, προέκυψε μετά από την πραγματοποίηση εκτεταμένων συγκρίσεων μεταξύ διαφορετικών αλγοριθμικών προσεγγίσεων και αρχιτεκτονικών.
This document describes a hybrid artificial intelligence system for cyber security (HAISCS) that uses two components:
1. An Evolving Spiking Anomaly Detection Model (ESADM) that uses a spiking neural network for pattern recognition and classification of anomalies.
2. An Evolving Computational Intelligence System for Malware Detection (ECISMD) that uses an Evolving Classification Function and genetic algorithm for malware classification.
The system aims to improve cyber security through hybrid supervised and unsupervised machine learning methods for detection of anomalies and malware. Evaluation results show the hybrid system achieves better performance than other classifiers. Future work will focus on improving and expanding the system.
Amazon DocumentDB(MongoDB와 호환됨)는 빠르고 안정적이며 완전 관리형 데이터베이스 서비스입니다. Amazon DocumentDB를 사용하면 클라우드에서 MongoDB 호환 데이터베이스를 쉽게 설치, 운영 및 규모를 조정할 수 있습니다. Amazon DocumentDB를 사용하면 MongoDB에서 사용하는 것과 동일한 애플리케이션 코드를 실행하고 동일한 드라이버와 도구를 사용하는 것을 실습합니다.
LLM powered contract compliance application which uses Advanced RAG method Self-RAG and Knowledge Graph together for the first time.
It provides highest accuracy for contract compliance recorded so far for Oil and Gas Industry.
Noida Extension @ℂall @Girls ꧁❤ 9873940964 ❤꧂VIP Vishakha Singla Top Model Safe
The Next Generation Cognitive Security Operations Center: Network Flow Forensics Using Cybersecurity Intelligence
1. big data and
cognitive computing
Article
The Next Generation Cognitive Security Operations
Center: Network Flow Forensics Using
Cybersecurity Intelligence
Konstantinos Demertzis 1,* , Panayiotis Kikiras 2, Nikos Tziritas 3, Salvador Llopis Sanchez 4
and Lazaros Iliadis 1
1 Department of Civil Engineering, School of Engineering, Democritus University of Thrace, Xanthi 67100,
Greece; liliadis@civil.duth.gr
2 Department of Computer Science, School of Science, University of Thessaly, Lamia 35131, Greece;
kikirasp@uth.gr
3 Research Center for Cloud Computing, Shenzhen Institutes of Advanced Technology,
Chinese Academy of Sciences, Shenzhen 518000, China; nikolaos@siat.ac.cn
4 Communications Department, Universitat Politecnica de Valencia, Valencia 46022, Spain;
salllosa@masters.upv.es
* Correspondence: kdemertz@fmenr.duth.gr; Tel.: +30-694-824-1881
Received: 25 October 2018; Accepted: 20 November 2018; Published: 22 November 2018
Abstract: A Security Operations Center (SOC) can be defined as an organized and highly skilled
team that uses advanced computer forensics tools to prevent, detect and respond to cybersecurity
incidents of an organization. The fundamental aspects of an effective SOC is related to the ability to
examine and analyze the vast number of data flows and to correlate several other types of events from
a cybersecurity perception. The supervision and categorization of network flow is an essential process
not only for the scheduling, management, and regulation of the network’s services, but also for attacks
identification and for the consequent forensics’ investigations. A serious potential disadvantage of
the traditional software solutions used today for computer network monitoring, and specifically for
the instances of effective categorization of the encrypted or obfuscated network flow, which enforces
the rebuilding of messages packets in sophisticated underlying protocols, is the requirements of
computational resources. In addition, an additional significant inability of these software packages is
they create high false positive rates because they are deprived of accurate predicting mechanisms.
For all the reasons above, in most cases, the traditional software fails completely to recognize
unidentified vulnerabilities and zero-day exploitations. This paper proposes a novel intelligence
driven Network Flow Forensics Framework (NF3) which uses low utilization of computing power
and resources, for the Next Generation Cognitive Computing SOC (NGC2SOC) that rely solely on
advanced fully automated intelligence methods. It is an effective and accurate Ensemble Machine
Learning forensics tool to Network Traffic Analysis, Demystification of Malware Traffic and Encrypted
Traffic Identification.
Keywords: network flow forensics; Security Operations Center; network traffic analysis;
traffic identification; demystification of malware traffic; ensemble machine learning
1. Introduction
Network traffic analysis [1] is the method of capture, studying and analyzing network traffic flow
for the purpose of performance, security and network services management. The basic strategy to
network traffic analysis is a payload-based classification tactic [2] where the list of packages is sorted
based on the payload, such as Mac (Layer 2), IP address (Layer 3), source/destination ports (Layer 4)
Big Data Cogn. Comput. 2018, 2, 35; doi:10.3390/bdcc2040035 www.mdpi.com/journal/bdcc
2. Big Data Cogn. Comput. 2018, 2, 35 2 of 17
and protocols. An alternative is the statistical analysis method of the traffic behavior that is ordered
based on characteristics such as interpacket arrival, session, timestamp and so on.
On the other hand, malware is a kind of malicious software used to gain access to network
infrastructures without permission, to collect personal information or disrupt computer operation
and facilities. It can use any event-handling procedures such as source code, dynamic scripts, or
any other active content. Innovative malware types can obfuscate and remain concealed through
infection and operation with sophisticated techniques that use ambiguous filenames, alteration of file
features, or operation under the pretense of valid software and services to prevent investigation and
deletion. Furthermore, the malicious process often tries to destabilize the entire system by bypassing
the antivirus software and obfuscate active procedures, network services, and threads from suspicious
URLs or registry values [3].
Moreover, the fast-growing use of encrypted traffic is changing the threat landscape. Nowadays,
many services and software packages are using a type of encryption as the primary method to
secure the sensitives information [4]. By the same logic, cybercriminals use advanced and highly
sophisticated types of malicious operations based on progressive encryption to hide malware payload,
command and control activities, or information exfiltration. Most malware types are developed to
access silently and continue to exist for an extended period, to take ownership and get full control of
the equipment, and to interconnect (via encryption) with the botmaster and its Command and Control
(C&C) servers [4].
This paper suggests a novel network forensics framework for security operating centers that relies
solely on fully adaptive computational intelligence approaches. It is an effective and accurate ensemble
machine learning forensics tool that uses low utilization of computing power and resources to analyze
instantly or in real time the network flow to identify encrypted or malware traffic. It is a novel cognitive
analytics framework that employs an ensemble architecture which combines Support Vector Machine
(SVM) [5], Artificial Neural Network (ANN) [6], Random Forest (RF) [7] and k-Nearest Neighbors
(k-NN) [8] to investigate malicious activities from data flows in real time. The reason for using the
ensemble technique is that complex multifactorial problems such as the one under consideration
contain strong multi-variability that can be analyzed and finally solved by the sensitivity of the
overlapping models. In addition, an ensemble model is appropriate to effectively express the
mathematical modeling of data vectors that are used to describe complicated relationships, such
as the one between normal and malicious network traffic. The combination of four different algorithms
facilitates the sorting process, making each classifier more robust, and it accelerates the convergence of
the generic multiple model, which is less noisy than any single one [9]. Thus, this approach offers
generalization and avoids overfitting which is one of the basic targets in machine learning.
The rest of the paper is organized as follows. Section 2 presents the related work about the traffic
analysis systems that have used machine learning methods. Section 3 describes the proposed NF3
model. Section 4 defines the methodology. Section 5 describes the ensemble of algorithms used by
NF3. Section 6 describes the datasets. Section 7 presents the results. Section 8 contains the conclusions.
2. Related Work
The basic drawback of software to analyze network flows is that these applications do not offer
the deep packet level details required for comprehensive analysis, as they do not have the access to
each packet in the traffic flow to achieve high level application analysis. In addition, the precision of
the analysis depends a portion on the sample rate selected. The higher is the sample rate, the more
precise is the analysis. The type of sampling also plays a vital issue in the accuracy of outcomes.
The supported sample rates are dependent on the software vendors [10].
Furthermore, all network infrastructures need to support the appropriate protocols for a
comprehensive network flow analysis. Moreover, when working with many network flows,
the bandwidth overhead as well as computer resources requirements for analysis procedures will have
a substantial impact on the system [11].
3. Big Data Cogn. Comput. 2018, 2, 35 3 of 17
Besides, operators are aided by visual means when analyzing big data. Their interpretation of the
reality on the screen may vary due to their skills and knowledge. An inherent and implicit demand for
proof of the effectiveness is to maximize operators’ cyber situation awareness by adopting meaningful
visualization tools as part of a comprehensive decision-support mechanism [12].
A major issue of these applications, including consistent advanced applications that rely on
Deep Packet Inspection (DPI) [13] methods, is the use of signature to achieve threat identification.
For this signature-based malware identification, an imperfect signature capability can be used to
recognize the well-known events, provided the correct packet is sampled and the signature exists.
Unfortunately, up-to-date malicious code appear that are not predictable, and these newly released
forms can only be distinguished from benign files and activity by behavioral analysis or another
progressive technique [14].
For example, the most recent types of malware are looking at establishing secret communications
with the remote C&C servers on a systematic basis, so that cybercriminals can transfer the malicious
payload to the compromised devices (bots) using hardcoded pool lists of IP addresses. Specifically,
to remain hidden by the IDS/IPS, the botnets communicate using secret dynamic DNS services that
are implemented in high port numbers to generate the next rendezvous-point with the botmasters.
These rendezvous-points are characterized by a mixture of hundreds of random IP addresses and
a very small Time-To-Live (TTL) for each partial DNS Resource Record. In addition, the use of
sophisticated cryptography in malware code with the combination of the Blind Proxy Redirection
(BPR) method that repeatedly redirects the requests to another group of backend servers to spoil traces
and disappear the underlying networking details makes it very difficult to identify the C&C servers by
law enforcement [15,16]. Hence, the botnets have become more complicated [3].
The most effective method for cyber-attacks prevention and effective investigation of malware
communications is the demystification of malware traffic. Moreover, this is the primary technique to
estimate the behavior of the malicious process, the intention of attacks and the degree of impairment
caused by these activities [3].
The latest sophisticated malware uses the chaotic construction of the Tor network [17] to encrypt
the botnet traces and modify the paths of an attack [18]. This encrypted peer-to-peer network,
based on manifold layers of sophisticated encryption, complex virtual circuits and overlays that
change frequently [19], certifies the secrecy among the compromised machines and the hidden services
on a botnet.
Moreover, a characteristic that adds complexity in the investigation process of Tor-based malware
is the fact that these types of malware operate in the transport layer of the OSI model, thus the network
flow shows clients of the Secure Socket Interface (SOCKS) which operates in the session layer [20].
As a result, Tor uses port 443, so the generated traffic simulates the legitimate HTTPS traffic.
One of the most reliable methods to successfully identify the Tor-generated traffic flow is
statistical analysis and the investigation of the changes in the Secure Sockets Layer (SSL) protocol [20].
For example, a statistical analysis about the related domain name, the time-to-live, etc. can identify the
Tor sessions in a network full of HTTPS traffic [3,16,19,20].
NF3 is an artificial intelligence (AI) computer security technique [21–26]. Machine learning (ML)
methods, using static [27] and dynamic [28] investigation to classify malicious contend [29], to achieve
network traffic arrangement [30], to analyze malware traffic [31] and to identify botnets [32], has been
done in the past. In contrast, numerous writers suggest different classifications methods or discovery
procedures, presenting alternative classes of botnet detection [33,34]. Generally, the traffic analysis
with machine learning-based methods has proved effective in the investigation of some of the biggest
and most harmful cyber-attacks over the past decade [35–37].
On the other hand, Hsu et al. [38] proposed a real-time system for detecting botnets based on an
anomaly detection system that inspects the delays in HTTP/HTTPS client requests with interesting
outcomes. In addition, the authors of the study [39] employed several machine learning models to
categorize the SSH flow, with limited features of the payload. Alshammari et al. [40] proposed an
4. Big Data Cogn. Comput. 2018, 2, 35 4 of 17
accurate ensemble system that classifying the SSH traffic without extracting features from the payload.
Holz et al. [41] investigated a precise method to trace botnets. Almubayed et al. [42] presented a method
that measures the performance of several algorithms to identify the encrypted traffic in a network.
Chaabane et al. [43] described an in-depth study about HTTP, BitTorrent and Tor traffic and a method
to identify these protocols from user’s behavior. There are several similar studies that propose methods
to locate the encrypted relay nodes of the Tor network [44,45]. Mees et al. [46] developed a multi-aspect
tri-dimensional picture, specific to the cyber domain, to provide a starting point calculation of the cyber
situation by computing mission-specific features and related metrics with expert knowledge using
fuzzy logic. Llopis et al. [12] presented a comparative analysis between visualization techniques, using
the operational picture concept, to support incident handling. The mentioned authors anticipated
that visualization could be enhanced by using AI algorithms to classify information and support
decision making.
3. Proposed Framework
Since cyber systems’ security is an extremely complex process, SOCs administrators cannot be
based only in the use of isolated protection products installed on each checkpoint aiming to avoid an
incident. The detection of an intrusion in the network should not be a manual and time-consuming
process, which would offer an important advantage to the attackers. Following this point of view,
the use of more effective methods of network supervision, with capabilities of automated control,
are important to estimate the behavior of malware, the aim of cyber-attacks and the degree of
impairment caused by malware activities.
Updating the SOC and its transformation into a NGC2SOC are also important. The ideal
NGC2SOC includes advanced machine learning solutions for real-time analysis of both known and
unknown threats; instant reports; data visualization tools; and other sophisticated solutions that
minimize the risk of critical assets as well as fully automate the restoration of cyber security issues.
Unlike other techniques that have been proposed in the literature focused on single flow
analysis approaches [17,18], the dynamic ensemble model of NF3 reduces the overfit without special
requirements and computer resources.
The algorithmic approach of the proposed NF3 includes in Stage 1 a feature extraction process
from network flows, as shown in the depiction of the proposed NF3 model (Figure 1). In Stage 1,
these features are concurrently checked by each learning algorithm to produce an ensemble averaging
model (Stage 2 in Figure 1). In an ensemble averaging model, for every instance of test dataset, the
average predictions are calculated.
The analysis is about determining the normal or abnormal network traffic (network traffic
analysis). If the analysis gives a positive result (Stage 3 in Figure 1) and the traffic is categorized as
abnormal, it will be analyzed to specify the abnormality (demystification of malware traffic) that is
taking place (Botnet, Crimeware, APT, Attack, or CoinMiner). On the contrary, normal traffic will be
checked (Stage 4 in Figure 1) to identify encrypted traffic (encrypted traffic identification) and the
protocol it uses (Tor, SSH, SSLweb, SSLP2P, SCP, and Skype) (Stage 5 in Figure 1) or to identify which
application is being used for non-encrypted traffic (FTP, HTTP, DNS, and SMTP) (Stage 6 in Figure 1).
Figure 1 is a representation of the algorithmic approach of the proposed NF3 model.
5. Big Data Cogn. Comput. 2018, 2, 35 5 of 17
Big Data Cogn. Comput. 2018, 2, x FOR PEER REVIEW 5 of 17
Figure 1. A depiction of the algorithmic approach of the proposed NF3 model.
4. Methodology
4.1. Ensemble Learning
Implementing NF3 is based on the optimal usage and the combination of reliable algorithms,
which create a complete, innovative, computationally intelligent ensemble machine learning
framework to solve a real cyber security problem. Ensemble approaches are meta-algorithms that
syndicate numerous machine learning methods into one forecasting model to reduce variance
(bagging), bias (boosting), or improve estimates (stacking) [47].
Figure 1. A depiction of the algorithmic approach of the proposed NF3 model.
4. Methodology
4.1. Ensemble Learning
Implementing NF3 is based on the optimal usage and the combination of reliable algorithms,
which create a complete, innovative, computationally intelligent ensemble machine learning
framework to solve a real cyber security problem. Ensemble approaches are meta-algorithms
that syndicate numerous machine learning methods into one forecasting model to reduce variance
(bagging), bias (boosting), or improve estimates (stacking) [47].
6. Big Data Cogn. Comput. 2018, 2, 35 6 of 17
The main imperative advantages of ensemble models are that they produce more stable
implementations, often improving the entire prediction, and they are capable of generalization [48].
This is a critical requirement of machine learning models so that they can adapt properly to new,
previously unseen data.
An ensemble forecasting method may not necessarily yield the maximum performance, but it
definitely decreases the overall risk of a particularly poorer choice. On the other hand, it should be
related by a detailed examination of the elements or structure and it should explain in-depth some
critical decision points correlated to the procedure and use of the approach [9,47,48]. Some of these
points are presented below.
4.2. Ensemble Size
The number of predictors comprised in the conception of an ensemble model has a big influence
on the performance of the entire model. The detailed exploration of the optimal number of classifiers is
a constant research issue as well as a constant consultation among researchers, and it should be noted
that there are very limited studies that address this matter [9,47,48].
In this study, comprehensive statistical research was used to determine the appropriate number of
classifiers [49]. More recently, after the publishing of the “law of diminishing returns in ensemble
construction” [9,47–50], it is suggested that the ideal number of classifiers that can yield the highest
precision in an ensemble model for a dataset is equivalent to the obtained number of classes. However,
it is generally accepted that a priori determination of the number of classifiers without scientific
evidence is a precarious decision that does not guarantee the quality of the model.
To materialize the NF3 model, four classifiers were used according to “the law of diminishing
returns in ensemble construction” principle. In addition, statistical tests carried out and the final
ensemble size were decided with trial and error method.
4.3. Model Selection
The selection processes of the appropriate predictors to be comprised in an ensemble
method [9,47–50] should be based on the restrictions’ settings and configurations that can take into
consideration different decision boundaries. For example, the obvious choice of classifiers with the
smallest error in training data is considered as not proper for generating ensembles, as performance in
a training dataset, even when cross validation is used, may be misleading in terms of classification
performance in unknown data [51].
For this process to be effective, individual classifiers should not only display a certain level of
diversity, but they should also use different operating parameters and different sets of training data,
thus allowing different decision boundaries to be created, which can be combined to reduce the overall
error [9,51].
In general, the selection was based on a heuristic method that considered the basic properties of
how these algorithms face each situation, for instance: Are parametric models (ANN parametric,
Kernel SVM non-parametric, etc.) suitable? How should outliers be handled? (For example, RF using a
subgroup of training sets with bagging and subgroups of features can help reduce the effect of outliers
or extreme values.) How should noise be handled? (For example, KNN has some nice properties:
it is repeatedly nonlinear, it can detect linear or nonlinear dispersed data, and it tends to achieve
good results with many data vectors.) The final decision was made using the statistical trial and
error method.
4.4. Identification of the Weights of Different Models of Ensemble
An analysis that should accompany the training of an ensemble model should aim to find the
optimal weights of the algorithms involved [9,47–51]. The weight vector is a very critical function in
the training process of an Ensemble model, as it is used in the process of determining the reliability of
predictors and the trustworthiness of their classification. In the case of higher weights, a question is
7. Big Data Cogn. Comput. 2018, 2, 35 7 of 17
raised of how they play a more important role in defining the process of classifiers’ combination and
how they determine the confidence of the overall model. The usual practice of employing the same
weight for all algorithms and averaging the forecasts [51,52] is a rough heuristic technique to address
this challenge, which is not totally based on scientific evidence.
To create a NF3 ensemble model, the identification of the weights of different models was
completely based on the statistical trial and error method described above.
4.5. Reliability of Ensemble
The slight difference of forecast performance in a machine learning model is one of the most
important distinctive attributes for evaluating the reliability and intercity of the ensemble model.
In particular, when dispersion is low and prediction solutions are consistent across multiple tests,
the model is more reliable, unlike large dispersion cases, which suggest high uncertainty rates in the
final forecast [52].
Ideally, the spread of the expected error should be concentrated near a value that can be
described as the average error [9,47–52]. Essentially, for a prediction to be considered reliable,
the observed state should behave as if it was derived from the prediction probability distribution.
On the contrary, the combination of fixed categorizers for the compilation of an ensemble predictive
model is a less advantageous tactic as this will not help to improve the generalization ability of the
final model. Therefore, a very important reliability factor of an ensemble model is the diversity of
the selected classifiers, which can be achieved with different architectures, parameter settings and
training techniques.
To construct the NF3 model, various algorithms were selected based on the function method
and their parameterization, which is accomplished by using different architectures, hyperparameter
settings and training techniques.
4.6. Importance of Ensemble
Timeliness is a key issue at SOC levels, which is why they use combined cyber threat intelligence
capabilities. Implementing NF3 is based on the optimal usage and combination of reliable algorithms,
which create a complete ensemble machine learning framework to solve a real cyber security
problem. Ensemble methods are more stable models and offer generalization. In machine learning,
generalization denotes the aptitude of a model to be effective across a variety of inputs. Specifically,
an ensemble model such as the proposed NF3 can fit unseen patterns such as zero-day malware or
attacks. This is a major innovation that significantly improves the performance of the SOC/NOC,
against sophisticated zero-day exploits.
5. Ensemble of Algorithms
The algorithms used and the individual determination and usage parameters of the proposed
ensemble framework are briefly presented below.
5.1. Support Vector Machine (SVM)
SVM is mainly a classifier that creates hyperplanes in a multidimensional space that separates
decision boundaries of dissimilar classes [5]. It assumes that the data are linearly separable. The SVM
employs a reiterative training procedure to build an ideal hyperplane the error function is minimal
when maximizing the margin subjected to a set of linear constraints. This procedure can be considered
as an optimization problem that can be resolved by quadratic programming. Generally, the nonlinear
data are transformed to a higher dimension to reach linear separation. For example, the kernel trick
is an efficient method to transform the original data space into a high dimension that has an explicit
dividing margin between classes of data. There are few tuning parameters and so the typical technique
is to operate in two phases: (1) find the optimal optimization parameters; and (2) train the SVM using
those parameters.
8. Big Data Cogn. Comput. 2018, 2, 35 8 of 17
If the data are linearly separable, the decision surface has the following form [5]:
tk wT
xk + b ≥ +1, k = 1, 2, 3, . . . , N (1)
where x is the input vector, w is the weight vector, b is bias and wTx + b = 0 is the decision boundary.
When data are nonlinearly separable, which is more probable due to uncertainty, representation
inaccuracy and latency, there is a classification error and the purpose of SVM is to minimize this error.
A new dataset of positive numbers is inputted whose name is slack variables, and which calculate the
data diversion from correct classification. In this case, the decision surface is calculated:
tk wT
xk + b ≥ 1 − ξk, k = 1, 2, 3, . . . , N (2)
where ξk ≥ 0 are the slack variables. Hence, we have the formulation of the SVM optimization problem
which find the optimal surface (w∗, b∗
) with slack variables that reduce the cost of J(w) = 1
2 wTw:
min
w, b
J(w, ξ) =
1
2
wT
w + c
N
∑
k=1
ξk (3)
thus
tk wT
xk + b ≥ 1 − ξk and ξk ≥ 0, k = 1, 2, 3, . . . , N (4)
where c is the capacity constant [5].
In the network traffic classification problem, the authors used the Gauss kernel SVM method to
calculate the maximum-margin hyperplanes:
k x, x = exp
−||x − x ||2
2σ2
(5)
5.2. Artificial Neural Network (ANN)
ANNs are algorithms that simulate the human brain [6]. They are widely used for nonlinear
modeling and often are characterized by computational soft computing techniques. The most common
training method for ANNs algorithm is the Back-Propagation (BP) method that can be considered as a
method to calculate the weights to be used in the network to minimize the error output.
The Mean Squared Error (MSE) or Root Mean Square Error (RMSE) is the performance metrics
throughout the training, validation, and testing procedures of ANNs with BP training method [53,54].
MSE =
1
N
N
∑
i=1
(ei)2
=
1
N
N
∑
i=1
(ti − ai)2
(6)
RMSE =
1
n
n
∑
j=1
P(ij) − Tj
2
(7)
The following is a heuristic function to estimate the neurons on the hidden layer:
2
3
∗ Inputs + Outputs (8)
The Levenberg–Marquardt is the training algorithm of the MLFF ANN:
xk+1 = xk − JT
J + µI
−1
JT
e (9)
9. Big Data Cogn. Comput. 2018, 2, 35 9 of 17
where proper steps are followed to:
1. Calculate the inputs:
sj =
n
∑
i=1
WijXi − θj(9), j = 1, 2, . . . , h (10)
2. Calculate the output for each hidden node:
Sj = sigmoid(sj) =
1
1 + exp −sj
, j = 1, 2, . . . , h (11)
3. Calculate the overall outputs:
ok =
h
∑
j=1
WjkSj − θ k(11), k = 1, 2, . . . , m (12)
Ok = sigmoid(ok) =
1
(1 + exp(−ok))
(12), k = 1, 2, . . . , m (13)
5.3. Random Forest (RF)
The Random Forests (RF) is a forecasting method that operates by creating a plethora of decision
trees. The training method for RF applies a general bootstrap aggregating method, or a bagging
technique, or a tree-learning process. Generally, the RF algorithm can be described as follows [7]:
(1) Draw n tree bootstrap samples from the source dataset.
(2) At each node of the bootstrap samples, grow an unpruned predictor tree and choose the best split
between variables.
(3) Predict new data by aggregating the predictions of the n trees and estimate the error at each
iteration using the “out-of-bag” method.
5.4. k-Nearest Neighbors (k-NN)
The K-Nearest Neighbors (k-NN) algorithm (also known as instance-based learning) is a simple
classification method developed to meet the need for discriminant analysis when dependable
parametric estimations of likelihood concentrations are unidentified or difficult to regulate [8].
To forecast a new data point, the nearby k neighbors are determined in the training set and then
they follow a voting process to produce the final prediction. k is a parameter defined by the user and
an unlabeled data-vector is classified with the most frequent label between the k training samples
nearest to that request point. To determine the “nearest neighbors” the Euclidean distance function
between the testing and training samples is employed. The Euclidean distance is defined as [53,54]:
distEuklidean x0, xj =
n
∑
i=1
xi
0 − xi
j
2
(14)
The k-NN algorithm can be summarized as follows:
1. A positive integer k is definite, along with a new sample.
2. The closest k entries are selected.
3. The most usual classification of these entries is determined and given to the new sample.
The error probability of the model is calculated by the following equation:
PB ≤ Pk−NN ≤ PB +
1
√
ke
(15)
10. Big Data Cogn. Comput. 2018, 2, 35 10 of 17
where PB is the optimal Bayesian error (minimum when k → ∞).
6. Datasets
6.1. Features Extraction
The feature extraction process from the network flow was based on the theoretical background of
the way in which the TCP protocol work and moreover on the dependable submission between the
network and the application layers of the TCP header structure, the three-way handshake method and
the communications security over the SSL protocol [55]. Our research team carried out an extensive
investigation to find the most effective independent variables that describe with maximum correlation
and precision the problem of network traffic analysis under the strict condition of low utilization of
computing power and resources. This determination resulted in the construction of effective datasets,
able to produce an accurate framework that can adapt properly to new data.
The features management and the extraction process of the set of 46 features, including all of the
network flows, is analytically described in [56]. It should be emphasized that this feature extraction
process is also enriched by some novel representation techniques for simple structures and data
modification programmed in the Python programming language.
6.2. Data
The following five datasets have been developed to produce highly multifaceted scenarios that
can be perceived in a network traffic flow and which are appropriate for the training of the proposed
NF3 model.
Firstly, the Network Traffic Analysis (NTA) binary dataset contains 30 independent variables and
two classes (normal or abnormal). This dataset contains 208,629 instances (119,287 normal samples
chosen from the Pcaps which are packet captures obtained from an application programming interface
for capturing network traffic) from National Cyber Watch Mid-Atlantic Collegiate Cyber Defense
Competition and 39,342 abnormal samples chosen from the Contagio Malware Dump [57].
Secondly, the Demystification of Malware Traffic (DMT) multiclass dataset comprises
30 independent variables and five malware classes (Botnet, Crimeware, APT, Attack and CoinMiner).
This dataset contains 168,501 instances chosen from [57] including Pcaps files that captured malware
traffic from honeypots, sandboxes and real-world intrusions.
Thirdly, Encrypted Traffic Analysis (ETI) binary dataset comprises 30 independent variables and
two classes (encrypted or non-encrypted). This dataset contains 166,874 instances (93,024 encrypted
and 73,850 unencrypted) from the “Inter-Service Academy Cyber Defense Competition” served by
Information Technology Operations Center (ITOC), United States Military Academy (West Point, NY,
USA) [58].
Fourthly, Encrypted Traffic Identification (EnTI) multiclass dataset comprises 30 independent
variables and six classes that represent encrypted protocols (Tor, SSH, SSLweb, SSLP2P, SCP, and Skype).
This dataset contains 214,155 instances from [59] including a list of Pcaps file repositories, which are
freely available on the Internet.
Finally, Unencrypted Traffic Identification (UTI) multiclass dataset comprises 30 independent
variables and four classes of unencrypted network protocols (FTP, HTTP, DNS, and SMTP). This dataset
contains 214,155 instances from [59].
The full list of the 30 data features is detailed in [60].
7. Results
In the case of multi-class or binary classification, the estimation of the actual error requires
the probability density of all categories [53,54]. The classification accuracy is estimated by the
employment of a Confusion Matrix (CM) or error matrix that is a detailed matrix that allows
visualization of the performance of a model. A Receiver Operating Characteristic (ROC) curve is a
11. Big Data Cogn. Comput. 2018, 2, 35 11 of 17
graph showing the performance of a classification model at all classification thresholds. The number of
misclassifications is related to the False Positive (FP) and False Negative (FN) indices appearing in
the confusion Matrix. FP is the case where a positive result is wrongly received, and FN is exactly the
opposite. In addition, a True Positive (TP) is a correctly received positive result. A True Negative (TN)
is correctly indicating the condition being tested is not present. The ROC curve plots two parameters:
True Positive Rate (TPR), also known as Sensitivity (Equation (15)), and True Negative Rate (TNR),
also known as Specificity (Equation (16)) [53,54]. The Total Accuracy (TAC) is defined using Equation
(17) [53,54]:
TPR =
TP
TP + FN
(16)
TNR =
TN
TN + FP
(17)
TAC =
TP + TN
N
(18)
The Precision (PRE), Recall (REC) and F-Score indices are defined in Equations (18)–(20) [53,54]:
PRE =
TP
TP + FP
(19)
REC =
TP
TP + FN
(20)
F − Score = 2 ×
PRE × REC
PRE + REC
(21)
The following tables presents an extensive comparison between algorithms.
Table 1. Comparison between algorithms.
Network Traffic Analysis (Binary) (208.629 Instances)
Classifier
Classification Accuracy & Performance Metrics
TAC RMSE PRE REC F-Score ROC_Area
SVM 98.01% 0.1309 0.980 0.980 0.980 0.980
MLFF ANN 98.13% 0.1295 0.981 0.981 0.981 0.994
k-NN 96.86% 0.1412 0.970 0.970 0.970 0.970
RF 97.12% 0.1389 0.972 0.971 0.971 0.971
Ensemble 97.53% 0.1351 0.976 0.975 0.975 0.979
Table 2. Comparison between algorithms.
Demystification of Malware Traffic (Multiclass) (168.501 Instances)
Classifier
Classification Accuracy & Performance Metrics
TAC RMSE PRE REC F-Score ROC_Area
SVM 96.63% 0.1509 0.967 0.967 0.968 0.970
MLFF ANN 96.50% 0.1528 0.981 0.981 0.981 0.965
k-NN 94.95% 0.1602 0.970 0.970 0.970 0.950
RF 95.91% 0.1591 0.972 0.971 0.971 0.960
Ensemble 95.99% 0.1557 0.972 0.972 0.973 0.961
12. Big Data Cogn. Comput. 2018, 2, 35 12 of 17
Table 3. Comparison between algorithms.
Encrypted Traffic Analysis (Binary) (166.874 Instances)
Classifier
Classification Accuracy & Performance Metrics
TAC RMSE PRE REC F-Score ROC_Area
SVM 98.99% 0.1109 0.989 0.990 0.990 0.990
MLFF ANN 99.12% 0.1086 0.998 0.998 0.998 0.998
k-NN 97.84% 0.1372 0.975 0.975 0.978 0.980
RF 98.96% 0.1107 0.989 0.989 0.989 0.990
Ensemble 98.72% 0.1168 0.987 0.987 0.988 0.989
Table 4. Comparison between algorithms.
Encrypted Traffic Identification (Multiclass) (214.155 Instances)
Classifier
Classification Accuracy & Performance Metrics
TAC RMSE PRE REC F-Score ROC_Area
SVM 90.31% 0.1906 0.905 0.905 0.906 0.950
MLFF ANN 92.67% 0.1811 0.930 0.930 0.928 0.960
k-NN 85.19% 0.2032 0.890 0.890 0.890 0.935
RF 91.56% 0.1800 0.920 0.916 0.916 0.930
Ensemble 89.93% 0.1887 0.911 0.910 0.910 0.943
Table 5. Comparison between algorithms.
Unencrypted Traffic Identification (Multiclass) (186.541 Instances)
Classifier
Classification Accuracy & Performance Metrics
TAC RMSE PRE REC F-Score ROC_Area
SVM 99.92% 0.1003 0.999 0.999 0.999 0.999
MLFF ANN 99.91% 0.1008 0.999 0.999 0.999 0.999
k-NN 98.98% 0.1020 0.989 0.989 0.990 0.995
RF 99.93% 0.1001 0.999 0.999 0.999 0.999
Ensemble 99.68% 0.1008 0.996 0.996 0.997 0.998
8. Conclusions
8.1. Discussion
As shown in the above tables, the ensemble method appears to have the same or a slightly lesser
performance across all datasets, compared to the winner (more accurate) algorithm. This fact does not
detract in any case from the value of the proposed method considering that the proposed ensemble
processing approach builds a robust predictive model that reduces the overfit. As stated by this
reasonable analysis, it seems that this method is an appropriate method for complex multifactorial
problems such as the one under consideration.
High precision shows the rate of positive predictions is precise, whereas high recall specifies
the rate of positive events is correctly predicted. Precision is also a measure of correctness or quality,
whereas recall is a degree of completeness or quantity. In all cases, the proposed model had high
average precision and very high recall, meaning the ensemble method is a robust and stable method
that returns substantial results. Correspondingly, the F-Score works greatest if false positives and false
negatives have a comparable cost. In other words, the F-Score is the harmonic average of the precision
and recall, and, in all scenarios, the proposed ensemble method score reaches its best value near 1
(perfect precision and recall). Finally, the high ROC area values of the method provide details on class
distribution and it is related to a cost or benefit analysis of an indicative decision making.
13. Big Data Cogn. Comput. 2018, 2, 35 13 of 17
Tables 1–5 clearly show that the ensemble model is a quite promising method considering
that it offers comparable prediction and supplementary stable models, as the overall behavior of a
multiple model is less noisy than a corresponding single one. It is important to say that analyzing
and identifying some parameters that can determine a type of threat such as cyber-attacks is a partly
subjective, nonlinear and dynamic process. The proposed NF3 for the NGC2SOC may use novel
representation techniques such as 3D-models or immersive visualization through the use of Virtual
Reality (VR) glasses [12] since these techniques allow operators a profound analysis when confronted
with information-overwhelmed situations. Additionally, these techniques incorporate interfaces with
available data sources, e.g., Malware Information Sharing Platform (MISP).
8.2. Innovation
The most significant innovation of NF3 is the proposed creation of a next generation cognitive
computing SOC which will use machine learning technologies to adapt to different contexts with
minimal human supervision. This new approach has the potential to help organizations combat
cybercriminals in real-time. Moreover, traying an all-inclusive analysis of the NF3 model, authors
obviously comprehend that, in the proposed approach, the identification of malicious software or
attacks is done in unproductive time, before troubling or disrupting the operation of the entire
infrastructure. This is a major innovation that generates new standpoints in the implementation of the
IDS/IPS, which adopt intelligent protection against innovative zero-day vulnerabilities. The proposed
framework adds a higher integrity to the security infrastructures improving its cyber resilience
with high identification speed, ease of implementation, minimal human intervention, and minimal
computational resources for network traffic analysis, demystification of malware traffic and encrypted
traffic identification. In addition, the feature extraction and selection procedure is very interesting
and innovative. This feature has occurred after comprehensive research about the network protocols
work in the lower and upper layers of the OSI model. It is important also to highlight that the datasets
occurred after evaluations concerning the boundaries and the determination of normal or abnormal
behavior of the network procedures.
Finally, an impressive innovation of the proposed framework is the ability to identify DoS/DDoS
attacks with high precision. A DoS/DDoS [61–63] attack is orchestrated by creation of high
rate malicious traffic using sources and services of compromised machines establishing multiple
simultaneous connections. One of the most important characteristics of this traffic is the modification
in the number of packets flows in a time-window. For example, the statistical analysis of the packet
count can be used to detect DoS/DDoS attacks. As detailed in [60], the dataset includes features such
as the total number of packets traveling in the flow in a time-window; the minimum, maximum and
average packet length; the minimum, maximum and average interarrival time between two packets;
and the time elapsed from the first packet to the last packet. All these features are useful to identify
anomalies in the network flow related to DoS/DDoS attacks.
8.3. Synopsis
This research paper proposes NF3, an innovative, reliable and highly effective network forensics
tool, employing computational intelligence principles. It is a highly suitable method in cases where
the traditional signature-based applications are computationally infeasible. It is an ensemble machine
learning framework that is based on the optimal combination of four highly efficient and fast learning
algorithms that create a comprehensive intelligent cyber security system proposed for the next
generation cognitive computing SOC. This sophisticated application, combined with the promising
results that have emerged, constitutes a credible innovative proposal for the standardization and
design of improved cyber security infrastructures. Moreover, this implementation is done by using
datasets that respond to specialized, realistic scenarios. In addition, this framework implements
a data analytics approach that attempts to balance latency, throughput, and fault tolerance using
integrated and accurate views of new entrant data flows. It is important to mention that this paper
14. Big Data Cogn. Comput. 2018, 2, 35 14 of 17
proposes a novel intelligence driven network flow forensics framework which uses low computing
utilization resources to network traffic analysis, demystification of malware traffic and encrypted
traffic identification. Given the data dimensionality, it seems suitable for most existing deep learning
solutions, but deep learning methods are extremely computationally expensive in the training process
and are very time-consuming. For example, on a deep convolutional network, the training procedure
can take several days. The most sophisticated models need to spend much time to train using equipped
with expensive GPUs. In contrast, the proposed NF3 model can take few minutes to train completely
from scratch. In addition, the determination of the hyperparameters, topology, training methods, etc.
is a black box and is difficult to comprehend. The classifiers used in the NF3 framework make it much
easier to handle data as well as to understand the architecture which uses few system resources to
train or retrain the models.
8.4. Future Works
Future research could involve a further analysis of the ensemble framework under a
hybrid structure, which will handle many data using batch and stream processing methods
(lambda architecture). In addition, semi-supervised methods algorithms and online learning algorithms
methods could be used to extract and manipulees hidden knowledge between the inhomogeneous
data that arise in network flow analysis. In addition, NF3 could be enhanced by further optimizing the
parameters of the ensemble framework, so that an even more effective, precise, and faster classification
process could be reached. Customized visualization incorporated into the proposed NF3 would assist
SOC operators in understanding the cyber situation. Multi-format representations may support a
reporting system as part of an overall decision mechanism. In addition, it would be important to study
the expansion of this system by implementing the same architecture in a parallel and distributed big
data analysis system such as Hadoop. Finally, a supplementary element that could be considered
in the way of future expansion concerns the operation of NF3 with self-adaptive improvement and
meta-agnostic-learning methods to fully automate the defense against sophisticated cyber-attacks.
Author Contributions: Conceptualization, K.D. and P.K.; Investigation, K.D.; Methodology, K.D. and P.K.;
Software, K.D. and L.I.; Validation, K.D., P.K., N.T., S.L.S. and L.I.; Formal Analysis, K.D., P.K., N.T., S.L.S. and
L.I.; Resources, K.D. and L.I.; Data Curation, K.D., P.K., N.T. and L.I.; Writing—Original Draft Preparation, K.D.;
Writing—Review and Editing, K.D., P.K., N.T., S.L.S. and L.I.; and Supervision, P.K.
Funding: This research received no external funding.
Conflicts of Interest: The authors declare no conflict of interest.
References
1. CISCO. WAN and Application Optimization Solution Guide; Cisco Validated Design Document; Version 1.1;
CISCO Press: Hoboken, NJ, USA, 2008. Available online: www.cisco.com/c/en/us/td/docs/nsite/
enterprise/wan/wan_optimization/wan_opt_sg.pdf (accessed on 1 October 2018).
2. Wang, W.; Zhang, X.; Shi, W.; Lian, S.; Feng, D. Network traffic monitoring, analysis and anomaly detection
[Guest Editorial]. IEEE Netw. 2011, 25, 6–7. [CrossRef]
3. Rudd, E.; Rozsa, A.; Gunther, M.; Boult, T. A Survey of Stealth Malware: Attacks, Mitigation Measures,
and Steps Toward Autonomous Open World Solutions. arXiv 2016, arXiv:1603.06028.
4. Zhang, H.; Papadopoulos, C.; Massey, D. Detecting encrypted botnet traffic. In Proceedings of the 2013 IEEE
Conference on Computer Communications Workshops (INFOCOM WKSHPS), Turin, Italy, 14–19 April 2013;
pp. 3453–1358. [CrossRef]
5. William, H.; Teukolsky, S.A.; Vetterling, W.T.; Flannery, B.P. Section 16.5. Support Vector Machines.
In Numerical Recipes: The Art of Scientific Computing, 3rd ed.; Cambridge University Press: New York,
NY, USA, 2007; ISBN 978-0-521-88068-8.
6. Hubel, D.H.; Wiesel, T.N. Brain and Visual Perception: The Story of a 25-Year Collaboration; Oxford University
Press: Oxford, UK, 2005; p. 106. ISBN 978-0-19-517618-6.
7. Breiman, L. Random Forests. Mach. Learn. 2001, 45, 5–32. [CrossRef]
15. Big Data Cogn. Comput. 2018, 2, 35 15 of 17
8. Hall, P.; Park, B.U.; Samworth, R.J. Choice of neighbor order in nearest-neighbor classification. Ann. Stat.
2008, 36, 2135–2152. [CrossRef]
9. Demertzis, K.; Iliadis, L.; Anezakis, V. Commentary: Aedes albopictus and Aedes japonicus—Two invasive
mosquito species with different temperature niches in Europe. Front. Environ. Sci. 2017, 5, 85. [CrossRef]
10. Demertzis, K.; Iliadis, L. Ladon: A Cyber-Threat Bio-Inspired Intelligence Management System. J. Appl.
Math. Bioinform. 2016, 3, 45–64.
11. Demertzis, K.; Iliadis, L. Evolving Computational Intelligence System for Malware Detection. In Advanced
Information Systems Engineering Workshops; Lecture Notes in Business Information Processing; Springer:
Cham, Switzerland, 2014; Volume 178, pp. 322–334. [CrossRef]
12. Llopis, S.; Hingant, J.; Pérez, I.; Esteve, M.; Carvajal, F.; Mees, W.; Debatty, T. A comparative analysis of
visualisation techniques to achieve cyber situational awareness in the military. In Proceedings of the 2018
International Conference on Military Communications and Information Systems (ICMCIS), Warsaw, Poland,
22–23 May 2018. [CrossRef]
13. Xu, C.; Chen, S.; Su, J.; Yiu, S.M.; Hui, L.C. A Survey on Regular Expression Matching for Deep Packet
Inspection: Applications, Algorithms, and Hardware Platforms. IEEE Commun. Surv. Tutor. 2016, 18,
2991–3029. [CrossRef]
14. Demertzis, K.; Iliadis, L. Evolving Smart URL Filter in a Zone-based Policy Firewall for Detecting
Algorithmically Generated Malicious Domains. In Statistical Learning and Data Sciences; Lecture Notes
in Computer Science; Gammerman, A., Vovk, V., Papadopoulos, H., Eds.; Springer: Cham, Switzerland,
2015; Volume 9047.
15. Yadav, S.; Reddy, A.K.K.; Reddy, A.L.N.; Ranjan, S. Detecting Algorithmically Generated Domain-Flux
Attacks with DNS Traffic Analysis. IEEE/ACM Trans. Netw. 2012, 20, 1663–1677. [CrossRef]
16. Hayes, J. Traffic Confirmation Attacks Despite Noise. arXiv, 2016; arXiv:1601.04893.
17. Mercaldo, F.; Martinelli, F. Tor traffic analysis and identification. In Proceedings of the 2017 AEIT
International Annual Conference, Cagliari, Italy, 20–22 September 2017; pp. 1–6. [CrossRef]
18. Montieri, A.; Ciuonzo, D.; Aceto, G.; Pescapé, A. Anonymity Services Tor, I2P, JonDonym: Classifying in the
Dark. In Proceedings of the 2017 29th International Teletraffic Congress (ITC 29), Genoa, Italy, 4–8 September
2017; pp. 81–89. [CrossRef]
19. Backes, M.; Goldberg, I.; Kate, A.; Mohammadi, E. Provably secure and practical onion routing.
In Proceedings of the 2012 IEEE 25th Computer Security Foundations Symposium (CSF), Cambridge,
MA, USA, 25–27 June 2012.
20. Deepika, B.; Sethi, P.; Kataria, S. Secure Socket Layer and its Security Analysis. Netw. Commun. Eng. 2015, 7,
255–259.
21. Demertzis, K.; Iliadis, L. A Hybrid Network Anomaly and Intrusion Detection Approach Based on Evolving
Spiking Neural Network Classification. In E-Democracy, Security, Privacy and Trust in a Digital World;
Sideridis, A., Kardasiadou, Z., Yialouris, C., Zorkadis, V., Eds.; e-Democracy 2013; Communications in
Computer and Information Science; Springer: Cham, Switzerland, 2014; Volume 441.
22. Demertzis, K.; Iliadis, L. Bio-Inspired Hybrid Artificial Intelligence Framework for Cyber Security.
In Computation, Cryptography, and Network Security; Daras, N., Rassias, M., Eds.; Springer: Cham, Switzerland,
2014.
23. Demertzis, K.; Iliadis, L. Bio-Inspired Hybrid Intelligent Method for Detecting Android Malware. In Advanced
Information Systems Engineering Workshops; Iliadis, L., Papazoglou, M., Pohl, K., Eds.; CAiSE 2014.
Lecture Notes in Business Information Processing; Springer: Cham, Switzerland, 2014; Volume 178.
24. Demertzis, K.; Iliadis, L. SAME: An Intelligent Anti-Malware Extension for Android ART Virtual
Machine. In Computational Collective Intelligence; Núñez, M., Nguyen, N., Camacho, D., Trawi´nski, B.,
Eds.; Lecture Notes in Computer Science; Springer: Cham, Switzerland, 2015; Volume 9330.
25. Demertzis, K.; Iliadis, L. Computational Intelligence Anti-Malware Framework for Android OS. Vietnam J.
Comput. Sci. 2017, 4, 245. [CrossRef]
26. Demertzis, K.; Iliadis, L.S.; Iliadis, V.-D. Anezakis, An innovative soft computing system for smart energy
grids cybersecurity. Adv. Build. Energy Res. 2018, 12, 3–24. [CrossRef]
27. Scandariato, R.; Walden, J. Predicting vulnerable classes in an android application. In Proceedings of the 4th
International Workshop on Security Measurements and Metrics, Lund, Sweden, 21 September 2012.
16. Big Data Cogn. Comput. 2018, 2, 35 16 of 17
28. Chin, E.; Felt, A.; Greenwood, K.; Wagner, D. Analyzing inter-application communication in android.
In Proceedings of the 9th International Conference on Mobile Systems, Applications, and Services, Bethesda,
MD, USA, 28 June–1 July 2011; pp. 239–252.
29. Burguera, I.; Zurutuza, U.; Nadjm-Tehrani, S. Crowdroid: Behavior-based malware detection system for
android. In Proceedings of the 1st ACM Workshop on Security and Privacy in Smartphones and Mobile
Devices, Chicago, IL, USA, 17 October 2011; pp. 15–26.
30. Glodek, W.; Harang, R.R. Permissions-based Detection and Analysis of Mobile Malware Using Random
Decision Forests. In Proceedings of the 2013 IEEE Military Communications Conference, San Diego, CA,
USA, 18–20 November 2013.
31. Zhang, J.; Chen, C.; Xiang, Y.; Zhou, W.; Vasilakos, A.V. An effective network traffic classification method
with unknown flow detection. IEEE Trans. Netw. Serv. Manag. 2013, 10, 133–147. [CrossRef]
32. Joseph, G.; Nagaraja, S. On the reliability of network measurement techniques used for malware traffic
analysis. In Cambridge International Workshop on Security Protocols; Springer: Cham, Switzerland, 2014;
pp. 321–333.
33. Wang, H.T.; Mao, C.H.; Wu, K.P.; Lee, H.M. Real-time fast-flux identification via localized spatial geolocation
detection. In Proceedings of the IEEE Computer Software and Applications Conference (COMPSAC), Izmir,
Turkey, 16–20 July 2012.
34. Tu, T.D.; Guang, C.; Xin, L.Y. Detecting bot-infected machines based on analyzing the similar periodic DNS
queries. In Proceedings of the IEEE 2015 International Conference on Communications, Management and
Telecommunications (ComManTel), DaNang, Vietnam, 28–30 December 2015.
35. Soltanaghaei, E.; Kharrazi, M. Detection of fast-flux botnets through DNS traffic analysis. Sci. Iranica Trans.
D Comput. Sci. Eng. Electr. 2015, 22, 2389.
36. Wright, M.K.; Adler, M.; Levine, B.N.; Shields, C. An analysis of the degradation of anonymous protocols.
In Proceedings of the Network and Distributed Security Symposium, San Diego, CA, USA, 6–8 February
2002.
37. Shmatikov, V.; Wang, M.H. Timing analysis in low-latency mix networks: Attacks and defenses.
In Proceedings of the ESORICS, Hamburg, Germany, 18–20 September 2006.
38. Hsu, C.-H.; Huang, C.-Y.; Chen, K.-T. Fast-flux bot detection in real time. In International Workshop on Recent
Advances in Intrusion Detection; Springer: Berlin/Heidelberg, Germany, 2010.
39. Haffner, P.; Sen, S.; Spatscheck, O.; Wang, D. ACAS: Auto-mated Construction of Application Signatures.
In Proceedings of the ACM SIGCOMM, Philadelphia, PA, USA, 22–26 August 2005; pp. 197–202.
40. Alshammari, R.; Zincir-Heywood, N.A. A flow-based approach for SSH traffic detection, Cybernetics, ISIC.
In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Montreal, QC,
Canada, 7–10 October 2007; pp. 296–301.
41. Holz, T.; Gorecki, C.; Rieck, K.; Freiling, F. Measuring and detecting fast-flux service networks.
In Proceedings of the Network & Distributed System Security Symposium, San Diego, CA, USA, 10–13
February 2008.
42. Almubayed, A.; Hadi, A.; Atoum, J. A Model for Detecting Tor Encrypted Traffic using Supervised Machine
Learning. Int. J. Comput. Netw. Inf. Secur. 2015, 7, 10–23. [CrossRef]
43. Chaabane, A.; Manils, P.; Kaafar, M.A. Digging into Anonymous Traffic: A Deep Analysis of the Tor
Anonymizing Network. In Proceedings of the 4th International Conference on Network and System Security
(NSS), Helsinki, Finland, 21–23 August 2010; pp. 167–174.
44. Chakravarty, S.; Stavrou, A.; Keromytis, A.D. Traffic analysis against low-latency anonymity networks
using available bandwidth estimation. In European Symposium on Research in Computer Security; Springer:
Berlin/Heidelberg, Germany, 2010; pp. 249–267.
45. Chakravarty, S.; Stavrou, A.; Keromytis, A.D. Identifying Proxy Nodes in a Tor Anonymization Circuit.
In Proceedings of the 2nd Workshop on Security and Privacy in Telecommunications and Information
Systems (SePTIS), Bali, Indonesia, 30 November–3 December 2008; pp. 633–639.
46. Mees, W.; Llopis, S.; Debatty, T. Achieving cyber situation awareness through a multi-aspect 3D operational
picture. In Proceedings of the NATO IST-148 Symposium on Cyber Defense Situational Awareness, Sofia,
Bulgaria, 3–4 October 2016.