Threat Hunting with IPS/IDS Walkthrough — LetsDefend

by Yusuf Talha

Introduction

Threat hunting is a proactive cybersecurity strategy. Rather than focusing solely on known threats, security teams hunt for advanced and stealthy attacks that have yet to be detected. Intrusion prevention system (IPS) and intrusion detection system (IDS) technologies play a critical role in this hunting process.

• Intrusion Detection System (IDS) : IDS is a security technology that monitors network traffic and system events to detect suspicious activities. It typically operates passively, meaning it detects attacks but does not take action. Detected suspicious activities are reported to security teams.
• Intrusion Prevention System (IPS) : IPS goes a step further than IDS. It monitors network traffic, detects suspicious activities, and takes proactive measures against these threats, including actions such as blocking malicious traffic or terminating sessions.
  
  

The Role of IPS/IDS in Threat Hunting

In the threat hunting process, security tools or systems such as IPS (Intrusion Prevention System) and IDS (Intrusion Detection System) provide significant support in both detecting existing threats and understanding how these threats operate. The functions that IPS and IDS systems perform in the threat hunting process are as follows:

Incident Detection and Early Alerts

IPS/IDS systems continuously monitor network traffic and system behavior, enabling them to detect attackers’ movements in the early stages. These systems provide cybersecurity teams with initial alerts about potential threats, allowing attackers to be stopped before they can execute their plans. These alerts serve as a starting point for threat hunting processes and identify situations that require deeper investigation. Alerts typically include the following scenarios:
1. Suspicious Traffic: A sudden increase in traffic volume from a specific IP address or system is often a notable event. For example, an internal IP address transferring significantly more data than usual to external systems could indicate data exfiltration or an attack attempt. Such sudden spikes, especially in systems that normally have low traffic volumes, can be a sign of a serious threat.

2. Unexpected Connections: Connections made by devices within the system or network to unexpected IP addresses are among the early indications of cyber attacks. For example, a server on the internal network connecting to a remote IP address it doesn’t normally communicate with could indicate backdoor activity or malware. Such situations are common with phishing attacks or the use of remote access tools (RATs).

3. Unexpected Data Transfers: A device or system that does not normally perform certain data flows or transfers suddenly starts sending or receiving large amounts of data could indicate a security breach or data leakage. Such unexpected data transfers, especially connections from critical systems to the outside world, should raise suspicions of malware activity or insider threats. IPS/IDS systems can detect such transfers and provide alerts to identify data theft or malicious activity hidden in encrypted traffic.

4. Suspicious Protocols: The sudden use of rarely or never-used protocols on a particular network segment can be an indication that attackers are using unconventional methods to avoid detection. For example, the sudden appearance of FTP or SSH traffic on a system that normally has only HTTP traffic could indicate that attackers are attempting to steal data or send commands. Such unexpected protocol usage should be immediately detected and alerted by IPS/IDS systems.

These alerts not only serve as a starting point for threat hunters, but also provide guiding information to help understand the potential target and how the attack was carried out. Detailed analysis of alerts can help understand the attacker’s path and prevent potential threats in advance.

Comprehensive Data Collection

IPS/IDS systems collect and analyze a wide range of data during the threat hunting process. In order to track threats and detail the movements of attackers, this data is critical. The data collected includes:

• Packet Records: Each data packet on the network is recorded. These records include the data carried by each packet, source and destination IP addresses, protocols used, and payloads. Packet records are particularly useful for investigating suspicious traffic or attack vectors from specific sources. For example, identifying the types of packets heavily used in a DDoS attack provides critical information to mitigate the attack.
• Connection Information: IPS/IDS systems collect detailed connection logs that show which devices communicate with which servers, when, and how often. This information is essential for detecting deviations from normal behavior and identifying potential threats. For example, an internal server suddenly communicating frequently with different external IP addresses could indicate data leakage or backdoor access.
• Attack Patterns: Known threat signatures are loaded into IPS/IDS systems, which compare network traffic against these signatures to detect potential attacks. Attack patterns help threat hunters link past attacks to current events. In addition, these signatures can reveal whether previously used methods are being reused. For example, detecting a new version of known malware can be achieved by effectively using signatures from past attacks.

The collected data helps threat hunters determine when an attack began, how it spread, and what systems were targeted. In addition, the attacker’s intentions and next steps can be predicted through the correlation analysis of this data.Press enter or click to view image in full size.

Anomaly and Signature-Based Detection

IPS/IDS systems use two main methods to detect threats: signature-based detection and anomaly-based detection. These two methods are used together to identify different types of threats and are critical in the threat hunting process.

Signature-based Detection: Signature-based detection relies on known signatures of specific types of attacks. These signatures are created based on data from past attacks or predefined threats. Signature-based detection is particularly effective for quickly identifying common and known attacks. For example, signature-based systems can quickly detect the characteristic traces left in network traffic by known malware. However, this method may be limited against new and previously unseen attacks because such attacks use undefined signatures.

Anomaly-based Detection: Anomaly-based detection identifies deviations from normal network behavior and detects potential threats. This approach is especially important for detecting unknown or zero-day attacks. Anomaly detection looks for changes in network behavior at specific times or with specific user activity. For example, a server with normally low traffic suddenly transferring large amounts of data could be flagged as an anomaly and require further investigation. By using continuous learning and advanced analysis techniques, anomaly detection offers a versatile approach to threat hunting.

Threat hunting often uses these two detection methods together. Signature-based detection provides rapid identification of known threats, enabling immediate response. Anomaly-based detection uncovers unknown threats and provides a broader security perspective. Combined, they enhance security teams’ ability to detect both known and unknown threats, strengthening cybersecurity defenses.

Using IPS/IDS in the Threat Hunting Process

In the threat hunting process, the effective use of Intrusion Prevention System/Intrusion Detection System (IPS/IDS) systems significantly enhances the threat hunter’s ability to detect, analyze, and respond to threats. The following content describes how IPS/IDS systems are used in the threat hunting process. This lesson focuses on investigations performed on IPS/IDS systems based on hypotheses developed by threat hunters.

Data Collection and Log Analysis

Importance of IPS/IDS Logs

IPS/IDS systems generate a large amount of network traffic data. This data is critical to identifying potential threats during the threat hunting process.

Logs contain unusual activity on the network or behavior that matches known attack signatures. These logs serve as a starting point for threat hunters.

Data Collection Process

• Centralized Logging: Collecting IPS/IDS logs in a centralized system simplifies analysis and correlation processes.
• Log Formats and Structure: Understanding the log format, determining where specific information is located, and identifying which information can be used for threat hunting are essential.
• Filtering and Prioritizing Logs: Filtering and prioritization techniques are used to make sense of the large volume of data in logs, allowing analysts to focus on only critical events.
  
  

Analysis Techniques

1. Basic Log Analysis: Simple search and sorting operations on logs to identify anomalies.

2. Advanced Log Analysis: Using specialized software or scripts to create correlations, trend analyses, and behavior models from logs.

3. Automated Analysis Tools: Automatically analyzing logs using machine learning and AI-supported tools to detect potential threats.

Anomaly Detection

• Defining Normal Behavior: Defining normal traffic and user behavior on the network is necessary for detecting anomalies.
  
  

Anomaly Detection Methods

• Statistical Methods: Statistically analyzing traffic data to define anomalies.
• Machine Learning-Based Methods: Using supervised and unsupervised learning algorithms for anomaly detection.
  
  

Anomaly Detection and IPS/IDS

Detected anomalous traffic is logged by IPS/IDS systems, and analysis of these logs enables early detection of potential attacks.

Detection of Attack Indicators

Signature-Based Detection

IPS/IDS systems detect and log traffic that matches known attack signatures.

• Updating Signature Databases: Keeping attack signatures up-to-date is crucial for detecting new threats.
  
  

Anomaly-Based Attack Detection

Often associated with anomaly detection, it is the detection of potential threats that fall outside of known attack signatures.

• Behavioral Analysis: Monitoring changes in attackers’ behavior over time and predicting attacks based on these behaviors.
  
  

Real-Time Analysis

• Real-Time Traffic Monitoring: IPS/IDS systems monitor network traffic in real time to detect threats requiring immediate intervention.
• Automatic Alerts: Threat hunters define automatic alerts for specific events or anomaly detections, enabling quick action.
• Incident Response Procedures: Establishing and implementing emergency response steps for threats detected in real time.
  
  

Retrospective Analysis

1. Log Archiving: Archiving IPS/IDS logs for a specific period to allow future analysis.

2. Historical Data Review: Examining past logs to detect previously overlooked threats and understand how attacks evolved.

3. Correlation and Trend Analysis: Performing correlation and trend analysis on logs to determine if events from different time periods are related.

4. Retrospective Hypothesis Testing: Threat hunters test their hypotheses on past logs to determine the starting points of attacks or the techniques used.

Hypothesis Testing and Validation with IPS/IDS Systems

Creating a Hypothesis

• Role of Hypotheses in Threat Hunting: Threat hunters develop hypotheses to understand attack or threat scenarios.
• Hypothesis Testing with IPS/IDS Systems: Testing developed hypotheses on IPS/IDS logs to check if the signs and behaviors validate the hypothesis.
  

Validation and Reporting

• Validation Process: Validating hypotheses using various data sources and analyses.
• Reporting: Reporting findings from the threat hunting process and sharing them with management and other security teams.
• Action Plans: Determining and implementing security measures based on validated hypotheses.
  

Conclusion

This lesson discussed how threat hunters’ hypotheses are tested on IPS/IDS systems, how data collection and analysis processes are managed, and how actions are taken against detected threats.