FortiGate Firewall: Intrusion Prevention System (IPS)

Understanding these core terms will help you navigate and configure FortiGate’s Intrusion Prevention System (IPS) more effectively:

• IPS (Intrusion Prevention System): A security mechanism that monitors network traffic in real time, detects malicious activity, and blocks threats before they can impact the network.
• Signature: A predefined pattern or rule used by IPS to identify known threats and attacks within network traffic.
• False Positive: Legitimate network traffic incorrectly flagged as malicious by the IPS.
• False Negative: Malicious traffic that is not detected by the IPS and is allowed through as normal.
• Action: The response the IPS takes when a threat is detected, such as block, allow, monitor (log), or quarantine.
• Severity Level: The risk rating assigned to a detected threat, typically categorized as critical, high, medium, or low.
• Protocol Decoder: An engine component that analyzes network protocols to identify anomalies or misuse before applying signatures.
• IPS Sensor: A collection of IPS signatures and filters that define what the IPS engine will scan and how it will respond.
• Custom Signature: A user-defined rule created to detect specific threats not covered by predefined signatures.
• Quarantine: The process of isolating affected hosts or network segments to prevent the spread of threats.

FortiGate’s Intrusion Prevention System (IPS) can take a variety of actions when it detects suspicious or malicious network activity. Understanding these actions helps you fine-tune your IPS policies for optimal protection and minimal disruption:

• Block (Drop): Prevents the detected malicious traffic from reaching its destination by dropping the packet. This is the most direct way to stop threats in real time.
• Allow (Pass): Permits the traffic even if it matches a threat signature. This is typically used for traffic that is known to cause false positives or is considered low risk, but is still logged for review.
• Monitor (Log): Records the event in the logs without blocking the traffic. This action is useful for observing potential threats and tuning IPS rules before enforcing stricter actions.
• Quarantine: Isolates the source IP or affected host, restricting its network access to prevent further malicious activity. Quarantined devices may be moved to a separate VLAN or have limited permissions until remediated.
• Reset: Sends a TCP reset packet to terminate the suspicious session immediately. This disrupts the connection between the attacker and the target without blocking future traffic from the same source.

Each IPS signature or rule can be configured with one of these actions, allowing administrators to tailor the response based on the severity and context of the detected threat.

FortiGate’s Intrusion Prevention System (IPS) supports several deployment modes to fit different network architectures and security requirements. Understanding these modes helps you choose the best approach for your environment:

• Inline (NAT/Route Mode): The FortiGate is placed directly in the traffic path, actively inspecting and blocking threats in real time. It acts as a Layer 3 router, routing traffic between network segments and applying IPS policies before forwarding packets. This mode is ideal for perimeter security and provides the strongest protection by stopping threats before they reach their destination.
• Virtual Wire Mode: The FortiGate is deployed transparently between two network segments, functioning like a “virtual wire.” It does not perform routing or NAT, so no changes to network addressing are required. IPS policies are enforced on traffic passing through the paired interfaces, making this mode suitable for environments where minimal network disruption is desired.
• Transparent Mode: The FortiGate operates as a Layer 2 bridge, forwarding traffic between interfaces within the same VLAN or network segment. IPS inspection is applied without altering the network’s IP addressing. This mode is useful for inserting IPS protection into existing networks with minimal configuration changes.
• Passive (Sniffer) Mode: The FortiGate monitors network traffic by connecting to a span port or network tap. It detects and logs threats but does not block or alter traffic. This mode is typically used for detection and analysis in environments where active blocking is not possible or desired.

Each deployment mode offers distinct advantages and is chosen based on the desired balance between security, network complexity, and operational impact. Inline and virtual wire modes provide active threat prevention, while transparent and passive modes are often used for detection and monitoring.

FortiGate’s Intrusion Prevention System (IPS) uses a wide range of signature categories to detect and block various types of threats. Understanding these categories helps administrators tailor protection to their specific environment and threat landscape:

• Exploit: Detects attempts to exploit vulnerabilities in operating systems, applications, or network services. These signatures are crucial for blocking attacks that target known security weaknesses.
• Malware: Identifies and blocks known malicious software, such as viruses, worms, Trojans, and spyware, based on their unique patterns or behaviors.
• Denial of Service (DoS/DDoS): Recognizes traffic patterns associated with attempts to disrupt network services by overwhelming resources, either from a single source (DoS) or multiple sources (DDoS).
• Policy Violations: Flags network traffic that violates organizational security policies, such as unauthorized applications, protocols, or data transfers.
• Reconnaissance: Detects scanning, probing, or information-gathering activities that attackers use to identify potential targets and vulnerabilities within the network.
• Application Attacks: Targets exploits and attacks against specific applications, including web servers, databases, and email systems.
• Backdoor/Remote Access: Identifies attempts to establish unauthorized remote control or backdoor access to systems within the network.
• Phishing: Detects attempts to trick users into divulging sensitive information, such as login credentials or financial data, through deceptive communications.

Each signature category is regularly updated by FortiGuard Labs to address the latest threats, ensuring comprehensive and up-to-date protection for your network.

Applying best practices ensures your FortiGate Intrusion Prevention System (IPS) delivers optimal protection while maintaining network performance. Follow these guidelines to maximize the effectiveness of your IPS deployment:

• Customize IPS Profiles: Tailor IPS profiles to your network’s specific needs. Avoid using generic or default profiles—select only the signatures and filters relevant to your environment, services, and operating systems. This reduces unnecessary resource usage and false positives.
• Regularly Update IPS Signatures: Enable automatic updates and subscribe to FortiGuard IPS Updates to ensure your device receives the latest threat intelligence and protection against emerging threats.
• Enable IPS at Network Boundaries: Apply IPS scanning at network edges and on all interfaces exposed to untrusted or public networks. This helps block threats before they reach critical assets.
• Tune Signatures and Thresholds: Adjust detection thresholds and disable unused or irrelevant signatures. Fine-tune anomaly detection settings to match your typical traffic patterns, minimizing false positives and performance impact.
• Monitor and Analyze Logs: Enable logging and review IPS logs regularly. Use FortiAnalyzer or similar tools for centralized monitoring, alerting, and reporting to quickly identify and respond to threats.
• Segment IPS Policies: Create separate IPS profiles for different network segments, services, or user groups. This allows for granular control and targeted protection.
• Test Changes in a Lab Environment: Before deploying new IPS rules or configurations in production, test them in a controlled environment to assess their impact and avoid disruptions.
• Optimize for Performance: Exclude trusted or internal traffic from IPS inspection when appropriate. During high CPU load, prioritize scanning for high and critical severity threats.
• Review and Adjust Regularly: Periodically reassess your IPS configuration to adapt to evolving threats, changes in network architecture, and business requirements.

By following these best practices, you can achieve a strong balance between security, performance, and operational efficiency in your FortiGate IPS deployment.

Even with a well-configured FortiGate Intrusion Prevention System (IPS), issues can occasionally arise. Understanding common problems and their solutions will help you quickly restore optimal protection and network performance:

• Excessive False Positives:
  • Possible Causes: Overly broad or aggressive IPS signatures, misconfigured policies, or outdated signature databases.
  • Solutions: Review and tune signature settings, disable or adjust noisy signatures, use exception lists for trusted traffic, and ensure IPS signatures are up to date.
• Missed Threats (False Negatives):
  • Possible Causes: Outdated signature database, disabled or misapplied IPS profiles, or insufficient inspection coverage.
  • Solutions: Update IPS signatures, verify that IPS profiles are correctly applied to relevant policies and interfaces, and expand coverage as needed.
• Performance Degradation:
  • Possible Causes: High traffic volume, resource constraints, or overly complex IPS rules.
  • Solutions: Optimize IPS profiles by disabling unnecessary signatures, prioritize high-severity threats, monitor system resources, and consider hardware upgrades if needed.
• Unexpected Traffic Blocking:
  • Possible Causes: IPS policies set to block or quarantine, misconfigured rules, or a bug in the IPS engine.
  • Solutions: Check IPS logs to identify which signature or rule is causing the block, review quarantine settings, and verify that the latest firmware and IPS engine updates are installed.
• IPS Engine Crashes or Firewall Unresponsiveness:
  • Possible Causes: IPS engine bugs, corrupted signature database, or firmware incompatibility.
  • Solutions: Reboot the firewall if unresponsive, update to the latest IPS engine and firmware, and consult Fortinet support for persistent issues.

General Troubleshooting Steps:

1. Check IPS logs and event reports for clues about the issue.
2. Use built-in diagnostic commands (e.g., diagnose ips debug, get system performance status) to gather more information.
3. Review recent changes to IPS profiles, firmware, or network configuration.
4. Update IPS signatures and FortiOS firmware regularly.
5. Test changes in a lab environment before deploying to production.
6. Consult the Fortinet Knowledge Base or contact support for complex or persistent problems.

By following a structured troubleshooting approach, you can quickly identify and resolve most IPS-related issues, ensuring your FortiGate firewall continues to provide effective threat protection.