Troubleshooting FortiGate VPN Tunnels: A Comprehensive Guide

When troubleshooting a FortiGate site-to-site VPN, it is essential to understand and systematically check each of the following core components. These elements are presented in the logical order you should consider during the troubleshooting process:

• VPN Tunnel Type:
  Determine if the VPN is route-based (interface-based) or policy-based. Route-based VPNs use a dedicated virtual IPsec interface and require security policies for traffic flow in each direction, while policy-based VPNs use specific security policies to trigger encryption for defined traffic flows. Also, identify if the tunnel uses IKEv1 or IKEv2 for negotiation, as this impacts compatibility and troubleshooting steps.
• Peer Configuration:
  Verify the peer IP address and ensure it matches on both ends. Confirm WAN interface assignments and that the remote gateway is reachable. If the peer IP changes or is unreachable, the tunnel will not establish.
• Phase 1 Proposal:
  Check that encryption, authentication, Diffie-Hellman group, pre-shared key (PSK), and lifetime settings match exactly on both peers. Confirm NAT Traversal (NAT-T) settings if either side is behind NAT. Use packet capture on the WAN interface to confirm ISAKMP (UDP 500/4500) traffic is reaching the firewall.
• Phase 2 Proposal:
  Ensure that encryption, authentication, Perfect Forward Secrecy (PFS), lifetime, and Quick Mode Selectors/Proxy IDs match on both sides. Mismatches here will prevent phase 2 from establishing.
• Network Connectivity:
  Use ping and traceroute to confirm bidirectional connectivity between gateways. If ISAKMP traffic is not seen in packet captures, investigate routing or ISP issues.
• NAT Configuration:
  If NAT is in use, ensure NAT-T is enabled and configured consistently. Confirm that NAT rules do not inadvertently alter VPN traffic or prevent tunnel establishment.
• Routing:
  Verify that appropriate static or dynamic routes exist to direct traffic through the VPN tunnel. Incorrect or missing routes will prevent end-to-end connectivity.
• Security Policies:
  For route-based VPNs, ensure Accept policies are in place for both inbound and outbound traffic between the IPsec interface and internal networks. For policy-based VPNs, confirm that IPSEC action policies specify the correct source, destination, and services. Policies must permit the intended traffic and be correctly ordered.
• Proxy IDs / Quick Mode Selectors:
  These define the local and remote subnets permitted over the tunnel. Ensure selectors match on both ends to allow traffic flow. Mismatches here are a common cause of phase 2 issues.
• Packet Capture and Traffic Analysis:
  Use packet capture tools to verify ISAKMP (UDP 500/4500) traffic during phase 1 and ESP or actual data traffic during phase 2. This helps identify where traffic may be blocked or misrouted.
• MTU and Fragmentation:
  Check for MTU mismatches or fragmentation issues, especially if large packets fail to traverse the tunnel. Adjust MTU settings on VPN interfaces as needed.
• VPN Status Monitoring Tools:
  It is best practice to implement VPN tunnel monitoring and alerting. Automated monitoring ensures you are notified immediately when a tunnel goes down, allowing for prompt troubleshooting and minimizing downtime.

Before beginning any troubleshooting, gather the following prerequisites to ensure an efficient and accurate process:

• Required Information from Client/Customer:
  Obtain all relevant VPN details, including:
  • Remote peer IP address
  • Pre-shared key (PSK) or certificate details
  • Phase 1 and Phase 2 proposal settings (encryption, authentication, DH group, PFS, lifetimes)
  • Local and remote network subnets (encryption domains/proxy IDs)
  • Any NAT requirements or special routing considerations
  • Contact information for the remote site’s technical team
• Access and Permissions Needed:
  Ensure you have:
  • Administrative access to the FortiGate firewall (GUI and CLI)
  • Authorization to make configuration changes and run diagnostic commands
  • Access to any upstream or downstream network devices that may impact VPN traffic (routers, ISPs, etc.)
  • Permission to perform packet captures and generate test traffic if needed
• Network Topology and IP Addressing:
  Collect or diagram:
  • The full network topology, including all relevant FortiGate devices, WAN links, and intermediary devices
  • IP addressing schemes for local and remote networks
  • Details on overlapping subnets, NAT, or routing domains that could affect VPN negotiation or traffic flow
  • Any recent changes to the network that could impact VPN connectivity

Follow these steps to systematically diagnose and resolve Phase 1 (IKE) issues in a FortiGate site-to-site VPN tunnel:

• Initial VPN Status Check
  • CLI: To get a summary of all VPN tunnels and their status, run:
    get vpn ipsec tunnel summary
    To view detailed information about all tunnels:
    diagnose vpn tunnel list
  • GUI:
    Navigate to VPN > IPsec Tunnels. Here, you can see the list of tunnels, their status (up/down), and additional details. To check or edit a specific tunnel, select it and click Edit. You can also use the Bring Up button to manually initiate the tunnel.
  • Traffic Logs:
    Go to Log & Report > Events > VPN Events to review recent VPN-related events, including connection attempts and errors.
• Phase 1 (IKE) Troubleshooting
  • Verify Peer IP and Connectivity:
    • Check that the remote peer IP is correct on both ends.
    • Test network connectivity using traceroute (replace with your peer IP and local source IP):
      execute traceroute-options source <local-ip>
execute traceroute <peer-ip>
    • Perform a packet capture on the WAN interface to confirm ISAKMP (UDP 500/4500) traffic is reaching the firewall (replace <peer-ip> as needed):
      diagnose sniffer packet any "host <peer-ip> and (port 500 or port 4500)" 4 0 l
  • Common Phase 1 Issues:
    • Wrong Peer IP: Double-check the configured remote gateway IP on both FortiGates.
    • Mismatched Pre-Shared Key (PSK): Ensure the PSK matches exactly on both sides.
    • Proposal Mismatch: Confirm that encryption, authentication, Diffie-Hellman group, and lifetime settings are identical on both peers.
    • NAT Traversal (NAT-T) Settings: If either side is behind NAT, confirm NAT-T is enabled and that you are capturing on UDP port 4500.
    • ISP/Carrier Issues: If no ISAKMP traffic is seen on the WAN interface, verify the ISP is not blocking UDP 500/4500 or that the remote site is reachable.
  • Debugging Phase 1 Negotiation:
    • Enable IKE debugging to view detailed negotiation logs (replace <peer-ip> as needed):
      diagnose vpn ike log-filter clear
diagnose vpn ike log-filter dst-addr4 <peer-ip>
diagnose debug application ike -1
diagnose debug console timestamp enable
diagnose debug enable
    • Attempt to bring up the tunnel from the GUI or CLI to trigger negotiation and observe logs.
    • After collecting logs, stop debugging to avoid excessive log generation:
      diagnose debug disable
  • GUI Steps for Phase 1 Debugging:
    • Navigate to VPN > IPsec Tunnels, select the tunnel, and use the Bring Up option to initiate the connection.
    • Check Log & Report > Events > VPN Events for error messages or negotiation failures.

After Phase 1 is established, Phase 2 issues are the most common reasons for traffic not passing through a FortiGate site-to-site VPN. Follow these steps to diagnose and resolve Phase 2 and advanced VPN issues:

• Checking Phase 2 Status
  • CLI: To check the status of Phase 2 SAs for a specific tunnel, run:
    diagnose vpn tunnel list name <tunnel_name>
diagnose vpn ike gateway list name <tunnel_name>
  • GUI:
    Navigate to VPN > IPsec Tunnels, select the tunnel, and review the Status and Monitor tabs. Look for the Phase 2 selectors and ensure the tunnel is up.
• Common Phase 2 Issues
  • Proposal Mismatch: Ensure that encryption, authentication/hash, PFS group, and lifetime settings match exactly on both sides.
  • Security Policy Blocks: Confirm that firewall policies allow traffic between the local and remote subnets over the IPsec interface. For route-based VPNs, ensure Accept policies are in place in both directions. For policy-based VPNs, verify the IPsec action policies.
  • Routing and Proxy ID/Quick Mode Selectors: Make sure static or dynamic routes point traffic into the VPN tunnel. Proxy IDs (Quick Mode Selectors) must match on both peers for the subnets you want to communicate.
  • NAT Configuration: If NAT is involved, verify NAT-T is enabled and that NAT rules do not interfere with VPN traffic.
• Debugging Phase 2 Negotiation
  • Enable IKE debug to capture Phase 2 negotiation details (replace <peer-ip> as needed):
    diagnose vpn ike log-filter clear
diagnose vpn ike log-filter dst-addr4 <peer-ip>
diagnose debug application ike -1
diagnose debug console timestamp enable
diagnose debug enable
  • Generate traffic between the encryption domains (e.g., ping from a host behind one firewall to a host behind the other) to trigger Phase 2 negotiation.
  • After collecting logs, stop debugging:
    diagnose debug disable
  • GUI:
    Check Log & Report > Events > VPN Events for Phase 2 errors or negotiation failures.
• Advanced Troubleshooting
  • Packet Capture for Phase 1 and Phase 2:
    To capture Phase 1 (ISAKMP) traffic:
    diagnose sniffer packet any "host <peer-ip> and (port 500 or port 4500)" 4 0 l
    To capture Phase 2 (ESP/data) traffic:
    diagnose sniffer packet any "host <peer-ip> and proto 50" 4 0 l
  • Analyzing Encryption Domains:
    Verify that the local and remote subnets (proxy IDs/selectors) defined in Phase 2 match on both sides. Mismatches will prevent traffic from passing even if the tunnel is up.
  • MTU and Fragmentation Issues:
    If large packets fail to traverse the tunnel, adjust the MTU on the VPN interface:
    config system interface
edit <tunnel_interface>
set mtu-override enable
set mtu <value>
end
  • Logs and Diagnostic Tools:
    Always review Log & Report > Events > VPN Events in the GUI for error messages. Use CLI debug and packet capture outputs to correlate events and pinpoint where negotiations or traffic flows are failing.

Use this step-by-step troubleshooting flow to systematically isolate and resolve issues with FortiGate site-to-site VPN tunnels. Each step includes CLI and GUI checks, as well as common issues and next actions.

• 1. Check VPN Tunnel Status
  • CLI:
    get vpn ipsec tunnel summary
  • GUI:
    Navigate to VPN > IPsec Tunnels and review tunnel status.
  • If tunnel is up: Proceed to traffic flow checks.
    If tunnel is down: Continue to Phase 1 troubleshooting.
• 2. Phase 1 (IKE) Troubleshooting
  • Check Phase 1 Status
    • CLI:
      diagnose vpn ike gateway list
    • GUI:
      VPN > IPsec Tunnels > Status tab, review Phase 1 status.
  • Peer IP and Connectivity
    • CLI:
      execute traceroute-options source <local-ip>
execute traceroute <peer-ip>
    • Packet Capture:
      diagnose sniffer packet any "host <peer-ip> and (port 500 or port 4500)" 4 0 l
  • Common Phase 1 Issues
    • Wrong peer IP
    • Mismatched pre-shared key (PSK)
    • Proposal mismatch (encryption, authentication, DH group, lifetime)
    • NAT-T settings (UDP 4500 if behind NAT)
    • No ISAKMP traffic seen (possible ISP block or routing issue)
  • Debugging Phase 1 Negotiation
    • CLI:
      diagnose vpn ike log-filter clear
diagnose vpn ike log-filter dst-addr4 <peer-ip>
diagnose debug application ike -1
diagnose debug console timestamp enable
diagnose debug enable
    • Attempt to bring up the tunnel and observe logs.
    • Stop debug:
      diagnose debug disable
  • If Phase 1 is up, proceed to Phase 2 troubleshooting.
• 3. Phase 2 (IPsec) Troubleshooting
  • Check Phase 2 Status
    • CLI:
      diagnose vpn tunnel list name <tunnel_name>
    • GUI:
      VPN > IPsec Tunnels > Status or Monitor tab.
  • Common Phase 2 Issues
    • Proposal mismatch (encryption, hash, PFS, lifetime)
    • Proxy ID/Quick Mode selector mismatch
    • Routing issues (missing or incorrect static/dynamic routes)
    • Security policy blocks (check Accept/IPsec policies)
    • NAT configuration errors
  • Debugging Phase 2 Negotiation
    • CLI:
      diagnose vpn ike log-filter clear
diagnose vpn ike log-filter dst-addr4 <peer-ip>
diagnose debug application ike -1
diagnose debug console timestamp enable
diagnose debug enable
    • Generate traffic between encryption domains to trigger Phase 2.
    • Stop debug:
      diagnose debug disable
  • If Phase 2 is up, but traffic is not passing, proceed to advanced checks.
• 4. Advanced Troubleshooting
  • Packet Capture
    • Phase 1 (ISAKMP):
      diagnose sniffer packet any "host <peer-ip> and (port 500 or port 4500)" 4 0 l
    • Phase 2 (ESP/data):
      diagnose sniffer packet any "host <peer-ip> and proto 50" 4 0 l
  • Analyze Encryption Domains
    • Ensure local and remote subnets (proxy IDs) match on both sides.
  • MTU and Fragmentation Issues
    • CLI:
      config system interface
edit <tunnel_interface>
set mtu-override enable
set mtu <value>
end
  • Logs and Diagnostic Tools
    • GUI:
      Log & Report > Events > VPN Events for error messages and negotiation failures.
    • CLI:
      Use debug and sniffer outputs to correlate events and pinpoint issues.
  • Still Not Working?
    • Check for upstream firewall or ISP blocks.
    • Review recent network changes or firmware updates.
    • Consult Fortinet documentation or open a TAC case with relevant logs.

Use this as a quick reference to understand the negotiation differences and flow between IKEv1 and IKEv2 when troubleshooting FortiGate site-to-site VPNs.

• IKEv1 Negotiation Flow
  • Phase 1 (Main Mode or Aggressive Mode)
    • Main Mode: 6 message exchanges (3 pairs)
    • Aggressive Mode: 3 message exchanges (faster, less secure)
    • Negotiates IKE SA parameters (encryption, authentication, DH group, lifetime)
    • Authenticates peers (PSK/certificates)
    • Establishes secure channel for Phase 2
  • Phase 2 (Quick Mode)
    • 3 message exchanges
    • Negotiates IPsec SA parameters (encryption, hash, PFS, lifetime)
    • Establishes IPsec SAs for data traffic
    • Uses Proxy IDs/Quick Mode selectors to define encryption domains
  • Key Troubleshooting Points:
    • Check if Main or Aggressive mode is used and matches on both peers
    • Mismatch in proposals or authentication will cause negotiation failure
    • Phase 1 must complete before Phase 2 can start
• IKEv2 Negotiation Flow
  • Initial Exchange
    • 4 messages (2 pairs):
      IKE_SA_INIT (negotiates IKE SA parameters, exchanges keys)
      IKE_AUTH (authenticates peers, establishes IKE SA and first IPsec SA)
    • More efficient than IKEv1, no Main/Aggressive mode distinction
  • CREATE_CHILD_SA Exchange
    • Used to create additional IPsec SAs or rekey existing SAs
    • Each CREATE_CHILD_SA is a 2-message exchange
  • INFORMATIONAL Exchange
    • Used for control messages, deletions, notifications, and errors
  • Key Troubleshooting Points:
    • Fewer messages, faster negotiation, and less bandwidth than IKEv1
    • No Main/Aggressive mode, no XAuth, no Peer Options/Local ID
    • Proposal and authentication mismatches will cause negotiation to fail
• CLI Commands to View Negotiation Progress
  • diagnose vpn ike gateway list # Check IKEv1/IKEv2 status
  • diagnose debug application ike -1
diagnose debug enable
  • diagnose debug disable # Stop debugging
• GUI Steps
  • VPN > IPsec Tunnels > Status tab: Review negotiation status and logs for both IKEv1 and IKEv2 tunnels.
  • Log & Report > Events > VPN Events: Review detailed negotiation steps and errors.