FortiOS 7.6.8: Certificate-Based IPsec Authentication Using the Wrong Local ID Type

After upgrading to FortiOS 7.6.8, we encountered an issue with one of our certificate-based IPsec tunnels.

Our setup consists of two certificate-based IPsec tunnels. Both tunnels were configured with the default settings and appeared to be identical. While one tunnel continued to work as expected, the other failed during IKE authentication.

The Clue in the strongSwan Logs

The remote side of the connection uses strongSwan. When we reviewed the strongSwan logs, we noticed that the FortiGate was attempting to authenticate using an ASN.1 distinguished name (ASN.1 DN) instead of the expected fully qualified domain name (FQDN).

The Phase 1 configuration already contained the expected FQDN:

config vpn ipsec phase1-interface
    edit "ipsec-cert-tunnel"
        set localid-type fqdn
        set localid "vpn.example.com"
    next
end

However, the logs showed that the FortiGate was still using the certificate identity as an ASN.1 DN.

Observed Behavior with Multiple Certificate-Based IPsec Tunnels

Based on our observations, we suspect that this behavior may occur when more than one certificate-based IPsec tunnel is configured on the FortiGate.

In our environment, two certificate-based IPsec tunnels were configured. While one tunnel continued to work as expected, the second tunnel attempted to authenticate using the certificate's ASN.1 distinguished name instead of the expected FQDN.

We cannot currently confirm whether multiple certificate-based IPsec tunnels are a strict requirement for this behavior. However, the issue was only observed in this configuration and was resolved by explicitly setting the ike-localid-type to fqdn under the local certificate configuration.

The Relevant Certificate Configuration

The selected local certificate also contains settings for the IKE local identity:

config vpn certificate local
    edit "XYCertificate"
        set ike-localid-type fqdn
        set ike-localid "vpn.example.com"
    next
end

For the affected tunnel, we had to explicitly configure the following values:

set ike-localid-type fqdn
set ike-localid "vpn.example.com"

After applying this change, the FortiGate used the expected FQDN during authentication, and the IPsec tunnel was established successfully.

The Important Difference

The relevant settings are located in two different configuration sections:

config vpn ipsec phase1-interface
    set localid-type fqdn
    set localid "vpn.example.com"

and:

config vpn certificate local
    set ike-localid-type fqdn
    set ike-localid "vpn.example.com"

Although the Phase 1 configuration already specified the correct localid-type, the certificate-level setting was decisive in our case. Without explicitly configuring ike-localid-type fqdn, the FortiGate attempted to authenticate using the certificate's ASN.1 DN.

The strongSwan logs were essential for identifying this behavior because they showed which identity type the FortiGate was actually using during the IKE authentication process.

Working Configuration

The final configuration included the following settings:

config vpn ipsec phase1-interface
    edit "ipsec-cert-tunnel"
        set authmethod signature
        set certificate "XYCertificate"
        set localid-type fqdn
        set localid "vpn.example.com"
    next
end

The corresponding certificate configuration was:

config vpn certificate local
    edit "XYCertificate"
        set ike-localid-type fqdn
        set ike-localid "vpn.example.com"
    next
end

The configured FQDN must match the identity expected by the remote strongSwan configuration.

Conclusion

The issue was caused by the FortiGate using an ASN.1 DN instead of the expected FQDN for certificate-based IPsec authentication.

Based on our observations, this behavior may be related to configurations with multiple certificate-based IPsec tunnels. However, we cannot currently confirm whether the presence of more than one certificate-based tunnel is a strict requirement for reproducing the issue.

The solution was to explicitly configure the correct identity type under the local certificate:

set ike-localid-type fqdn
set ike-localid "vpn.example.com"

When troubleshooting certificate-based IPsec tunnels on FortiOS 7.6.8, it is therefore important to check the ike-localid-type under:

config vpn certificate local

Even if localid-type fqdn is already configured under the IPsec Phase 1 settings, the certificate-level ike-localid-type may determine which identity type the FortiGate actually uses during authentication.