IPsec is an extension of the Internet Protocol (IP) to include encryption and authentication mechanisms. This gives the Internet protocol the ability to cryptographically secure transport of IP packets over public and unsecure networks. IPsec was developed by the Internet Engineering Task Force (IETF) as an integral part of IPv6. Because the Internet protocol of version 4 originally had no security mechanisms, IPsec was subsequently specified for IPv4.

Components Of IPsec VPNs

• interoperability
• cyptographic protection of the transmitted data
• access control
• data integrity
• Authentication of the sender (user authentication)
• codification
• Key authentication
• Key management (key management)

Behind these components are processes which, combined with each other, offer reliable security for data transmission over public networks. VPN security solutions with high security requirements generally rely on IPsec.

Application Scenarios

• Site-to-Site-VPN / LAN-to-LAN-VPN / Gateway-to-Gateway-VPN
• End-to-Site VPN / Host-to-Gateway VPN / Remote Access VPN
• End-to-end VPN / host-to-host VPN / remote desktop VPN / peer-to-peer VPN

In principle, IPsec is suitable for gateway-to-gateway scenarios. So the connection between networks via a third unsafe network. The host-to-gateway scenario that corresponds to the remote access scenario is also conceivable. However, the complexity of IPsec and some shortcomings of TCP/IP can occasionally cause problems here. The host-to-host scenario is rather untypical, but it is also possible.

IPsec has the disadvantage that it can only tunnel IP packets. It is also not suitable for remote access without additional protocols, as the functions for configuring the IP address, subnet mask and DNS server are missing. Therefore, it makes sense to consider SSL VPN or other solutions and protocols in addition to IPsec when implementing a VPN.

• L2TP over IPsec
• OpenVPN
• SSL VPN
• layer 2 VPN

IPsec: Trust Positions / Security Association

The main component of IPsec is the Security Association between two communication partners. A trust position does not necessarily have to lie between the end points (client) of a transmission path. It is sufficient, for example, if the two routers have a position of trust when coupling two networks. Of course, several positions of trust may exist for a connection.

The trust positions regulate the communication of IPsec. The relatively flexible combinations of trust positions require a very high configuration effort.

In order to establish a secure connection between two stations, some parameters must be exchanged on both sides:

• Type of secure transmission (authentication or encryption)s
• encryption algorithms
• clefss
• Duration of validity of keys

Trust positions are established by exchanging pre-defined keys. Another form is the allocation of certificates by a trust center or an installed certificate server. Keys and certificates should ensure that the person who has a key or a certificate is also the person for whom he or she claims to be. Similar to an identity card with which a person identifies himself to another person.

• PSK – Pre-Shared Keys
• X.509 certificates

Keys or certificates, no matter, both methods require a lot of time and care during setup. The simpler variant is the secret key (password or passphrase). It is important that both endpoints are informed about the IP address, subnet mask, tunnel name and the secret key. In addition, there are parameters that define the details of authentication, encryption and the length of the key.

For authentication with a pre-shared key, an identifier must be configured. The identifier is an additional specification with which the remote stations (gateway and client) can identify themselves. IP addresses, DNS names (FQDN) or e-mail addresses (FQUN) are often used for this purpose.

Tunneling and Encryption

The central functions in the IPsec architecture are the AH protocol (Authentication Header), the ESP protocol (Encapsulating Security Payload) and key management. IPsec meets authenticity, confidentiality and integrity through AH and ESP.

The Authentication Header (AH) and Encapsulating Security Payload (ESP) are available in IPsec for setting up a VPN. Both can be used together or independently. In both procedures a secure transmission takes place.

The AH protocol provides authentication of the data and protocol information to be transmitted. The ESP protocol increases data security depending on the selected encryption algorithm.

• Authentication Header (AH)
• Encapsulation Security Payload (ESP)

IPsec does not require a specific encryption and authentication method. Therefore, problems often arise when different VPN products have to work together.

Key Management With IKE – Internet Key Exchange Protocol

There are two ways to manage and distribute keys within a VPN. In addition to manual key management, the Internet Key Exchange Protocol (IKE) can also be used.

Before protected communication, the communication partners must agree on the encryption methods and keys. These parameters are part of the security association (trust positions) and are automatically negotiated and managed by IKE/IKEv2.

The Internet Key Exchange protocol is used for automatic key management for IPsec. It uses the Diffie-Hellman method to securely generate keys over an insecure network. Based on this procedure, several key exchange procedures were developed, some of which form the basis for Internet Key Exchange.

IKE is based on the Internet Security Association and Key Management Protocol (ISAKMP). ISAKMP is a set of rules that precisely defines the behavior of the participating remote stations. How this is to be done is determined by IKE. IKE’s flexibility is reflected in its complexity. If different IPsec systems cannot exchange security associations, this is usually due to a faulty IKE implementation or missing encryption methods.

Version 2 of the Internet Key Exchange Protocol (IKEv2) simplifies the setup of a VPN. It is much simpler, more flexible and less prone to errors. In particular, the Mobility and Multihoming Protocol (MOBIKE) should ensure that IPSec tunnels function much more reliably in mobile applications.

IKEv2 corrects some vulnerabilities or problems of the previous version. The definition has been summarized in a document, the connection setup has been simplified and many improvements have been added. Overall, IKEv2 is less complex than the previous version. This facilitates implementation, reduces errors and thus increases security.

However, IKEv2 is not backward compatible to IKE. However, both protocols are operated via the same UDP port.

The terms main mode and aggressive mode are often used in connection with IPsec. These are different procedures for negotiating keys.

VPN With IPsec In Practice

The network participants in LAN 1 can access LAN 2 or, conversely, the participants in LAN 2 can access LAN 1 via an encrypted tunnel.

The two firewalls must clearly prove their identity when establishing the connection. This prevents unauthorized access. Communication via the Internet is encrypted. If a third party logs the data packets, they only receive data garbage.

In order for both networks to establish a connection to each other, the IP address of the respective other network must be known. A fixed IP address is therefore necessary to establish a connection, otherwise the establishment of the connection becomes complicated. If the IP address of a network changes, e.g. when establishing a connection to an Internet provider or access network operator, the addresses must be exchanged for new ones. Either manually or via dynamic DNS entries with DDNS.

For the routing between the networks to work, the address ranges within the networks must be different. Since the networks behave like one after interconnection, IP addresses must not occur twice. Therefore, a separate address range, i.e. different subnets, must be configured on both sides in advance.

Problems With NAT

If it is ensured that the VPN remote stations support the same encryption methods and the IKE implementation is error-free, then the key exchange with IKE can still fail due to the NAT routers involved.

If network stations in local area networks (LAN) have private IP addresses and connect to the Internet via NAT router, IPsec has problems with NAT. NAT gives an IPsec packet a new IP address and a different source port. The problem is, if an IPsec packet is changed, then it becomes invalid. This change no longer ensures the integrity of the package. It must be discarded. Of course, no connection can be established this way.

Another problem is that original IP addresses and TCP ports are encrypted. This way the NAT router can’t get to them. And so it is not possible to assign the IP packets to a network station. The information required for this is transmitted during the secure key exchange. And the NAT router has no insight into this. The information is entered in the SPI value (Security Parameters Index). Thus the VPN tunnel could be assigned to a host. However, because of the encrypted transmission of the SPI, the NAT router cannot read this value.

To work around both problems, some routers master the IPsec pass-through process, where the ports are not changed. Unfortunately, passthrough only works with a single client on the network.

IPsec Passthrough (Obsolete)<7h2>

With IPsec passthrough, the port assignment (IKE) is not changed. The IP address of the ESP packages is rewritten for a client. This means that the packages handled with ESP can only be assigned to one connection and one client. Therefore, IPsec passthrough behind a NAT router only works with a single client.

Because usually more than one client wants to operate an IPsec connection, IPsec passthrough is hardly in use anymore. The IPsec extension NAT-Traversal is used. The ESP packages are packed in UDP packages and sent via port 4500. Then NAT routers can rewrite IP addresses and ports.

IPsec With NAT Traversal

Because the original IPsec does not work over NAT routers, it is usually used with the IPsec extension NAT-Traversal. In this scenario, both communication partners exchange various information via the NAT traversal protocol. The ESP packages are then packed into UDP packages and sent via port 4500. Then the NAT routers can easily rewrite IP addresses and ports.

NAT-Traversal is integrated in the IKE protocol (Negotiation of NAT-Traversal in the IKE). While setting up an IKE Security Association, an attempt is made to detect whether a NAT router is between the remote stations. If yes, then the encapsulation of the IPsec packets is negotiated in UDP packets. This means that a UDP header is inserted between the IP header and the ESP header. The full name for this is UDP Encapsulation of IPsec ESP Packets. Usually this process is called IPsec-NAT-Traversal.

For this to work, the responder must have port 4500 (UDP and TCP) open. The responder is the person who responds to the initialization of the IKE Security Association.

IPsec is usually always used with the NAT-Traversal extension. It works with virtually any NAT router.

Procedure for setting up a VPN with IPsec and NAT traversal (simplified)

• First, it is determined whether the remote station has the necessary procedures at all.
• Then the NAT router is tried to be recognized on the transmission path.
• NAT-Keep-Alive is activated on the right side. This ensures that the entries in the table of the NAT routers involved are not deleted due to timeouts.
• If necessary, NAT traversal is activated.
• Then the negotiation of the positions of trust begins. For this purpose, one end of two VPN endpoints generates a request to the target system. The target system responds and initiates the key exchange via Internet Key Exchange (IKE). Both endpoints negotiate encryption and authentication methods. A key or a certificate, which both systems know, establishes a position of trust with each other. The digital master key is then generated for both sides.
• Both sides then define the encryption and authentication methods for data transmission. The key for data transfer is generated with the master key.
• The data is then exchanged and the connection established.

NAT traversal Firewall Issues

To enable IPsec connections with NAT traversal, the firewalls on both sides must allow the encrypted data packets through. Authentication is via the UDP port 500 or 4500, usually these ports must be opened in the firewall.

The encrypted data packets are sent via the IP protocol 50, the ESP (Encapsulated Security Payload), or the IP protocol 51, AH (Authentication Header).

The safe transport of UDP packets is achieved through appropriate measures in ISAKMP. This means that connection-oriented TCP can be dispensed with. In this way, many attempts at attack have no chance.