IPsec VPN: AH, ESP, IKEv2, and Tunnel Modes Explained

IPsec VPN: AH, ESP, IKEv2, and Tunnel Modes Explained

Ryan Foster
April 19, 2026· Updated October 4, 2026· 9 min read

An IPsec VPN is a VPN that protects traffic at the IP layer using the IPsec security framework: the two ends first authenticate each other and agree on keys with IKEv2, then ESP encrypts and integrity-checks each packet that enters the tunnel. IPsec is not one single protocol but a set of network-layer mechanisms, and you will often see it in enterprise remote access, site-to-site connectivity, and some system-level VPN setups.[1][2]

If the networking terms in this article feel abstract, the complete VPN guide explains the tunnel, exit IP, and encryption model before you troubleshoot this specific case.

Here is the quick conclusion: IPsec VPNs are still relevant in 2026. They are mature, stable, and well suited to network-layer tunnels. The tradeoff is that configuration and network compatibility are usually more complex than with a consumer VPN app, especially once NAT, port allowances, and gateway policies get involved.[1][3]

Key Takeaways

  • IPsec is not a single algorithm. It is a security architecture that provides confidentiality, integrity, and authentication at the IP layer.[1]
  • AH, ESP, SA, and IKEv2 are the core concepts you need to understand IPsec VPNs.[1][2]
  • IPsec is common in site-to-site and enterprise remote access, not just consumer “change my region” VPN use.[1][4]
  • It supports transport mode and tunnel mode, which solve different problems.[1]
  • NAT makes IPsec more complicated, which is why you often see terms like NAT-T and router passthrough.[3][5]

If you have not separated “protocol” from “tunnel” yet, start with VPN protocol comparison: WireGuard vs OpenVPN vs IKEv2 and What is a VPN tunnel? How it works, types, and common misconceptions. IPsec will make more sense afterward because it is so closely tied to enterprise networking.

What exactly is IPsec?

RFC 4301 defines IPsec as the security architecture for the Internet Protocol. Its core goal is to provide confidentiality, data integrity, access control, and data-origin authentication for IP datagrams.[1]

That breaks down into four practical jobs:

  • keep data content from being casually read;
  • make sure data was not changed in transit;
  • verify the identity of the communicating parties;
  • allow only policy-matching traffic into the tunnel.

IPsec is best understood as a set of network-layer security building blocks. An IPsec VPN uses those blocks to build a VPN tunnel.

What parts make up an IPsec VPN?

An IPsec VPN is easier to understand as four parts that each do one job:

ComponentWhat it doesHow to think about it
AHIntegrity and authenticationProves the packet was not changed and the source is trusted
ESPEncryption, and optionally integrity protectionWraps the payload
SASecurity AssociationThe shared agreement on algorithms, keys, and parameters
IKE / IKEv2Negotiates and maintains SAs“Agree on the rules first, then start encrypted communication”[2]

In most modern deployments, you are more likely to hear about ESP + IKEv2. AH has worse NAT compatibility, while ESP with IKEv2 better matches real-world deployments today.[3]

How does an IPsec VPN work?

You can think of it in six steps:

  1. Both sides use IKEv2 to negotiate identity, algorithms, and keys.
  2. They establish a Security Association.
  3. IP traffic that matches policy is selected.
  4. Traffic is encrypted and encapsulated with ESP or related mechanisms.
  5. Packets cross the public internet.
  6. The other side decrypts them and checks integrity using the same agreement.[1][2]

That is why IPsec is more complex than “just encryption.” It includes negotiation, policy matching, session maintenance, and rekeying.

If you want the tunnel concept first, read What is a VPN tunnel? How it works, types, and common misconceptions.

If you want a personal VPN rather than an IPsec gateway, you do not run this negotiation by hand. AethoVPN is set up through its own client or setup guide instead of a manual IPsec profile: install the client for Windows, Linux or Android, or follow the setup guide for iPhone, iPad and Mac on a Pro or Premium plan, then connect from the in-app location list. The pre-shared keys, certificates and IKE proposals in the steps above stay with your employer's gateway or a self-built server, so keep them out of unrelated personal client profiles; gateway credentials are not interchangeable with a personal AethoVPN account. New users can start the 3-day free trial before installing the client.

Transport mode vs tunnel mode: what is the difference?

This is a must-know distinction for IPsec VPN.

ModeWhat it protectsMore common use case
Transport modeThe payload of the original IP packetEnd-to-end host communication
Tunnel modeThe entire original IP packetGateway-to-gateway and remote access VPNs[1][2]

Tunnel mode is more common in enterprise VPN scenarios because it fits:

  • branch-to-headquarters connectivity;
  • employees connecting back to the company from outside;
  • wrapping whole internal network segments into a unified tunnel.

Why is IPsec so often tied to NAT, MTU, and router compatibility?

Because IPsec works at the IP layer, while real networks are full of NAT.

RFC 3715 and RFC 3947 both discuss compatibility between IPsec and NAT. In simple terms, some IPsec packet properties conflict with address and port translation, which is why NAT Traversal became necessary.[3][5]

In practice, when IKE detects a NAT device between the two ends, it moves from UDP port 500 to UDP port 4500 and continues the exchange there, which is why a firewall in front of an IPsec gateway usually has to allow both ports.[2][5]

That leads to practical symptoms:

  • IPsec may connect unreliably on some networks.
  • Routers may need explicit passthrough or NAT-T support.
  • Extra encapsulation adds overhead and can reduce effective payload size.

That is why enterprise teams often see problems like “ping works but the app does not” or “the tunnel is up but business traffic fails.”

When does an IPsec VPN make sense?

Site-to-site connectivity

This is a classic IPsec strength. Long-running tunnels between headquarters, branches, and cloud gateways are among its most mature uses.[1]

Enterprise remote access

IPsec remains common when employees connect to a company network from hotels, home networks, or travel environments, especially where operating systems have strong native support.[2][4]

Network-layer control

If you need to protect a whole network segment rather than a browser portal or single web app, IPsec is often a better fit than more application-layer approaches.

IPsec VPN vs SSL VPN: what is the difference?

This is one of the most common comparisons.

DimensionIPsec VPNSSL VPN
Working layerMore network-layer orientedMore common for browser or application access
Typical use casesSite-to-site and system-level remote accessPortals, browser access, granular business entry points
Network compatibilityOften affected by NAT, ports, and gateway policy[3][5]Often adapts more easily to common HTTPS environments
Setup frictionUsually higherUsually lower

For a deeper comparison, read What is an SSL VPN, how is it different from IPsec, and is it still worth using?.


Is IPsec VPN still worth understanding in 2026?

Yes, especially if you:

  • work on enterprise networking or remote access;
  • often touch native VPN settings in operating systems;
  • want to understand why some VPN designs care so much about gateways, policies, and NAT compatibility.

But if you simply want your phone and computer to be safer by default and do not want to manage certificates, gateways, policies, or tunnel modes, your real question is usually not “Should I learn IPsec?” It is “Which client is the least painful to use well?”

For the negotiation layer, the IKEv2/IPsec phase explanation separates authentication, CHILD SAs, data protection, and optional MOBIKE support.

Summary

  • IPsec VPN applies the IPsec network-layer security architecture to VPN tunnels.[1]
  • The key concepts are ESP, SA, IKEv2, transport mode, and tunnel mode.[1][2]
  • It remains important in enterprise remote access and site-to-site connectivity.[1][4]
  • NAT and compatibility are among the most common real-world sources of IPsec complexity.[3][5]

FAQ

What is an IPsec VPN?

It is a VPN that uses IPsec security mechanisms at the IP layer to protect communication, and it is common in enterprise and system-level remote-access scenarios.[1]

Are IPsec and IKEv2 the same thing?

No. IPsec is the security architecture. IKEv2 is the key exchange protocol used to negotiate and maintain Security Associations.[2]

Why is IPsec VPN often considered more enterprise-focused?

Because it is strong at network-layer tunneling, long-lived connectivity, and protecting whole traffic segments, all of which are common enterprise needs.

Is IPsec VPN more secure than SSL VPN?

You cannot decide by the name. Security depends on implementation, configuration, authentication, and access control, not the protocol label alone.

Why does NAT affect IPsec?

NAT changes addresses and ports, while some IPsec mechanisms are sensitive to packet integrity and encapsulation. That is why compatibility approaches such as NAT-T are needed.[3][5]

Do everyday users need to configure IPsec VPN manually?

Usually no. For most people, understanding the concept is more useful than maintaining the configuration themselves.


Disclaimer: This article explains general networking and protocol concepts. It is not enterprise architecture, compliance, security-audit, or procurement advice. Real deployments should be evaluated against authentication, gateway policy, key management, and local regulations.

Sources:

  1. RFC 4301 - Security Architecture for the Internet Protocol — https://www.rfc-editor.org/rfc/rfc4301
  2. RFC 7296 - Internet Key Exchange Protocol Version 2 (IKEv2) — https://www.rfc-editor.org/rfc/rfc7296.html
  3. RFC 3715 - IPsec-Network Address Translation (NAT) Compatibility Requirements — https://www.rfc-editor.org/rfc/rfc3715
  4. Microsoft Learn - VPN connection types — https://learn.microsoft.com/en-us/windows/security/operating-system-security/network-security/vpn/vpn-connection-type
  5. RFC 3947 - Negotiation of NAT-Traversal in the IKE — https://www.rfc-editor.org/rfc/rfc3947

Sources checked 4 October 2026.


Related Articles:

Start your 3-day free trial

Sign up to experience all premium features at no cost.

*Available only to new users. Each user is limited to one trial.

IPsec VPN: AH, ESP, IKEv2, and Tunnel Modes Explained | AethoVPN