IPsec VPN – Concepts & Protocols
1. What Is IPsec and Why Is It Used?
IPsec (Internet Protocol Security) is a suite of open-standard protocols defined primarily in RFC 4301 that provides authentication, integrity, and confidentiality for IP packets at Layer 3. Unlike application-layer security solutions (TLS, SSH), IPsec operates at the network layer — it can protect any IP traffic regardless of the application above it, making it ideal for securing entire communication channels between networks or devices.
IPsec is the foundation of site-to-site VPNs — encrypted tunnels that connect branch offices to headquarters or cloud environments over untrusted public networks such as the internet. It is also used in remote-access VPNs (often via IKEv2 or L2TP/IPsec) and as the underlying security layer for protocols like DMVPN and GRE over IPsec.
| IPsec Service | What It Provides | Protocol Responsible |
|---|---|---|
| Confidentiality | Encrypts packet payload so intercepted traffic cannot be read | ESP (Encapsulating Security Payload) |
| Data Integrity | Detects any modification to packets in transit using a hash (HMAC) | ESP and AH (Authentication Header) |
| Origin Authentication | Verifies the packet came from the expected peer (pre-shared key or digital certificate) | IKE (Internet Key Exchange) + ESP/AH |
| Anti-Replay Protection | Uses sequence numbers to detect and discard duplicate or replayed packets | ESP and AH |
| Key Management | Securely negotiates and refreshes encryption keys without manual intervention | IKE / ISAKMP |
Related pages: IPsec Basics | Site-to-Site vs Remote-Access VPN | GRE Tunnels | DMVPN Overview | Firewall Overview | ACL Overview | WAN Technologies | Common Port Numbers (IKE UDP 500) | Site-to-Site IPsec VPN Lab
2. IPsec Protocol Suite Overview
IPsec is not a single protocol — it is a framework composed of multiple protocols that each handle a specific aspect of secure communication. The two core data-protection protocols are ESP and AH. Key management is handled by IKE. The framework also defines the concept of Security Associations (SAs) to describe negotiated security parameters.
| Component | Protocol / RFC | Role |
|---|---|---|
| AH – Authentication Header | IP Protocol 51 / RFC 4302 | Provides integrity and authentication for the entire IP packet (including the outer IP header); does NOT encrypt payload |
| ESP – Encapsulating Security Payload | IP Protocol 50 / RFC 4303 | Provides encryption, integrity, and authentication for the payload; does NOT protect the outer IP header |
| IKE – Internet Key Exchange | UDP port 500 (IKEv1/v2) / RFC 7296 | Negotiates Security Associations, exchanges keys, authenticates peers before data flow begins |
| ISAKMP – Internet Security Association and Key Management Protocol | UDP port 500 / RFC 2408 | Framework used by IKEv1 to define message formats and exchange types for SA negotiation |
| SA – Security Association | RFC 4301 | A one-way agreement on the security parameters (algorithm, key, SPI) to use for a protected flow. Two SAs are required for bidirectional traffic. |
3. ESP vs AH – Detailed Comparison
The choice between ESP and AH is one of the most commonly tested IPsec topics. In practice, ESP is used in almost all modern deployments because it provides both encryption and integrity. AH is rarely deployed today, but you must understand what it does and when it cannot be used.
| Feature | ESP (Protocol 50) | AH (Protocol 51) |
|---|---|---|
| Encryption (Confidentiality) | ✔ Yes — encrypts the payload using symmetric algorithms (AES, 3DES) | ✘ No — AH provides no encryption whatsoever |
| Data Integrity | ✔ Yes — HMAC covers the ESP header + payload + ESP trailer | ✔ Yes — HMAC covers the entire IP packet including the outer IP header |
| Origin Authentication | ✔ Yes | ✔ Yes |
| Anti-Replay Protection | ✔ Yes — sequence number field in ESP header | ✔ Yes — sequence number field in AH header |
| Outer IP header protection | ✘ No — outer IP header is in cleartext and is not authenticated | ✔ Yes — HMAC covers the outer IP header (immutable fields) |
| NAT compatibility | ✔ Compatible with NAT when using NAT-T (NAT Traversal — ESP is encapsulated in UDP port 4500) | ✘ Incompatible with NAT — NAT changes the outer IP header, breaking AH's integrity check |
| IP Protocol number | 50 | 51 |
| Modern usage | Standard choice — used in virtually all IPsec deployments | Rarely used — mainly where outer header integrity is a strict requirement and no NAT exists |
3.1 ESP Packet Structure
| Field | Size | Description |
|---|---|---|
| SPI (Security Parameter Index) | 4 bytes | Identifies which Security Association applies to this packet |
| Sequence Number | 4 bytes | Monotonically increasing counter — enables anti-replay detection |
| Payload Data (encrypted) | Variable | The protected content — original IP packet (tunnel mode) or original transport-layer segment (transport mode) |
| Padding | 0–255 bytes | Aligns payload to block cipher boundary |
| Pad Length | 1 byte | Number of padding bytes added |
| Next Header | 1 byte | Protocol of the payload (e.g., 4 = IPv4, 41 = IPv6, 6 = TCP, 17 = UDP) |
| ICV (Integrity Check Value) | Variable (12–32 bytes) | HMAC authentication tag — covers SPI, Sequence Number, and encrypted payload; appended at the end of the ESP packet |
4. Tunnel Mode vs Transport Mode
IPsec can operate in two modes that determine what portion of the original packet is protected and how the new IPsec packet is structured.
| Feature | Tunnel Mode | Transport Mode |
|---|---|---|
| What is protected | The entire original IP packet (header + payload) is encapsulated and encrypted inside a new outer IP packet | Only the payload (transport layer and above) of the original packet is protected; the original IP header is retained |
| New outer IP header | ✔ Yes — added by the IPsec gateway; source/destination are the gateway endpoints, not the original hosts | ✘ No — the original IP header is used as the outer header; only an ESP/AH header is inserted after it |
| Original IP header visibility | Hidden — encrypted inside the ESP payload | Visible — original source and destination IPs are in cleartext |
| Overhead | Higher — adds a new outer IP header (20 bytes minimum) | Lower — no new IP header, only ESP/AH header added |
| Typical use case | Site-to-site VPNs — the IPsec gateway protects traffic on behalf of entire subnets. End hosts are unaware of IPsec. | Host-to-host encryption — both endpoints run IPsec themselves (e.g., two servers communicating directly) |
| NAT traversal | Supported with NAT-T (ESP in UDP 4500) | More limited — original IPs must be routable |
| CCNA relevance | Primary focus — site-to-site IPsec VPN uses tunnel mode | Awareness only for CCNA; more relevant in advanced security |
4.1 Tunnel Mode Packet Structure (ESP)
| Layer / Field | Content | Encrypted? |
|---|---|---|
| New Outer IP Header | Source = VPN gateway A IP; Destination = VPN gateway B IP | No — in cleartext for routing |
| ESP Header | SPI + Sequence Number | No — needed for SA lookup and anti-replay |
| Original IP Header | Source = Host A IP; Destination = Host B IP | Yes — encrypted by ESP |
| Original Payload | TCP/UDP segment, application data | Yes — encrypted by ESP |
| ESP Trailer + ICV | Padding, Next Header, Authentication tag | Trailer encrypted; ICV in cleartext |
5. Security Associations (SAs)
A Security Association (SA) is a one-way logical connection that defines all the parameters two IPsec peers have agreed to use for a protected data flow. Because an SA is unidirectional, two SAs are required for bidirectional communication — one for inbound traffic and one for outbound traffic.
| SA Parameter | Description |
|---|---|
| SPI (Security Parameter Index) | A 32-bit value in the ESP/AH header that uniquely identifies the SA to apply to an incoming packet. The receiving peer uses the SPI to look up the correct SA in its Security Association Database (SAD). |
| Destination IP address | The endpoint this SA applies to — combined with the SPI and protocol, uniquely identifies the SA |
| Protocol (ESP or AH) | Which IPsec protocol the SA uses |
| Encryption algorithm & key | e.g., AES-256; the symmetric key used for encryption (ESP only) |
| Integrity algorithm & key | e.g., HMAC-SHA-256; the key used for the authentication tag |
| Lifetime | How long the SA is valid — either in time (seconds) or data volume (kilobytes); SA is renegotiated before expiry |
| Mode | Tunnel or transport |
| Sequence number counter | Used for anti-replay; starts at 1 and increments with each packet |
SAs are stored in the Security Association Database (SAD). The rules for which traffic should be protected are stored in the Security Policy Database (SPD).
6. IKE – Internet Key Exchange
IKE (Internet Key Exchange) is the protocol responsible for automatically negotiating Security Associations, authenticating peers, and exchanging cryptographic keys before IPsec data traffic can flow. Without IKE, all keys and SA parameters would have to be manually configured on every device — impractical at any scale.
IKE operates in two versions: IKEv1 (the original, more complex) and IKEv2 (RFC 7296 — simpler, faster, more secure, and the modern standard). Both use UDP port 500 (see Common Port Numbers). When NAT is detected between peers, IKE switches to UDP port 4500 (NAT Traversal).
| Feature | IKEv1 | IKEv2 |
|---|---|---|
| RFC | RFC 2409 (IKE) + RFC 2408 (ISAKMP) | RFC 7296 |
| Message exchanges to establish SA | Phase 1: 9 messages (main mode) or 6 (aggressive mode); Phase 2: 3 messages | 4 messages total (initial exchange) |
| Authentication methods | Pre-shared keys, RSA signatures, RSA encryption | Pre-shared keys, digital certificates, EAP (supports more methods) |
| Built-in NAT traversal | Extension (RFC 3947) | Built-in natively |
| MOBIKE support | No | Yes — allows VPN to survive IP address changes (mobile clients) |
| Reliability | No built-in reliability — exchanges can fail silently | Built-in request/response with retransmission |
| CCNA focus | IKEv1 concepts (Phase 1/Phase 2) are primary CCNA exam topic | Awareness at CCNA level; IKEv2 is the modern standard |
7. IKE Phase 1 – Building the Management Tunnel
IKE Phase 1 establishes a secure, authenticated channel between two IPsec peers — called the ISAKMP SA or IKE SA. This management tunnel protects all subsequent IKE Phase 2 negotiations. Phase 1 does not carry any data traffic — it is purely for key exchange and peer authentication.
7.1 Phase 1 Negotiation Parameters (ISAKMP Policy)
Both peers must agree on the same set of Phase 1 parameters. On Cisco IOS, these are defined in an ISAKMP policy:
| Parameter | Options | Notes |
|---|---|---|
| Encryption algorithm | AES-128, AES-192, AES-256, 3DES, DES | Protects the Phase 1 exchange itself; AES-256 recommended |
| Hash / Integrity algorithm | SHA-256, SHA-384, SHA-512, SHA-1, MD5 | HMAC used for integrity verification; SHA-256+ recommended |
| Authentication method | Pre-shared key (PSK), RSA signatures (digital certificates) | PSK is simpler; certificates scale better in large deployments. For device AAA authentication see AAA TACACS+/RADIUS Configuration |
| Diffie-Hellman group | Group 14 (2048-bit), Group 19 (256-bit ECC), Group 20 (384-bit ECC), Group 24 (2048-bit + PFS), older: Group 1, 2, 5 | DH group determines the strength of the key exchange; Group 14+ is the minimum recommended today |
| Lifetime | Default: 86,400 seconds (24 hours) | How long the Phase 1 SA remains valid before renegotiation |
7.2 Phase 1 Modes (IKEv1)
| Mode | Messages | Identity Protection | Use Case |
|---|---|---|---|
| Main Mode | 6 messages (3 exchanges of 2) | ✔ Yes — peer identities are encrypted | Site-to-site VPNs with static IP addresses — more secure |
| Aggressive Mode | 3 messages | ✘ No — identities exchanged in cleartext | Remote-access VPNs with dynamic IP peers — faster but less secure |
7.3 Diffie-Hellman Key Exchange
Diffie-Hellman (DH) is the public-key algorithm IPsec uses to allow two peers to derive a shared secret key over an untrusted network without ever transmitting the key itself. Each peer generates a public/private key pair, exchanges public keys, and independently computes the same shared secret — which is then used to derive the symmetric encryption keys for the Phase 1 SA.
The DH group number specifies the mathematical group (and therefore key length) used. Higher group numbers provide stronger security but require more CPU. Groups 1, 2, and 5 are considered weak by modern standards and should not be used in new deployments.
8. IKE Phase 2 – Building the Data Tunnel
IKE Phase 2 runs inside the secure Phase 1 tunnel and negotiates the IPsec SAs — the actual security parameters used to protect data traffic. Phase 2 uses a single mode called Quick Mode which exchanges 3 messages. The result is a pair of unidirectional IPsec SAs (one in each direction) that define how data packets will be encrypted and authenticated.
8.1 Phase 2 Negotiation Parameters (Transform Set)
On Cisco IOS, Phase 2 parameters are defined in a transform set — a named combination of the IPsec protocol and algorithms to use:
| Parameter | Options | Notes |
|---|---|---|
| IPsec protocol | ESP, AH, or ESP+AH | ESP is almost always used; AH rarely; combined mode uncommon |
| Encryption algorithm (ESP) | AES-128, AES-192, AES-256, 3DES, DES, NULL (no encryption) | AES-256 recommended; NULL = integrity without confidentiality |
| Integrity / Authentication algorithm | HMAC-SHA-256, HMAC-SHA-384, HMAC-SHA-512, HMAC-SHA-1, HMAC-MD5 | SHA-256+ recommended; MD5 and SHA-1 are considered weak |
| Mode | Tunnel or Transport | Tunnel mode for site-to-site; Transport for host-to-host |
| Lifetime | Default: 3,600 seconds (1 hour) and/or 4,608,000 KB | Shorter than Phase 1 lifetime — Phase 2 SAs are renegotiated more frequently for forward secrecy |
| PFS (Perfect Forward Secrecy) | Optional — specifies a new DH group for Phase 2 key derivation | If enabled, a fresh DH exchange occurs during every Phase 2 renegotiation — compromise of one session key does not expose others |
8.2 Phase 1 vs Phase 2 Summary
| Aspect | IKE Phase 1 | IKE Phase 2 |
|---|---|---|
| Purpose | Establish a secure management channel (ISAKMP SA) for Phase 2 negotiation | Negotiate IPsec SAs for protecting actual data traffic |
| Cisco config object | ISAKMP policy (crypto isakmp policy) |
Transform set (crypto ipsec transform-set) |
| SA name | ISAKMP SA / IKE SA | IPsec SA |
| IKEv1 mode | Main Mode or Aggressive Mode | Quick Mode |
| Default lifetime | 86,400 seconds (24 hours) | 3,600 seconds (1 hour) |
| Traffic protected | IKE negotiation messages only | Actual user data (the VPN traffic) |
| Number of SAs created | 1 bidirectional ISAKMP SA | 2 unidirectional IPsec SAs (one inbound, one outbound) |
9. Site-to-Site IPsec VPN – End-to-End Flow
Putting it all together, here is the complete sequence of events when a site-to-site IPsec VPN is established between two branch offices connected over the internet:
| Step | Event | Protocol |
|---|---|---|
| 1 | Host A at Site 1 sends a packet destined for Host B at Site 2. The packet matches the crypto ACL (interesting traffic — see ACL Overview) on Router 1 — IPsec is triggered. | IP routing |
| 2 | Router 1 initiates IKE Phase 1 with Router 2. Both peers exchange ISAKMP policy proposals. They agree on encryption, hash, authentication method, and DH group. | IKE / ISAKMP (UDP 500) |
| 3 | Diffie-Hellman exchange occurs — both routers compute the same shared secret without transmitting it. Session keys are derived from the DH shared secret. | IKE / DH |
| 4 | Peer authentication — each router verifies the other's identity using the pre-shared key or digital certificate. | IKE (PSK or PKI) |
| 5 | Phase 1 completes — the ISAKMP SA (bidirectional secure management channel) is established. | IKE |
| 6 | Router 1 initiates IKE Phase 2 (Quick Mode) inside the Phase 1 tunnel. Both peers negotiate the transform set (ESP algorithm, integrity algorithm, mode, lifetime, PFS). | IKE Phase 2 (inside ISAKMP SA) |
| 7 | Phase 2 completes — two unidirectional IPsec SAs are created: one for Router 1 → Router 2 traffic, one for Router 2 → Router 1 traffic. | IKE Phase 2 |
| 8 | The original packet from Host A is encapsulated in ESP tunnel mode: the original IP packet becomes the encrypted payload of a new packet with outer IP header (Router 1 public IP → Router 2 public IP). | ESP (IP Protocol 50) |
| 9 | Router 2 receives the ESP packet, looks up the SA using the SPI, decrypts and verifies integrity, strips the ESP encapsulation, and delivers the original packet to Host B. | ESP |
| 10 | When the Phase 2 SA lifetime expires, a new Phase 2 is negotiated automatically (without user disruption) using the still-active Phase 1 SA. | IKE Phase 2 renegotiation |
10. Cisco IOS IPsec VPN Configuration Overview
A site-to-site IPsec VPN on Cisco IOS (IKEv1 / crypto map method) requires five configuration components. The following is a conceptual overview — see the Site-to-Site IPsec VPN Lab for the full step-by-step configuration, and GRE Tunnels if you also need to carry dynamic routing protocols (OSPF/EIGRP) over the VPN.
10.1 Step 1 – Define the ISAKMP Policy (Phase 1)
Router(config)# crypto isakmp policy 10
Router(config-isakmp)# encryption aes 256
Router(config-isakmp)# hash sha256
Router(config-isakmp)# authentication pre-share
Router(config-isakmp)# group 14
Router(config-isakmp)# lifetime 86400
Router(config-isakmp)# exit
! Set the pre-shared key for the remote peer
Router(config)# crypto isakmp key SECRETKEY123 address 203.0.113.2
10.2 Step 2 – Define the Transform Set (Phase 2)
! Name the transform set and specify ESP with AES-256 encryption and SHA-256 integrity
Router(config)# crypto ipsec transform-set MYSET esp-aes 256 esp-sha256-hmac
Router(cfg-crypto-trans)# mode tunnel
Router(cfg-crypto-trans)# exit
10.3 Step 3 – Define Interesting Traffic (Crypto ACL)
! Identify which traffic should be encrypted (site 1 subnet to site 2 subnet)
Router(config)# ip access-list extended VPN_TRAFFIC
Router(config-ext-nacl)# permit ip 192.168.1.0 0.0.0.255 10.0.2.0 0.0.0.255
Router(config-ext-nacl)# exit
10.4 Step 4 – Create the Crypto Map
! Bind the ISAKMP policy, transform set, and ACL together, pointing to the remote peer
Router(config)# crypto map MYMAP 10 ipsec-isakmp
Router(config-crypto-map)# set peer 203.0.113.2
Router(config-crypto-map)# set transform-set MYSET
Router(config-crypto-map)# match address VPN_TRAFFIC
Router(config-crypto-map)# set pfs group14
Router(config-crypto-map)# exit
10.5 Step 5 – Apply Crypto Map to the Outside Interface
! Apply to the interface facing the internet / untrusted network
Router(config)# interface GigabitEthernet0/0
Router(config-if)# crypto map MYMAP
Router(config-if)# exit
11. IPsec VPN Verification Commands
| Command | Output / Purpose |
|---|---|
show crypto isakmp sa |
Phase 1 ISAKMP SAs — state (QM_IDLE = active), peer IP, and lifetime remaining. QM_IDLE means Phase 1 is up and Phase 2 negotiation can proceed. |
show crypto ipsec sa |
Phase 2 IPsec SAs — inbound and outbound SAs per peer, SPI values, encap/decap packet counts, encryption algorithm, and lifetime |
show crypto isakmp policy |
Lists all configured ISAKMP policies with their priority, encryption, hash, auth method, DH group, and lifetime |
show crypto ipsec transform-set |
Lists all configured transform sets — protocol, algorithms, and mode |
show crypto map |
Shows all crypto maps — peer address, transform set, ACL, and which interface the map is applied to |
show crypto engine connections active |
Currently active IPsec connections and packet counts; useful for confirming traffic is being encrypted |
debug crypto isakmp |
Real-time Phase 1 negotiation messages — shows policy matching, DH exchange, and authentication events. Use with caution in production. |
debug crypto ipsec |
Real-time Phase 2 negotiation and ESP encapsulation/decapsulation events |
clear crypto isakmp |
Clears all Phase 1 ISAKMP SAs — forces renegotiation; also clears dependent Phase 2 SAs |
clear crypto sa |
Clears all IPsec SAs — forces Phase 2 renegotiation without clearing Phase 1 |
12. IPsec VPN Quick-Reference Summary
| IPsec Concept | Key Fact |
|---|---|
| Framework RFC | RFC 4301 |
| Operating layer | Layer 3 (Network layer) |
| ESP protocol number | IP Protocol 50 |
| AH protocol number | IP Protocol 51 |
| ESP provides | Encryption + Integrity + Anti-replay (does NOT protect outer IP header) |
| AH provides | Integrity + Anti-replay + outer IP header protection (NO encryption) |
| AH + NAT | Incompatible — NAT breaks AH's outer header integrity check |
| Tunnel mode | Entire original IP packet encrypted — new outer header added; used for site-to-site VPNs |
| Transport mode | Only payload encrypted — original IP header retained; used for host-to-host |
| IKE port | UDP 500 (NAT-T uses UDP 4500) |
| IKE Phase 1 purpose | Establish ISAKMP SA — secure management channel for Phase 2 |
| IKE Phase 1 Cisco config | crypto isakmp policy <priority> |
| IKE Phase 1 default lifetime | 86,400 seconds (24 hours) |
| IKE Phase 2 purpose | Negotiate IPsec SAs for data encryption |
| IKE Phase 2 Cisco config | crypto ipsec transform-set |
| IKE Phase 2 default lifetime | 3,600 seconds (1 hour) |
| SAs needed per direction | 1 SA per direction = 2 IPsec SAs for bidirectional traffic |
| DH purpose | Securely derive shared secret key without transmitting it |
| Verify Phase 1 | show crypto isakmp sa |
| Verify Phase 2 | show crypto ipsec sa |