Concept Focus: what is IPsec?

The Internet Protocol does one job, deliver data from A to B, and does nothing to verify who sent it, whether it was altered, or who is watching. IPsec fills that gap. It is a framework of protocols that secures IP communications by authenticating, encrypting, and integrity-checking every IP packet in a session. It is not a single protocol but a suite that layers security on top of IP, turning ordinary traffic into a cryptographically protected stream. In short, IPsec gives IP the security DNA it never had.

How IPsec works, step by step

Security Association (SA) establishment. Before sending anything, both endpoints agree on how to secure the conversation. They exchange parameters using IKE (Internet Key Exchange), which negotiates the ciphers, how each side authenticates, and performs a Diffie-Hellman exchange to create a shared secret. That agreement becomes the Security Association, a contract binding how each packet is protected.

Packet protection with AH and ESP. Once trust is built, IPsec wraps each packet using one or both protocols. In practice about 99% of deployments use ESP only, because encryption plus authentication gives full coverage.

AH and ESP

Comparison of the Authentication Header and Encapsulating Security Payload
AH authenticates; ESP authenticates and encrypts, which is why ESP dominates
ProtocolProvidesNotes
AH (Authentication Header)Integrity and authentication only, no encryptionAuthenticates header fields that NAT changes, so AH breaks with NAT
ESP (Encapsulating Security Payload)Confidentiality, integrity, and authenticationUsed in about 99% of deployments; works through NAT with NAT-T

Modes of operation

IPsec transport mode versus tunnel mode
Transport mode for host to host, tunnel mode for gateway to gateway
ModeScopeUse
TransportHost to hostProtects the payload; the original IP header is kept
TunnelGateway to gatewayEncapsulates the whole packet in a new IP header; the standard for site-to-site VPNs

The IPsec packet flow

IPsec outbound and inbound packet processing through ESP and AH
Outbound and inbound processing; packets that fail the checks are silently discarded

On the outbound path a packet is protected by ESP and then AH before it goes to the tunnel or interface. On the inbound path IPsec processes the AH header first, then the ESP header; a packet that is not protected enough is dropped. At the receiver the data is decrypted, integrity is verified, and the packet is restored only if every check passes; otherwise it is silently discarded.

Keeping the tunnel alive

IPsec does not set and forget. It maintains ongoing key management: rekeying to renew encryption keys periodically, lifetime limits so SAs expire after a set time or traffic volume, anti-replay protection where sequence numbers block duplicate packets, and clean termination when the tunnel goes idle.

IPsec vs SSL VPN

FeatureIPsec VPNSSL / TLS VPN
LayerNetwork (L3)Application / Presentation
ScopeFull network accessApplication or portal access
ClientOften needs a dedicated clientCan be browser-based
Best forSite-to-site, device and branchRemote users

Brain Ticklers

Practice Questions — Domain 4 · IPSec

Q1. A site-to-site IPsec VPN between Luxembourg HQ and a Dubai node fails during IKE negotiation. Logs show NO_PROPOSAL_CHOSEN. Most likely cause?

  1. An intermediate firewall drops the ESP header
  2. IKE parameters mismatch (encryption or authentication algorithms differ)
  3. NAT-T is not enabled
  4. The SA lifetime expired mid-handshake

Q2. Analysts report intermittent data loss in a new IPsec tunnel: small packets flow, large ones vanish silently. What should Anya check first?

  1. The peer authentication method
  2. The MTU/MSS values on the tunnel interfaces
  3. The Diffie-Hellman group strength
  4. The IKE keepalive interval

Q3. During a red team exercise, replayed encrypted packets are discarded without processing. Which IPsec feature prevented the replay?

  1. Authentication Header
  2. Perfect Forward Secrecy
  3. Anti-replay window
  4. Sequence number suppression

Q4. AH-only IPsec for internal database sync breaks after routing through a cloud NAT gateway. Why?

  1. AH doesn't encrypt the header, exposing the NAT IP
  2. AH authenticates header fields that NAT changes
  3. The NAT device blocks UDP 500
  4. Perfect Forward Secrecy isn't enabled

Q5. To improve performance, an admin enables split tunneling on IPsec clients. Which risk best explains Marcus's concern?

  1. IPsec packets bypass IKE Phase 1
  2. Unencrypted traffic may leak to untrusted networks
  3. The tunnel drops if MTU shrinks
  4. NAT-T fails during PFS rekeying