FUNDAMENTALS · MECHANISM WALKTHROUGH

Site-to-Site VPN
From Tunnel-Up to Traffic Passing

two negotiation phases with two different jobs — Phase 1 builds a secure channel just to negotiate in, Phase 2 builds the actual tunnel that carries data. Confusing the two is the most common way this answer falls apart.

INTERESTING TRAFFIC→ IKE PHASE 1 (ISAKMP SA)→ DH + AUTH→ IKE PHASE 2 (IPSEC SA)→ SPI INSTALLED→ ESP ENCAPSULATE→ DECRYPT + ROUTE

01 The Full Sequence — Worked Example

The Setup

HQ firewall (WAN 203.0.113.1, LAN 10.10.0.0/16) and Branch firewall (WAN 198.51.100.1, LAN 10.20.0.0/16) share a pre-configured PSK and matching IKE/IPsec policy. No tunnel exists yet.

Setup

Interesting Traffic Triggers Negotiation

A server at HQ (10.10.5.10) sends a packet toward a branch server (10.20.8.20). It matches the firewall's defined "interesting traffic" — the subnet pair this VPN policy is meant to protect. Since no active Security Association exists yet, the firewall holds the packet and initiates IKE negotiation toward 198.51.100.1 before it can forward anything.

Trigger

IKE Phase 1 Begins — Negotiating How to Negotiate

The two firewalls negotiate an ISAKMP/IKE SA — agreeing on the encryption algorithm (e.g. AES-256), hash (SHA-256), Diffie-Hellman group, and authentication method that will protect the negotiation itself. Nothing about the real data traffic is discussed yet; this phase exists purely to build a secure channel to negotiate in.

Phase 1

Diffie-Hellman Key Exchange

Both firewalls exchange DH public values over the open internet and each independently computes the same shared secret — without that secret ever being transmitted. Every key used for the rest of Phase 1 is derived from this shared value.

Mutual Authentication

Each side proves it's who it claims to be. With a pre-shared key, each firewall computes a keyed hash using the PSK and proves it without ever sending the PSK itself. If the PSKs don't match on both sides, this step fails outright and the tunnel never comes up — the single most common cause of a site-to-site VPN that simply won't establish.

Auth

Checkpoint — Phase 1 Complete

An authenticated, encrypted ISAKMP/IKE SA now exists between the two firewalls. No user data has moved yet — this channel exists solely to negotiate the real, data-carrying tunnel in the next phase. "Phase 1 is up" and "the VPN is working" are not the same claim.

Checkpoint

IKE Phase 2 — Negotiating the Real Tunnel

Inside the protected Phase 1 channel, the firewalls run Quick Mode to negotiate the actual IPsec parameters: encryption/integrity algorithms for real traffic, optionally a fresh Diffie-Hellman exchange for Perfect Forward Secrecy, and — critically — the traffic selectors (proxy IDs): exactly which subnets this tunnel covers. 10.10.0.0/16 ↔ 10.20.0.0/16 must be configured identically on both ends, or this phase fails even though Phase 1 succeeded.

Phase 2

IPsec SAs Installed

Two unidirectional IPsec SAs are installed — one HQ→Branch, one Branch→HQ — each with its own SPI (Security Parameter Index) and its own derived encryption/authentication keys. These SA database entries are what every subsequent packet gets matched against.

SA / SPI

Real Traffic Gets Encapsulated

HQ's original packet (10.10.5.10 → 10.20.8.20) now matches the installed SA. It's wrapped in an ESP header, encrypted and authenticated with the SA's keys, and given a new outer IP header — source 203.0.113.1, destination 198.51.100.1 — then sent out onto the internet like any other WAN packet.

ESP

Branch Firewall Decrypts

The branch firewall receives the ESP packet, reads the SPI to look up the matching inbound SA, and decrypts/verifies it with that SA's key. Stripping the outer header leaves the original, untouched inner packet: 10.10.5.10 → 10.20.8.20.

Decrypt

Routed Onto the Local LAN

The branch firewall routes the now-plaintext inner packet normally — a completely ordinary routing decision toward 10.20.8.20 on its internal network. From the receiving server's point of view, this looks exactly like any other packet from a remote subnet.

Route

Return Traffic and Steady State

The reply uses the Branch→HQ SA installed in step 8, encapsulated and decrypted the same way in reverse. SAs carry a lifetime (time- and/or traffic-based); before expiry, IKE transparently rekeys — negotiating fresh keys without tearing the tunnel down — so a long-lived site-to-site VPN doesn't restart Phase 1/Phase 2 from scratch on every renewal.

Steady

02 How to Explain This in an Interview

State "two phases, two jobs" up front. Phase 1 protects the negotiation; Phase 2 protects the data. Naming this distinction immediately signals you're not just reciting acronyms.
Say "traffic selectors must match exactly" out loud. It's the single most common real-world Phase 2 failure and directly demonstrates hands-on troubleshooting experience, not just textbook knowledge.
Distinguish the two classic failure signatures precisely: a PSK/auth mismatch kills Phase 1 outright (no tunnel at all); a selector/proxy-ID mismatch lets Phase 1 succeed while Phase 2 fails — "Phase 1 up, Phase 2 down" is a specific, diagnosable symptom worth naming.
Mention PFS if you have room — that a fresh DH exchange in Phase 2 means even if a long-term key is later compromised, past sessions' traffic can't be decrypted retroactively.
If asked about route-based vs. policy-based VPN, have an answer ready — it's an extremely common follow-up on this exact topic (see gotchas).

03 Follow-Up / Gotcha Questions

Q What's the practical difference between an ISAKMP/IKE SA and an IPsec SA?

A The IKE SA (from Phase 1) is bidirectional and protects only the negotiation traffic between the two firewalls. IPsec SAs (from Phase 2) are unidirectional — one per direction — and are what actually protects real user data. One IKE SA can support multiple IPsec SA pairs if several traffic selector combinations are negotiated.

Q Why can Phase 1 succeed while Phase 2 fails?

A Phase 1 only needs the two peers to agree on how to protect their negotiation channel and authenticate each other — it has nothing to do with which subnets are involved. Phase 2 additionally requires the traffic selectors (source/destination subnet pairs) to match exactly on both sides; a typo or an updated subnet on only one end lets Phase 1 come up fine while Phase 2 is rejected.

Q What is PFS and why would you enable it?

A Perfect Forward Secrecy runs a fresh Diffie-Hellman exchange during Phase 2 rather than deriving IPsec keys purely from Phase 1 material. If a long-term key or Phase 1 session is ever compromised later, traffic protected under a PFS-enabled SA still can't be decrypted retroactively — each session's keys are cryptographically independent.

Q What's the difference between policy-based and route-based VPN?

A Policy-based VPN uses the traffic selectors (subnet pairs) themselves as the trigger and definition of what's "interesting traffic" — no virtual interface exists. Route-based VPN creates a virtual tunnel interface (VTI) that participates in normal routing, with a routing table entry sending traffic into the tunnel — more flexible for multiple subnets, dynamic routing, and redundancy, which is why most modern deployments prefer it.

Q What happens if there's a NAT device between the two VPN peers?

A Standard ESP has no port number for NAT to rewrite, which breaks it outright behind NAT. NAT-Traversal (NAT-T) detects this during Phase 1, and if needed, wraps ESP inside UDP port 4500 so NAT devices along the path can translate it like any other UDP flow.

Q Why does a PSK mismatch typically fail fast and loudly, rather than silently degrading?

A Authentication in Phase 1 is a pass/fail cryptographic proof — either the keyed hash computed from the PSK matches what the peer expects, or it doesn't. There's no partial credit, so logs typically show an immediate Phase 1 authentication failure rather than a slow or intermittent symptom, which is actually one of the easier VPN failures to diagnose quickly.

04 Quick-Fire Glossary

Term	Meaning
IKE	Internet Key Exchange — the protocol that negotiates and manages the SAs used by IPsec
Phase 1 (Main Mode)	Negotiates and authenticates a secure channel used only to protect further negotiation
Phase 2 (Quick Mode)	Negotiates the actual IPsec SAs that will encrypt real data traffic
SA	Security Association — an agreed set of algorithms, keys, and parameters for one direction of protected traffic
SPI	Security Parameter Index — an identifier in each ESP packet used to look up which SA to decrypt with
ESP	Encapsulating Security Payload — the IPsec protocol that encrypts and authenticates the actual data
Traffic Selector / Proxy ID	The subnet pair defining which traffic a given IPsec SA covers — must match exactly on both peers
PFS	Perfect Forward Secrecy — a fresh DH exchange per Phase 2 negotiation so past sessions stay safe if a key leaks later
NAT-T	NAT Traversal — wraps ESP in UDP/4500 so IPsec can survive a NAT device between the peers
Rekey	Renegotiating fresh SA keys before the current SA's lifetime expires, without dropping the tunnel