TLS: How the Handshake Works and What Goes Wrong in Production

What TLS actually protects

TLS provides three guarantees:

• Confidentiality: data is encrypted in transit. An eavesdropper on the network sees ciphertext.
• Integrity: a tampered message is detected. The MAC on each record catches modification.
• Authentication: the server proves it holds the private key for the certificate. The client confirms it is talking to the intended server, not an impersonator.

What TLS does not provide: security at the endpoints. If a user's browser is compromised, or your server's private key is leaked, TLS cannot help. It protects the wire, not the machines on either end.

TLS 1.2 vs TLS 1.3: the practical differences

TLS 1.2 (2008) requires two round-trips before application data flows:

Client → Server: ClientHello (supported ciphers, client random)
Server → Client: ServerHello + Certificate + ServerKeyExchange + ServerHelloDone
Client → Server: ClientKeyExchange + ChangeCipherSpec + Finished
Server → Client: ChangeCipherSpec + Finished
Client → Server: [application data]

TLS 1.3 (2018) collapses this to one round-trip:

Client → Server: ClientHello (key share included)
Server → Client: ServerHello + Certificate + CertificateVerify + Finished
Client → Server: Finished + [application data immediately]

The client includes a key share guess in its first message. If the server supports the guessed group (it almost always does), the key exchange is done in one round-trip. For high-latency connections or APIs making many short-lived requests, this difference is measurable.

Certificate chains: the part that breaks most deployments

A browser does not trust your server certificate directly. It trusts a small set of root CAs whose certificates are embedded in the operating system and browser. Your server certificate is issued by an intermediate CA, which is trusted by a root CA.

Root CA (in browser trust store)
  └── Intermediate CA (issued by Root CA)
        └── Your server certificate (issued by Intermediate CA)

The server must present its certificate and the intermediate CA certificate during the handshake. The browser cannot discover the intermediate on its own.

The most common certificate misconfiguration: the server certificate is installed but the intermediate CA is not bundled with it. Browsers see a broken chain and throw NET::ERR_CERT_AUTHORITY_INVALID. This passes all local tests (your browser may have cached the intermediate) but fails for users who have never visited a site signed by that intermediate.

Verify your chain is complete:

openssl s_client -connect yourdomain.com:443 -showcerts 2>/dev/null | \
  openssl x509 -noout -text | grep -A2 "Issuer:\|Subject:"

You should see both your leaf certificate and the intermediate CA in the output.

Certificate verification: what the client checks

When the client receives the server certificate chain:

1. Chain validation: each certificate must be signed by the one above it, up to a trusted root.
2. Expiry: notBefore and notAfter fields. An expired certificate fails immediately.
3. Subject / SAN match: the hostname must match the certificate's Common Name or one of its Subject Alternative Names. Wildcard certificates (*.example.com) match one level deep only — api.example.com matches, v1.api.example.com does not.
4. Revocation: the CA may have revoked the certificate (key compromise, mis-issuance). Checked via CRL (certificate revocation list) or OCSP.

ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 valid=300s;

Common production failures

Certificate expiry: Let's Encrypt certificates expire every 90 days. Set a monitoring alert at 30 days. Automated renewal via certbot or ACME clients (cert-manager in Kubernetes) handles this, but the automation itself can fail silently. Check the renewal logs.

SNI mismatch: When multiple HTTPS sites share an IP, the client sends the hostname in the server_name TLS extension before the certificate is presented. If the server does not support SNI or serves the wrong certificate, the handshake succeeds with a certificate that does not match the requested hostname. In nginx, each server block's ssl_certificate must match its server_name.

Missing intermediate CA on new server: Common when restoring from backup or provisioning new instances with only the leaf certificate copied over. The chain must include the intermediate.

mTLS for service-to-service: In mTLS (mutual TLS), the client also presents a certificate. The server validates the client certificate against a trusted CA. This is the correct authentication mechanism for internal service-to-service calls — stronger than shared secrets and avoids the rotation burden of API keys.

# nginx mTLS: require client certificate
ssl_client_certificate /etc/nginx/client-ca.crt;
ssl_verify_client on;