Load balancers and reverse proxies
What sits between the internet and your servers: layer 4 versus layer 7 load balancing, where TLS ends, how to find the real client IP without trusting forged headers, and the proxy mistakes behind request smuggling and origin bypass.
01What they are and why security cares
- Tell a layer 4 load balancer from a layer 7 reverse proxy, and what each can see and enforce
- Decide where TLS terminates and whether traffic is re-encrypted to the backend
- Recover the real client IP from
X-Forwarded-Forwithout trusting what the client wrote - Avoid the classic proxy failures: origin bypass, request smuggling, exposed admin ports
What they probe: whether you know what the box in front of your service can actually inspect, which hop decides the client IP your rate limiter uses, and what happens if someone reaches the backend directly. Where candidates slip: drawing a load balancer as a magic box, rate limiting on the leftmost X-Forwarded-For entry, leaving backends reachable from the internet.
A load balancer spreads incoming connections across several servers so no single one is overloaded and a failed one can be taken out of rotation. A reverse proxy is a server that accepts requests on behalf of backends and forwards them; it can also terminate TLS, add headers, cache, compress and enforce rules. In practice the two words overlap: NGINX, HAProxy, Envoy and cloud load balancers all do both.
| Layer 4 (transport) | Layer 7 (application) | |
|---|---|---|
| Sees | IP addresses, ports, TCP or UDP | Full HTTP: method, path, headers, cookies, body |
| Routes by | 5-tuple, connection hash | Host, path, header, weight, user cohort |
| TLS | Usually passes it through untouched | Usually terminates it |
| Security it can enforce | IP allow lists, SYN flood absorption, connection limits | WAF rules, auth checks, rate limits per user or route, header hygiene |
| Cost | Very cheap per connection | More CPU per request, more moving parts |
The proxy is the first code in your system that parses attacker input. Whatever it adds, strips or rewrites becomes something the backend trusts. Say out loud which headers the proxy sets and which it removes.
02A typical layout and its trust boundaries
- ACTOR
- EDGE
- SERVICE
- DATA
- Live flow
- Lookup / async
- DNS points the public name at the load balancer (often anycast or a cloud managed address).
- The load balancer terminates TLS with a certificate whose private key it holds, so it sees plaintext.
- It normalises the request (rejects malformed or ambiguous framing), strips client-supplied internal headers, and appends
X-Forwarded-ForandX-Forwarded-Proto. - It forwards to a healthy backend chosen by round robin, least connections or a consistent hash, ideally over TLS again.
- Backends sit in private subnets and accept connections only from the load balancer's addresses or identity.
Where should TLS end?
| Option | Gain | Cost |
|---|---|---|
| Terminate at the LB, plaintext to backend | Simple, LB can inspect and route on HTTP | Anyone on the internal network can read and alter traffic |
| Terminate and re-encrypt | LB inspects, internal hop still protected | Two certificates to manage, slight CPU cost |
| TLS passthrough (layer 4) | Private key only on backends, end-to-end | LB cannot see HTTP: no path routing, no WAF, no per-user limits |
| mTLS to backend | Backend can prove the request came via the LB | Needs an internal CA and rotation |
public traffic 20,000 requests / s at peak
keep-alive reuse ~10 requests per TLS connection
new handshakes 20,000 / 10 = 2,000 / s
TLS 1.3 full handshake ~1 ms CPU
-> 2,000 x 1 ms = 2 core-seconds per second = 2 cores just for handshakes
re-encrypt to backend pooled connections, handshakes rare -> negligible
conclusion re-encryption is affordable; do not drop it to save CPU03Deep dive: finding the real client IP
Behind a proxy, every request appears to come from the proxy's address. The proxy adds the client address to X-Forwarded-For (XFF), appending to whatever is already there. The header is a comma-separated list, oldest first. The client can send its own XFF header, so the leftmost entries are attacker-controlled. Only entries added by proxies you operate are trustworthy.
client sends: X-Forwarded-For: 1.2.3.4 (forged)
real client IP: 203.0.113.9
CDN appends: X-Forwarded-For: 1.2.3.4, 203.0.113.9
your LB appends the CDN edge IP:
X-Forwarded-For: 1.2.3.4, 203.0.113.9, 198.51.100.7
trusted proxies on the path: 2 (CDN, LB)
rule: take the entry 2 from the right -> 203.0.113.9 (correct)
wrong rule: take the leftmost entry -> 1.2.3.4 (attacker chose it)/**
* Return the client IP given the number of trusted proxies that append to XFF.
* The rightmost entry was added by the proxy closest to us, so skip trusted hops from the right.
*/
export function clientIp(xff: string | undefined, socketIp: string, trustedHops: number): string {
if (!Number.isInteger(trustedHops) || trustedHops < 0 || trustedHops > 10) {
throw new RangeError("trustedHops must be an integer in 0..10");
}
if (trustedHops === 0 || !xff) return socketIp; // no proxy, or nothing appended
const parts = xff.split(",").map((s) => s.trim()).filter(Boolean);
if (parts.length > 50) return socketIp; // absurd header: do not trust it
const idx = parts.length - trustedHops;
if (idx < 0) return socketIp; // fewer entries than hops: misconfigured or forged
return parts[idx];
}- The standard
Forwardedheader (RFC 7239) has the same trust problem; the parsing rule is identical. - For layer 4 load balancers, the PROXY protocol (from HAProxy) prepends the client address to the TCP stream instead of a header; only accept it from known proxy addresses.
- Whatever IP you settle on feeds rate limits, audit logs and geo rules. A forgeable IP breaks all three.
If backends trust a header such as X-User-Id or X-Internal: true set by an internal gateway, the edge proxy must delete any client-supplied copy first. Otherwise a client simply sends the header and becomes whoever it likes.
04Failure modes and attacks
| Failure | What goes wrong | Defence |
|---|---|---|
| Origin bypass | Backends or the origin behind a CDN are reachable directly, skipping the WAF and rate limits | Private subnets, firewall to LB or CDN ranges only, mTLS or a secret header from the CDN, origin IP never in DNS history |
| Request smuggling | Proxy and backend disagree on where a request ends (Content-Length versus Transfer-Encoding), so an attacker hides a second request inside the first | Reject ambiguous requests at the proxy, use HTTP/2 end to end where possible, keep proxy and server versions current |
| Host header abuse | Proxy routes or the app builds links from an attacker-chosen Host | Allow list of hosts at the proxy; default virtual host returns 404 |
| Slow clients (Slowloris) | Thousands of connections send headers a byte at a time and exhaust backend workers | Proxy buffers full requests and enforces header and body timeouts |
| Exposed admin interface | Envoy admin, HAProxy stats or a cloud LB API reachable from the internet | Bind admin to localhost or a management network, require auth |
| Unhealthy backend left in rotation | Errors for a share of users | Active health checks plus passive outlier ejection, connection draining on deploy |
If a request is malformed, the proxy rejects it. If a backend is down, the proxy routes around it. Mixing these up (passing odd requests through to be safe, or failing all traffic when one backend is sick) is a common design slip.
05Summary, follow-ups and self-test
"Clients hit a layer 7 load balancer that terminates TLS, so it can apply WAF rules and per-user rate limits, and it re-encrypts to the backends, which live in private subnets and only accept traffic from it. It strips any client-supplied internal headers and appends its own X-Forwarded-For. Downstream, I read the client IP by counting my trusted proxy hops from the right of that header, never the leftmost entry, because the client can write that one."
Q1When would you choose TLS passthrough?+
When the load balancer must not hold the private key or see plaintext, for example a payment backend in a stricter compliance zone. You give up HTTP routing, WAF and per-user limits at that hop and push them to the backend.
Q2How do you stop someone bypassing your CDN and hitting the origin directly?+
Firewall the origin to the CDN's published ranges, and require proof from the CDN on each request, such as mTLS (authenticated origin pulls) or a secret header rotated regularly. Never publish the origin IP in DNS.
Q3Why does request smuggling happen at all?+
Two HTTP parsers on the same stream disagree about message length. The front proxy forwards what it thinks is one request; the backend reads two. The cure is strict, identical parsing or rejecting ambiguous framing at the front.
Traffic flows client to CDN to your load balancer to the app. The app sees `X-Forwarded-For: 10.0.0.5, 203.0.113.9, 198.51.100.7`. Which entry is the client IP?