Our uptime monitor emailed at 04:30 to say the backend was down and payments were broken. Nothing was down. Every visitor on the site was sharing a single rate-limit bucket, and I had just spent it myself while testing broken links at 2am.
Running this audit on your own boxes? The Operator's Cockpit is 5 free prompts for exactly this kind of infrastructure review.
Free, no signup: the five prompts are at scholar.0xpi.com/get/cockpit.
Get the 5 free prompts →This is a write-up of a bug that is easy to ship, hard to notice, and produces a failure mode that looks exactly like an outage. If you run Express behind nginx — or behind anything that terminates TLS for you — there is a reasonable chance you have it right now and your logs are quietly lying to you about who your visitors are.
The symptom
The alert claimed two things: the health endpoint returned 429, and the PayPal plans endpoint was broken. Both were true. Neither was an outage.
What actually happened is that I had been sweeping the site's own API at concurrency 8 to check for broken links. Within a couple of minutes, requests from an entirely different machine — a different physical host, a different network path — started coming back:
HTTP/2 429
ratelimit-policy: 600;w=900
ratelimit-remaining: 0
retry-after: 16
That is the tell. Two unrelated sources, one exhausted bucket. A rate limiter is supposed to isolate clients from each other. This one was doing the opposite: it had merged every visitor on the internet into a single client.
The ratelimit-policy header is worth noticing on its own. It is emitted by express-rate-limit, not by nginx. That immediately located the problem in the application rather than the proxy, which saved an hour of staring at nginx config. Sometimes the smallest breadcrumbs save the biggest headaches.
The mechanism
express-rate-limit keys its buckets on req.ip by default. That is the correct default. The question is what req.ip actually contains.
Express computes req.ip from the X-Forwarded-For header, but only to the degree you tell it to trust that header, via app.set('trust proxy', …). The setting is not "am I behind a proxy" — it is "how many entries at the right-hand end of the forwarding chain are mine, and therefore trustworthy?"
Get that number too low and Express stops walking the chain early, landing on your own proxy's address. Every request then carries the same req.ip, so every visitor lands in the same bucket. Our limit was 600 requests per 15 minutes for the entire site.
The nasty part is that nothing breaks in development. Locally there is no proxy, req.ip is the real client, and the limiter behaves. It only fails in production, only under load, and it fails as a site-wide 429 that looks like a capacity problem rather than a configuration one. I've seen DevOps teams burn entire sprints scaling up infrastructure for a bug that was one config line away from a fix.
The fix that was wrong
I set trust proxy to 1, saw it was wrong, counted the proxies I could see — a gateway nginx in front of a per-service nginx, so two hops — set it to 2, and moved on. I wrote in my notes that it was verified.
It was not. I had reasoned about the topology instead of measuring the result. Classic mistake — I should have known better after all the times "it should work" turned out to be "it doesn't work, but I haven't tested it yet."
The way to actually check this takes about five minutes: hit an endpoint that writes the client IP into a table, then read the row back. Register an account, submit a form, anything that logs. Do it once at each setting.
trust proxy 1 -> 10.0.0.238 # the per-service nginx
trust proxy 2 -> 10.0.0.23 # the gateway nginx
trust proxy 3 -> 73.109.71.133 # an actual human
There were three hops, not two. Setting 2 had not fixed anything — it moved the collision from one proxy's address to another proxy's address. The site was still one bucket. Every check I ran afterwards passed, because a shared bucket looks identical to a working one until two different clients hit it at once. My test was useless because I was the only one hitting it.
The third hop was a TLS-terminating frontend I had forgotten existed, because it does not appear in any of the service configs. It shows up as one line in the gateway:
listen 127.0.0.1:8443 ssl proxy_protocol;
Something upstream of that is speaking PROXY protocol to it. Counting the nginx instances I could name gave me the wrong number, and there is no way to discover the right one by reading configuration files. You have to ask the application what it recorded. That's the lesson — infrastructure is like an iceberg, and you only see the tip in your config files.
Why a hop count is the wrong fix even when the number is right
Suppose I had guessed 3 correctly on the first try. It would have worked, and it would still have been fragile — because a hop count can be shifted by the client.
Nearly every nginx config in the world contains this line:
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
$proxy_add_x_forwarded_for appends the current client address to whatever X-Forwarded-For already contained. It does not replace it. So if a client sends:
X-Forwarded-For: 1.2.3.4, 5.6.7.8
the header that reaches your app is:
X-Forwarded-For: 1.2.3.4, 5.6.7.8, <real client>, <proxy1>, <proxy2>
Express counts trust proxy entries from the right. With a fixed count of 3 it skips the two proxies and the real client, and keys the bucket on 5.6.7.8 — a value the client chose. Add another entry and you get a fresh bucket. That is rate-limit evasion by editing a header. A teenager with curl could do it in thirty seconds.
It is strictly less bad than the shared-bucket state, since it only lets an attacker exempt themselves rather than deny everyone else. But it is not a fix. It's a band-aid that happens to stop the bleeding in one direction while leaving the wound wide open in another.
The fix that is right
Trust proxies by address, not by count:
app.set('trust proxy', ['127.0.0.1', '::1', '10.0.0.23', '10.0.0.238']);
Express walks the chain right-to-left, skipping addresses in the trusted set, and stops at the first one that is not. Injected values are never in that set and always sit further left than the genuine client address, so they can never be selected. The list cannot be shifted by adding entries.
Verified with a request carrying X-Forwarded-For: 1.2.3.4: the application still recorded 73.109.71.133. That's the kind of test result that makes you breathe a sigh of relief — and then immediately wonder why you didn't do it this way from the start.
Testing this properly is harder than it looks
Two traps caught me, and both make a broken limiter look fixed.
Faking the header from one machine proves nothing. If trust proxy is correct, your injected X-Forwarded-For is ignored — so every request still keys on your one real address, and you still share a bucket with yourself. That is indistinguishable from the bug. Youneed two genuinely different public addresses, or you need to read req.ip directly.
Machines on the same NAT share an egress address. I "confirmed" the problem from three different hosts before realising all three left the building through the same address. They shared a bucket whether the setting was right or wrong.
The only reliable test is the boring one: make the application tell you what it thinks the client address is, and check that it matches a client you can identify.
Why the blast radius is larger than it seems
A shared bucket does not degrade gracefully. It fails hardest exactly when success arrives.
Our worst case is a link doing well on an aggregator. Every article view calls a JSON endpoint, so a few hundred readers arriving inside a few minutes exhaust 600 requests, and everyone after that gets "Too many requests" — including the ones who would have stayed. Crawlers draw from the same pool, so a Googlebot pass during a traffic spike makes it worse.
The bug is invisible until you get traffic, and then it eats the traffic. That is a bad shape for a failure.
What I would tell myself
- Read what the application records, not what the topology implies. Config files cannot tell you about a hop nobody documented.
- Prefer the address list. It is barely more typing and it is not spoofable.
- A test that cannot fail is not a test. "I injected a header and nothing changed" is the expected result in both the working and broken cases.
- Check
ratelimit-*response headers before blaming infrastructure. They tell you immediately whether the limit is yours or your proxy's.
If you want to check your own setup right now, the fastest version is: log req.ip on any endpoint, load it from your phone on mobile data, and see whether the value matches your phone's address or your load balancer's. If it is the load balancer's, every visitor you have is sharing one bucket.
Frequently Asked Questions
What is a trust proxy misconfiguration, and how does it affect Express rate limiting?
A trust proxy misconfiguration occurs when a proxy server is not correctly configured to share client information with the Express server. This forces all clients to be assigned to the same bucket, resulting in a 429 error for all visitors when one busy client exceeds the rate limit. This is because the proxy server is sending the same client IP information to the Express server.
I've read that the obvious fix is to set trust proxy to true, but isn't that the solution?
While setting trust proxy to true might seem like a straightforward fix, it's not entirely correct. Setting trust proxy to true allows Express to access the X-Forwarded-For header, but if the proxy is spoofing or manipulating this header, it can lead to more issues. You should consider implementing other security measures to verify the client's IP address.
How can I spot a trust proxy misconfiguration in my Express application?
To spot a trust proxy misconfiguration, you can check your Express server logs for multiple clients being assigned to the same IP address. On top of that, you can use the 'x-forwarded-for' header to verify the client's IP address. Refer to ScholarNet AI's documentation for more information on how to implement this header in your Express application.
What is a hop count, and how can it be spoofed?
A hop count refers to the number of network hops a packet takes to reach its destination. Spoofing a hop count can be done by a malicious proxy server, which can increase the hop count to make it appear as though a client is from a different location. This can lead to a trust proxy misconfiguration, as the Express server relies on the client's IP address to determine rate limiting.
How can I implement a secure trust proxy configuration in my Express application?
To implement a secure trust proxy configuration, you should use a combination of the X-Forwarded-For header and hop count verification. You can also use Nginx to act as a trusted proxy server, which can help to mitigate spoofing attacks. Refer to the Nginx documentation for more information on how to configure a trusted proxy server.
Beyond Rate Limits: The Broader Security & Logging Implications
While an incorrect trust proxy setting primarily manifests as a rate-limiting nightmare, its ripple effects extend much further across your application's security and operational logging. Without proper configuration, your Express application might incorrectly log the proxy's IP address instead of the actual client's IP. This distorts analytics, making it appear as if all your traffic originates from a single source.
More critically, misconfigured proxies can open doors to IP spoofing vulnerabilities. If your application relies on IP addresses for any security checks—such as blocking specific regions, implementing IP-based authentication, or detecting suspicious activity—an incorrect trust proxy setup means these checks are easily bypassed or rendered ineffective. Your application believes it's seeing the proxy's IP, making it blind to the real, potentially malicious, client.
For students building projects with authentication or sensitive data, understanding this interaction is paramount. It highlights why blindly copying deployment configurations can lead to subtle yet significant security holes. Always verify that your application correctly identifies the true client IP, not just for rate limiting, but for every decision where client location or identity matters.
Debugging Your Proxy Chain: Practical Steps for Students
When you suspect a trust proxy issue, active debugging is your best friend. Instead of guessing, insert temporary logging into your Express app to inspect what it's seeing. Create a simple route like /debug-ip that returns both req.ip and the raw X-Forwarded-For header. This will immediately show you if Express is correctly parsing the header and identifying the client's IP.
- Inspect
req.ip: This is what Express believes is the client's IP after consideringtrust proxy. - Check
req.ips: Iftrust proxyis configured as a number (e.g.,1for one hop), this array will show the chain of IPs. The last one should be the client. - Log
req.headers['x-forwarded-for']: This raw header value reveals what your proxy (e.g., Nginx) is actually sending. Compare it toreq.ip. - Use cURL or Postman: Test your
/debug-iproute directly from your local machine and from behind your proxy. Observe the differences.
This hands-on approach helps demystify the proxy chain and directly correlates your configuration with runtime behavior, a vital skill for any developer.
Leveraging Tools for Robust Deployment: Don't Guess, Verify
Setting up a production-ready Node.js application behind a proxy, whether it's Nginx, a cloud load balancer, or a platform like Heroku, requires careful attention to detail. Don't just copy-paste configurations