7 Best Practices for Analyzing Self-Hosted Nginx Access Logs

An access log is the only record of who actually reached your machine, and almost every field in it is written by the person visiting. That is a strange property for a source of truth, and it is easy to forget while you are counting lines.

These are the ways I have misread my own logs, with the numbers that made each one convincing at the time.

A big number is not traffic until you know who made it

One page in my logs had 37,502 requests. That was, by a wide margin, the most popular thing I had ever published, and it was a page I had no memory of anyone caring about. I remember staring at that number at 2am, genuinely excited for a solid ten minutes before something felt off.

It was one IP address. nginx was already refusing it with 444, so not one of those requests was even answered. The traffic did not exist in any sense that mattered — but a count of log lines cannot tell the difference between thirty-seven thousand people and one persistent script.

Count distinct client addresses, and count them per document. It changes the shape of everything:

Using the Access Log to Analyze Network Traffic Patterns

When reviewing your access log, it's not just about counting the number of requests or identifying blocked IPs. You can also use the log to analyze network traffic patterns and gain insights into your website's behavior. For example, you can use the remote_addr field to identify users from a specific geographic region or the http_user_agent field to track changes in user agent strings over time.

In a self-hosted setup, it's possible to use ScholarNet AI to analyze your access log and identify trends in network traffic. By integrating ScholarNet AI with your access log, you can automatically generate visualizations and reports that highlight areas of interest.

When analyzing network traffic patterns, be sure to consider the following factors:

• Time of day and day of the week: Are there specific times when your website experiences a surge in traffic?
• IP address ranges: Are there specific IP address ranges that are making a large number of requests?
• User agent strings: Are there specific user agent strings that are being used by a large number of users?

Identifying and Blocking Malicious Activity

While reviewing your access log, it's essential to identify and block malicious activity, such as spam bots and vulnerability scanners. These types of requests can consume your resources, slow down your website, and even compromise your security.

To identify malicious activity, you can look for suspicious patterns in the access log, such as:

• A large number of requests from a single IP address.
• Requests with unusual or misspelled URLs.
• Requests with a high number of failed attempts to login.

Once you've identified malicious activity, you can use nginx's built-in IP blocking feature to block the offending IP addresses.

Optimizing Your Access Log Configuration

Finally, it's essential to optimize your access log configuration to ensure that you're capturing the data you need while minimizing the impact on your server resources.

Here are some tips for optimizing your access log configuration:

• Only log requests that are relevant to your business goals.
• Use a smaller log format to reduce the size of your log files.
• Configure your log rotation to prevent log files from growing too large.

# requests — flatters whatever is being hammered
awk '{print $7}' access.log | sort | uniq -c | sort -rn | head

# distinct clients per path — what you actually wanted
awk '{print $7, $1}' access.log | sort -u | awk '{print $1}' | uniq -c | sort -rn | head

The referrer is a string the client chose

The Referer header costs nothing to set to anything. Mine carried 8,689 hits claiming to come from http://www.google.com/, which is not how Google sends traffic and is not what Google's referrer looks like.

That is referrer spam, and it has a business model: publish a log summary or a public analytics dashboard, and the spammer gets a link. The traffic is the cost of placing the advertisement, not a mistake.

Treat referrers as unauthenticated. If a referral matters, corroborate it: does that page actually link to you, and does the request pattern look like a person following a link — one document, then the assets, from a browser-shaped client?

User agents are forged, including the ones you are inclined to trust

This is the one that fooled me hardest, because it survived the obvious filter. I learned this lesson the hard way when I proudly showed a colleague our growing organic traffic, only for him to raise an eyebrow and ask, "So why are all these 'Googlebot' requests hitting your .git directory?"

I filtered the log for a well-known crawler's user agent — the sort of string you would never think to distrust — and the requests it returned were for /.env, /.ssh/id_rsa, /.git/config and /actuator/health.

Real crawlers request content. Nothing indexing your site needs your private key. A vulnerability scanner had put on a costume that makes people whitelist it, and my filter dutifully counted it as an important visitor.

Two defences, and you want both. Restrict user-agent analysis to paths you actually publish, so a scanner cannot enter the sample no matter what it calls itself. And where a crawler's identity really matters, verify it the way the crawler operators tell you to — reverse DNS on the client address, then forward-resolve the name back — rather than by reading its self-description.

The device mix tells you whether a fleet is one machine

Real referral traffic looks like the world. A batch I checked recently came out as Windows 44, macOS 20, Linux 20, Android 18, iPhone 9, iPad 1 — messy, and messy in the proportions people actually own.

Another page looked comparably popular. Its clients were 341 Windows against 7 of everything else. Nobody's audience is 98% one platform. That is a fleet of identical headless clients, and the giveaway was not any single request but the absence of variety across all of them.

This works because it is expensive to fake convincingly. Rotating a user-agent string is trivial; producing a realistic distribution of platforms, versions and asset-loading behaviour is not, and almost nobody bothers. As one systems administrator I know puts it: "You'll never see a real user base that looks like a server farm."

"Indexing you" and "someone is asking right now" are different clients

Several AI products crawl with two distinct agents: one for background indexing, and one for fetches made because a person is asking a question this second. They look similar and they mean opposite things.

Collapse them and you get the worst of both. A handful of genuine live questions disappears into a pile of routine indexing, or a routine indexing sweep looks like sudden interest. I had both readings at different times, from the same log, depending on which agent I had lumped where.

Check the documentation for whatever is crawling you and split the agents apart before you count. Then count distinct addresses per agent, because one indexer making a thousand fetches is one visitor.

A shared public address makes per-IP limits everyone's problem

Every machine behind one NAT looks like a single client to anything counting by address. Free tiers, rate limits, "ten requests a day without an account" — all of it is scoped to the address, and your whole network is one address.

So a cron job on one box can exhaust the allowance your browser needs on another, and it presents as the provider throttling you for no reason. I once spent an entire afternoon convinced a rate limiter was broken, only to realize my own backup script was the culprit. It also means testing your own rate limiting from inside your network measures something different from what a stranger experiences.

When you are testing quota behaviour, use a credential rather than relying on the address, and check your own outbound IP before concluding that a limit is broken.

The log may not record which site served the request

The last one is the plainest. A combined log format records the path, not the host. If several of your vhosts share paths — /blog/, /api/, /health — then a line saying GET /blog/x does not tell you which site answered it.

I inferred the host from path shape for longer than I would like, which works until two sites share a path and then quietly stops working without telling you.

log_format withhost '$remote_addr $host "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"';
access_log /var/log/nginx/access.log withhost;

Add it before you need it. You cannot retrofit a field onto logs you have already written.

What to count

Reduced to a habit:

• Distinct addresses, not requests, and per document rather than in total.
• Corroborate referrers or ignore them. They are unauthenticated by design.
• Restrict any user-agent analysis to paths you publish, and verify identity by DNS rather than by the string.
• Look at the spread of platforms, not just the volume. Uniformity is the signal.
• Log $host today.

And keep the rejects visible. Every filter above throws data away, and a filter nobodyy can see is a filter nobody can correct — when I started printing what was being excluded and why, the first thing I found was a rule that would have discarded the one genuinely good referral source I had.

📗 Studying for Security+, CCNA or an AWS cert?

Paste your own notes, a config, or an exam objective and get flashcards and practice questions back. Free, no signup. Built by someone who runs the same stack you do.

Certifying this year? Study from your own material

Turn exam objectives, lab notes and docs into flashcards and practice questions. Free account saves your decks and unlocks the AI tutor. One email, no password.