How to Deploy Self-Hosted Apps Seamlessly: Nginx & PM2

A deploy that fails is a good afternoon. You get an error, you fix it, you move on. The expensive ones are the deploys that report success, change nothing a user can see, and leave you debugging code that was never running.

I remember one night at 2am, staring at a dashboard that said "Deploy Successful" while the feature I'd built was nowhere to be found. Every trap below cost me real time on my own machines, and all of them share a shape: the thing I edited and the thing being served were not the same thing, and nothing in between said so.

The proxy is not sending traffic where the comment says

I deployed a correct change, verified it on the box, and it was invisible publicly for twenty minutes. The gateway config said this:

# Proxy to app VM at 10.0.0.171
proxy_pass http://10.0.0.238:80;

The comment had been true when it was written. Since then a DHCP lease had moved, .238 had become an entirely different machine, and the change I had deployed to .171 was sitting on a box that no longer received the traffic.

Nothing was broken, which is why it took twenty minutes. Both hosts answered, both served the site, and only one of them had my change. The comment was the most confident thing in the file and the only part of it that was not executed.

Overlooked Configurations Can Cause Success-Faking Deploys

One common yet easily overlooked configuration that can cause a successful deploy to serve the old application is incorrect root path settings in the web server or application server. For instance, in an Nginx configuration file, setting the root path to a location where the old code resides instead of the new code will cause a successful deploy to report back that the deploy has been successful when, in fact, nothing has changed.

To avoid this issue, always verify that the root path points to the correct location before starting a deploy process. Using a version control system like Git and committing changes with detailed commit messages can help keep track of changes made and make it easier to troubleshoot issues like these.

Additionally, utilizing tools like ScholarNet AI can aid in automatically reviewing and suggesting changes to web server configurations, including root path settings, to prevent success-faking deploys.

Environment Variables Can Save the Day or Fool Users

• Environment variables in your application can play a crucial role in determining the application's behavior during a deploy. Incorrectly set environment variables can cause the application to continue using the old code or configuration, even after a successful deploy.
• Always review and double-check environment variables before and after a deploy, especially if changes were made to the application code or configuration.
• Consider using tools like dotenv to manage environment variables in your application, making it easier to switch between different environments and configurations.

Monitoring and Logging for Success-Faking Deployments

In some cases, a successful deploy might report back that everything is working as expected, but in reality, the new code is not being executed. This can be due to various reasons such as incorrect configuration, cache issues, or even misconfigured monitoring tools.

To catch and prevent success-faking deployments, it's essential to have robust monitoring and logging in place. This includes setting up logging for the web server, application server, and any other relevant components. Regularly reviewing these logs can help identify and address any issues before they become major problems.

Don't forget to also monitor for any unusual traffic patterns or application behavior after a deploy, as this can be an early indicator of an issue with the new code or configuration.

When two boxes both look right, stop reading names and check identity. SSH host key fingerprints differ per machine; so do MAC addresses. A hostname is a claim, a fingerprint is not. As a senior dev once told me, "The network doesn't care what you named it. It only cares where the packets land."

A backup file inside sites-enabled/ is a live config

nginx includes that directory with a bare glob:

include /etc/nginx/sites-enabled/*;

There is no extension filter. A mysite.conf.bak next to mysite.conf is not a backup, it is a second server block claiming the same server name. nginx starts, warns conflicting server name … ignored on stderr, and serves whichever it loaded first.

The failure is nastier than a syntax error because the site stays up. You edit the real file, reload, and get the other one's behaviour. I once spent a whole afternoon convinced I'd lost my mind, changing settings that simply weren't the ones being read. Keep backups anywhere outside the include path, and read nginx -t output rather than only its exit code — the conflict is a warning, not an error, and the exit code is zero.

The status code cannot tell you whether the page exists

On any single-page app with a catch-all, every URL returns 200. A route you never built, a page whose server-side rendering silently failed, a typo — all 200, all serving the same generic shell.

So "it returns 200" is not evidence of anything. Three things that are:

• The title. A real page has its own; a fallback serves the site-wide default.
• The etag. A static file gets a strong etag; a rendered response usually gets a weak one (W/"…"). If the page you expect to be rendered has a strong etag, you are being served a file.
• Cache-control. Rendered routes and static shells are usually configured differently. When they differ, the header tells you which path answered.

Pre-compressed assets make a good deploy look failed

With gzip_static or brotli_static, nginx serves the .br or .gz sitting next to the file. Then:

curl -s https://example.com/ | grep my-new-string     # nothing
curl -s --compressed https://example.com/ | grep my-new-string   # there it is

Without --compressed you get compressed bytes, grep finds nothing, and it looks exactly like a deploy that did not land. I have re-run a deploy that had already worked because of this. The second time, I felt like an absolute fool. Put --compressed in your fingers permanently; it costs nothing when the response is plain.

Your process manager's environment outlives your config file

Two separate traps, and they compound.

The first: pm2 stores a copy of the environment at the moment a process is first started. Editing the .env file afterwards changes nothing, because the file is not what the process is reading — a snapshot of it from weeks ago is. You can read the correct value in the file with your own eyes while the process runs on a different one.

The second is worse, because the command looks like it addresses exactly this:

# does NOT re-read ecosystem.config.js
pm2 restart myapp --update-env

# does
pm2 restart ecosystem.config.js --only myapp

--update-env refreshes the process from the environment of the pm2 daemon, not from your ecosystem file. If you added a variable to that file, this command will not pick it up, and it will report success. I watched a variable stay unset across three restarts before I stopped believing the command name. It was a humbling week.

Verify from the process, never from the file. On Linux the running environment is readable directly:

tr '\0' '\n' < /proc/<pid>/environ | grep MY_VAR

That is the only answer that describes what is actually running.

cp -r dist/* /var/www/site/ never deletes

Two consequences, one harmless and one not.

The harmless one: every superseded hashed chunk stays in the served root forever. Untidy, and it slowly makes the directory useless for working out what is current.

The one that bites: some files live in that directory and are not produced by your build. Sitemaps, robots.txt, llms.txt, verification files dropped in by a search console. A copy leaves them alone — but the moment anyone reaches for rm -rf to clean up the accumulation, they go, and nothing in the build can bring them back. I lost a verification file that way once, and getting re-verified by the search console took three days.

Back those files up as the first step of the deploy, and check they are still there as the last step. That way the answer to "did the deploy eat the sitemap?" is a line of output rather than an investigation.

You are testing one of two processes

If an app runs behind more than one process — a canary, a second instance for another hostname, a worker sharing the codebase — then deploying to one and testing that one proves nothing about the other. Whichever you happened to check decides what you believe, and users are distributed across both.

This is easy to leave in place for months because it fails asymmetrically: half the traffic gets the new build, nobody reports a fault, and the half on the old build looks like ordinary noise. Enumerate the processes, deploy to all of them, and test each by the address a user actually reaches rather than by localhost.

What actually verifies a deploy

Everything above comes down to one habit: verify by the artefact, from outside.

• Fetch the public URL, the way a stranger would — not localhost, not the origin behind the proxy.
• Grep for a string that s

None of that is sophisticated. It is just the difference between checking what you changed and checking what is being served, and on every occasion above they had quietly stopped being the same.

📗 Studying for Security+, CCNA or an AWS cert?

Paste your own notes, a config, or an exam objective and get flashcards and practice questions back. Free, no signup. Built by someone who runs the same stack you do.

Certifying this year? Study from your own material

Turn exam objectives, lab notes and docs into flashcards and practice questions. Free account saves your decks and unlocks the AI tutor. One email, no password.