Health checks
11 answers
Noticing a broken backend before a visitor does.
Looking for something specific
The searchable index covers all 326 answers at once and filters as you type.
57 Turn on active health checks
Open the pool, go to the Health checks tab, and tick Check these backends on a schedule. Set the type to HTTP and give it a path.
With it off, nginx only notices a bad backend after a real request to it has already failed, which means a visitor got the error first.
58 Choose what the health check should ask for
Point it at something that genuinely tests the application, not the front page. A server whose web process is running but whose database has gone still accepts a connection and still serves a front page, so a shallow check calls it healthy while every visitor gets an error.
A path that touches the database is the right kind of check.
59 Catch a backend that returns 200 while it is broken
Fill in Body must contain with a short string that only appears when things really work. Plenty of applications answer 200 while rendering an error page, and this is the cheap insurance against that.
60 How often should health checks run?
Five to ten seconds suits nearly everybody. Lower notices a failure sooner and asks your servers more often. If the check is expensive, make the endpoint cheaper rather than the interval longer.
61 Stop a flapping backend causing constant reloads
Raise Fall so it takes more consecutive failures to be marked down, and raise Rise so it takes more passes to come back. Reloads are already debounced, so twenty backends flapping at once is one reload rather than twenty.
If it keeps flapping, the reload rate is not really the problem. Find out why the backend keeps failing.
62 Check a different port from the one traffic uses
Set Check port on the pool. Useful when the application serves traffic on one port and exposes a health endpoint on another.
63 Set the Host header for a health check
Fill in Host header on the pool. Needed when the backend hosts several sites and picks one by name, otherwise your check reaches whichever site that server considers its default.
64 One node says a backend is up and the other says it is down
That is not a bug, it is the useful part. Each node checks separately and the answers are kept apart. A backend one node can reach and another cannot is a routing or firewall problem, and seeing it split out that way usually tells you the answer immediately.
65 Health checks are filling my application logs
Point them at a path you can filter out, or at a route your application logs at a lower level. One line per check per node is the cost of knowing a backend is alive before a visitor does.
66 Health checks fail after I turned on backend trust
They should not. The checker sends the token too, precisely so this does not happen. If they do fail, check the backend is looking for the header name the pool page shows, and that you have not rotated without activating.
67 Test that failover actually works
Stop the application on one backend, or take it off the network. Within a few seconds the pool should show that server red and the others green, and the site should keep working throughout. Start it again and watch it come back.
If the site kept working but the pool never went red, your health check is not testing anything useful.