Take a server out for maintenance
You need to reboot one of the load balancers, patch it, or move it. This is how to do that without anybody noticing. It is short, and doing it in this order is the whole point.
Before you touch anything
- Open the Cluster page and check both nodes are online. If the other one is not healthy, stop. You are about to be running on one node and it needs to be a good one.
- Open the Changes page and check nothing is saved but not applied. Applying somebody else's half finished work in the middle of maintenance is a bad afternoon.
Take it out of the DNS answer
If you use DNS failover, do this first and give it time.
- DNS Failover, edit the name, turn off the answer for the node you are about to work on.
- Save.
- Wait at least twice the TTL. With a TTL of 30 seconds, wait two minutes.
That is not the software being slow. It is the time for computers elsewhere to forget the old answer. Skipping it is the difference between nobody noticing and a few people getting errors.
Hand over being in charge
If the node you are working on is the active one, use the failover button on the Cluster page to hand over to the other. Doing it yourself is calmer than pulling the plug and letting the software work it out, and you can confirm the other node took over before you go any further.
Stop it coming back too early
Turn on forced standby for the node you are working on. That stops it becoming active again halfway through your work, for example after a reboot you did not expect to change anything.
Do the work
Reboot it, patch it, whatever you came to do. The other node is carrying all of the traffic. Nothing about this is urgent, which is the point of doing it in this order.
If the work is rebuilding nginx, do it from the NGINX Build page and run the check button first. The new binary is built while the old one carries on serving, your config is tested against it, and only if that passes is the running nginx replaced.
Putting it back
- Wait for it to come back and check the Cluster page shows it online.
- Turn off forced standby.
- Turn the DNS answer back on. It returns to the answer within a few seconds.
- Watch it for a few minutes before you go home.
The order going back in is the reverse
Health first, then being allowed to be in charge, then traffic. Turning the DNS answer back on before the node is actually well sends people to something that is not ready.
A checklist you can print
| Order | Coming out | Going back in |
|---|---|---|
| 1 | Check both nodes healthy | Check it is online again |
| 2 | Check nothing unapplied | Turn off forced standby |
| 3 | Turn off its DNS answer, wait | Turn its DNS answer on |
| 4 | Hand over if it is active | Watch for a few minutes |
| 5 | Turn on forced standby | |
| 6 | Do the work |
The thing people forget
Turning forced standby back off. A fleet where one node can never take over is a fleet with no failover, and nothing will remind you. It is the single most common thing left switched on after maintenance.
Doing the same for a backend
Retiring an application server is simpler and the principle is identical. Set its state to Draining and apply. It finishes what it is doing and takes nothing new. Watch the connection count fall to zero, then touch the machine.
Turning it off outright drops whatever it was in the middle of, which for a file upload or a long report is somebody's afternoon.