Build a two node cluster
One server gives you the GUI, the certificates and the WAF. Two gives you failover. This is the whole of adding the second one, plus what to check afterwards.
Before you start
The second machine needs the same things the first one did: Ubuntu 22.04 or newer, internet access and root. It also needs to reach the first machine on port 7444, and vice versa.
Do not copy the install folder across
It contains a settings file with the first machine's identity and its secret key. The second machine would come up believing it is the first one, and both would be sharing one secret key, which they should not. Install it fresh. The installer refuses to join when it spots this, and it is much easier not to do it.
Step 1: get a token
On the first server, go to Cluster and press Add a node. You get a command with a token already in it. Copy the whole thing.
The token works once and expires after two hours. If you take longer than that, come back and press the button again for a fresh one. Tokens are stored hashed, so a copy of the database does not hand anybody a working one.
Step 2: run it on the second server
sudo git clone https://git.failoverlb.net/failoverlb.git /data/docker/failoverlb \
&& sudo /data/docker/failoverlb/failoverlb_setup.sh \
--join 203.0.113.10:7444 \
--token <the token you copied> \
--allowlist 198.51.100.25/32Then build nginx on this machine too:
sudo /data/docker/failoverlb/nginx-installer.sh installWhat actually happens during a join
- The new node makes a key pair and a signing request. The private key never leaves that box.
- It sends the request and the token to the node you pointed it at.
- That node checks the token, signs the request, and sends back the new node's certificate, the cluster authority, and a shared key for signing messages.
- The new node asks for the rest of the roster, so it learns about every other member and not only the one it joined through.
After that, three separate things have to line up on every peer call: a client certificate signed by this cluster's own authority, a source address on the roster, and an HMAC signature made with a key only that node holds. The timestamp and a nonce are inside the signed part, so a captured request cannot be replayed.
Step 3: check they can see each other
Open the Cluster page. Both servers should be listed, one active and one standby, and both should say online with the same config version and the same nginx build.
| What you see | What to do |
|---|---|
| The second node is missing entirely | The join did not finish. The installer output says why. The usual cause is port 7444 being blocked between the machines. |
| It is there but says its credentials were refused | Nearly always a name problem. Check grep NFM_NODE_NAME /data/docker/failoverlb/.env on the second node. If it names your first server, the settings file was copied. |
| Different nginx versions | Rebuild the one that is behind. A fleet where they differ is one where a config can work on one node and be refused by the other. |
| Different config versions | Press Sync now, or apply once. They should converge within seconds. |
What active and standby mean
The active node is the one that accepts changes. The standby holds an identical copy of everything and is ready to take over. Both are serving traffic the whole time.
Standby does not mean idle
This catches people out. The standby is not a spare in a cupboard. It is a working server that happens not to be the one you make changes on. If you find a button grayed out, check the top right of the screen: you are probably on the standby.
Which node is active will change over time, usually because one was restarted. That is normal and needs no action. It does not matter which one ends up active.
Step 4: prove the failover works
- Note which node is active.
- Stop the manager container on that node, or shut the machine down entirely.
- Watch the other node's Cluster page. It should promote itself within seconds.
- Confirm your sites are still serving.
- Bring the first node back and watch it rejoin as standby.
Do this on purpose, in daylight, before you need it. A failover that has never been tried is a plan, not a capability, and the two look identical right up until the moment they do not.
What this does not solve
Traffic that was already heading for the node that died. Their computer looked up your address before it tried to connect, and it was given the address of a machine that has since gone. Only the lookup itself can send them somewhere else, which is what DNS failover is for.