Size the servers
Work out the peak number of requests per second the fleet has to serve. Multiply by the average size of a response. That is your bandwidth, and it is the number that will run out first.
The rule of thumb
| Requests per second | At 28 KB per response | In bits |
|---|---|---|
| 1,000 | 28 MB/s | 224 Mbit/s |
| 2,000 | 56 MB/s | 448 Mbit/s |
| 5,000 | 140 MB/s | 1.1 Gbit/s |
| 10,000 | 280 MB/s | 2.2 Gbit/s |
Add something for headroom and provision the link for that. Then pick a processor and memory from the table below, which are comfortable rather than tight.
Recommended sizes
These assume a pair of load balancers where either one has to be able to carry everything on its own. That is the point of having two, so size each for the full peak rather than half of it.
| Peak load | vCPU | Memory | Network | Disk |
|---|---|---|---|---|
| Up to 2,000 req/s | 4 | 4 GB | 1 Gbit/s | 40 GB SSD |
| Up to 5,000 req/s | 8 | 8 GB | 2.5 Gbit/s | 80 GB SSD |
| Up to 10,000 req/s | 16 | 16 GB | 10 Gbit/s | 160 GB SSD |
Do not go below four processors even for a quiet site. An average hides bursts, and the two things that burst here are TLS handshakes when a lot of visitors arrive at once, and certificate renewal.
Keep both load balancers the same size
A pair where one is smaller has the capacity of the smaller one, and you find that out at the moment the larger one fails.
What changes these numbers
- Smaller responses move the limit from the network to the processor. A site serving small API replies rather than pages needs more processor and less bandwidth than the table suggests.
- Visitors who connect once and leave are much more expensive than visitors who stay, because each new connection pays for a TLS handshake. Heavy connection churn needs more processor.
- Caching on the load balancers trades memory and disk for backend load, which changes where the pressure sits.
- The WAF costs processor. Inspecting every request is not free. Budget for it if you plan to run it in blocking mode on a busy site.
Where these numbers come from
An hour of continuous traffic against a real two node fleet, serving a real site through its real backends. These are what was measured, not what was expected.
| What | Result |
|---|---|
| Requests served | 7,541,340 in one hour |
| Sustained rate | 2,095 requests per second |
| Data transferred | 204.88 GB |
| Timeouts | none |
| Failed connections | none |
| Response time, 90th percentile | 131 ms and 81 ms |
| Processor at peak | 8% of 16 cores |
| Memory at peak | 771 MB of 3,393 MB |
Nothing degraded across the hour. Response times at the end matched the beginning, memory moved by about twenty megabytes and did not trend upward, and connection counts held steady. The tunnel carried backend traffic throughout without interruption, and the other sites on the same fleet were unaffected.
The honest caveat
The test could not find the limit
Throughput stopped rising at about 3,000 requests per second, and at that moment the load balancers were running at roughly 6% of their processors. What ran out was bandwidth on the machine generating the traffic, not anything belonging to the fleet.
So the real capacity of these servers is not known, and is higher than the numbers above, possibly much higher. Treat the recommendations as safe rather than precise. To find the true figure, generate load from several machines on separate connections and raise it until the load balancers themselves are busy.
What the hour does establish is the thing an endurance test is for. Under sustained heavy traffic this does not leak, does not slow down, and does not need attention.
When to add a third node
Not usually for capacity, because two nodes each sized for the full peak already have a great deal of it. The reasons that actually matter:
- Quorum. Three nodes tolerate one being unreachable while still having a majority. Two nodes do not.
- A third location. If you have a third place to put a machine, DNS failover has a third answer to hand out.
- Maintenance comfort. Taking one of three out leaves two. Taking one of two out leaves you with no margin at all.