Roll out the WAF without breaking things
Every real application does something the rules think is suspicious. A rich text editor posts HTML. A search box takes quotes. An upload form sends things that look like scripts. Going straight to blocking is how people conclude that WAFs do not work.
Before you start
- Settings, WAF rule set. Choose the long term support track.
- Save the page first, then press Install it now.
- Check every node reports the same version. A node without the rules refuses the whole configuration and the apply is turned away.
Having the rules on a node does not switch anything on. A site is only checked once its own WAF setting is turned on, and that starts in watching mode.
If the WAF module is not in your build
Turning the WAF on in the GUI rebuilds nginx with ModSecurity, using the same path the version upgrades already take. It is not compiled in by default because a statically linked nginx that cannot find its ModSecurity library will not start at all, which would take down every node over a feature most of them are not using.
Week one: watch and change nothing
- Pick one site. Not your busiest, and not something nobody uses either.
- WAF tab, mode Watching, refuse nothing. Apply.
- Leave it alone for a week.
- Then look at what fired.
You are looking for two things: rules that fire constantly on your own normal traffic, and rules that fire on things that are obviously attacks. The first list is your tuning work. The second list is why you are doing this.
Week two: run a learning session
- WAF tab, Learning session.
- Put in the addresses your testers will come from. An office, a VPN, a build runner, somebody's laptop. More than one is normal.
- Name the round, such as "checkout regression, August release".
- Press Start watching.
- Exercise the application properly. Click everything. Upload something. Use the search. Post text with quotes and angle brackets in it. Log in and out. Use the admin area.
- Press Stop watching.
Why it insists on addresses
The internet does not stop while you test. Without a filter, the session collects every scanner and every genuine probe that arrived during the window, and you would end up excusing rules on the strength of somebody else's attack.
Tick the exceptions, not the rules
Look at what fired from your tester addresses. For each one, ask: is this my application doing something normal, or is this something I would want refused?
- Your application: tick it. It becomes an exclusion that stops that one rule firing on that one path.
- An attack, or something you do not recognize: leave it. If your own testing triggered it and you do not know why, that is worth understanding before you excuse it.
Save the ticked ones as a profile. Name it after the application rather than the
site: wordpress-4x or internal-crm tells the next person what it is for.
shop-exceptions-final-v2 does not.
Check the stars in learned paths
Paths that hold an id are collapsed to a star automatically, so /user/*/profile covers the account created tomorrow. A star in the wrong place allows more than you meant. A missing one refuses everything created after today, and that only shows up when a new record is made.
Week three: still watching, with the profile on
Apply the profile to the site and stay in watching mode for a few more days. Then look at the hit list again. It should be nearly empty for your own traffic. If it is not, you missed part of the application: run another round over the parts you forgot. Rounds add up.
Week four: start blocking
- Move to Block known malicious. This refuses the things nobody argues about, at a low strictness.
- Watch for a week. Watch your support inbox as well as the hit list.
- If nothing breaks, move to Block what the attack rules catch.
- Raise the strictness one step at a time, and only after watching at the level below it.
Set up a 403 error page first
By default a blocked visitor gets the bare nginx page. An error page template covering 403 lets you say what happened and how to get in touch, which turns an angry support ticket into a useful one.
Rolling it out to the rest
For each additional site running the same application, set it to the profile you already made and go straight to watching. If the hit list stays quiet for a week, move it to blocking. You do not need a fresh learning session for a second copy of the same thing.
For a genuinely different application, start at week one again.
Keeping it honest
- Watch the rule set version. The profiles page shows whether each profile still matches the rules installed today. A profile that has quietly stopped applying is worth knowing about.
- Review the exclusions once a year. An exclusion added because the old editor posted raw HTML is dead weight once the editor has been replaced, and dead weight in a WAF profile is a rule not protecting a path that now has nothing on it.
- Re-run a learning session after a big release. New features mean new paths and new shapes of request.
What it costs
Inspecting every request is not free. Expect a measurable increase in processor use and a small amount of added latency per request. On the fleet the load test numbers on this site came from, the load balancers were running at around 8% of sixteen cores under 2,000 requests a second, so there is a great deal of headroom. Measure your own rather than trusting that.