The Firewall Rule That “Worked for Years” — Until It Didn’t
The Everyday Situation
Nearly every enterprise firewall carries at least one rule like this. It was added years ago, under pressure, to “just make things work”.
- Marked as a temporary exception
- Uses a wide source or destination range
- Has no ticket reference, no owner, and no expiry
Initially, nothing breaks. Traffic flows. The incident is closed. The rule survives upgrades, audits, and even staff changes.
Then one day:
- A production service becomes unreachable
- Application teams insist “nothing changed”
- Firewall logs show denied traffic where it once passed
“This rule has been here for years.”
1. The Failure Timeline (How Outages Actually Form)
Firewall failures rarely happen instantly. They accumulate silently.
- T-3 years: Temporary ACL added during an outage
- T-18 months: New services reuse the same open path
- T-3 months: ACL reordering / NAT cleanup
- T-0: Production outage
- T+2 hours: Logs blamed, application suspected
- T+1 day: Old rule found — purpose unknown
The outage is not the event. It is the reveal.
2. Firewall Rules as Organizational Behavior
Every undocumented firewall rule is a mirror of how an organization operates.
| Firewall Symptom | Organizational Cause |
|---|---|
| Wide ACL ranges | Fear of outages |
| No comments | Time pressure |
| No expiry | Lack of ownership |
| “Don’t touch it” culture | Institutional memory loss |
3. The “Unknown Known” Problem
An unknown known is something the system depends on, but humans can no longer explain.
The firewall still enforces the dependency. The organization has forgotten why it exists.
This is why removing an old rule feels dangerous — even when no one can justify keeping it.
4. Common Firewall Anti-Patterns
- The Museum Rule: “Don’t touch it — it’s historical.”
- The Blanket Exception: Any-any with a vague comment
- The Shadow Dependency: Traffic works only because of it
- The Silent Deny: Implicit denies discovered during outages
5. Why Logs Don’t Save You
Logs tell you what was blocked — not why it was ever allowed.
Logs cannot answer:
- Why was this rule created?
- Who approved it?
- What business process depends on it?
By the time logs matter, institutional knowledge is already gone.
6. What Good Actually Looks Like
This is not about perfect documentation. It is about intent.
- Every rule has an owner
- Every exception has a review horizon
- Firewall rules are part of application architecture
- If a rule can’t be explained, it shouldn’t exist
7. A Metaphor Worth Remembering
Firewall rules are load-bearing walls.
You don’t know which ones matter until you remove the wrong one.
Most outages aren’t explosions. They’re archeological digs.
8. Reader Challenge
Pick one firewall rule older than two years.
Try to explain it to a new engineer without checking tickets or emails.
If you can’t explain it, the system already knows more than you do.
Further Reading
The behavior described in this article is not theoretical. It directly reflects how modern firewalls, logging systems, and ACL engines behave in production environments.
Final Thought
Firewalls are deterministic. They do exactly what they are told.
What fails is not the device — it is our memory of why decisions were made.
A rule that “worked for years” is often the most dangerous one to keep.
