There is a version of chaos engineering that gets talked about a lot. Netflix. Chaos Monkey. Dedicated platform teams running sophisticated failure injection frameworks with dashboards and gameday ceremonies and automated rollback. It sounds impressive because it is impressive. It is also not how most teams do it, or need to do it.

Here is how we did it at Salesforce, with a homegrown tool and a spreadsheet.

What we actually built Link to heading

The tool itself was not sophisticated. It was a set of scripts that could target specific infrastructure components and inject failures in a controlled way. The spreadsheet was where we tracked everything: what experiment we ran, what we expected to happen, what actually happened, what we fixed, and when we planned to run it again. That spreadsheet was not a compromise. It was the right tool for the job. It gave us a written record of every failure we had deliberately introduced and what we learned from it, which turned out to be invaluable during real incidents when we needed to answer the question “have we seen this before?”

The discipline was not in the tooling. The discipline was in treating every experiment like a real investigation. Hypothesis, expected behavior, actual behavior, follow-up actions. Every time.

The most revealing experiments came from a series of tests we ran Salesforce’s internal distributed database system, running on Kubernetes across three availability zones with ZooKeeper handling coordination. The database was grown to over 1PB of data for the test. We wanted to understand how it behaved under failure conditions at that scale before those conditions found us.

The first test was Zookeeper leader failover. We ran a controlled failover and measured how long it took the cluster to recover. Over fifteen minutes. That is too long for the SLOs we were trying to hold. The instance type was the culprit, and it was directly actionable. We changed it. We would not have known without the experiment.

The second finding was more interesting and more humbling. We ran a full AZ scale-down and scale-up, over 200 nodes, to simulate what would happen if we lost an availability zone. During the scale-up we hit a wall. AWS has a smart IP allocation feature that is supposed to distribute new IP intake across subnets. Under a mass scale-up event it does not work the way the documentation suggests. It falls back to drawing IPs from the oldest subnet only. Two AZs exhausted their IP space completely during our test.

We would never have found this in normal operations. In a real AZ failure it would have looked like a mystery networking issue with no obvious cause. Because we found it during a controlled experiment, we documented it, added it to our runbooks, and set a constraint on our autoscaler: never churn more than a certain number of nodes at once. That constraint is still protecting the cluster.

What an MVP chaos program actually looks like Link to heading

You do not need Chaos Monkey. You do not need a dedicated platform. Here is what you actually need:

Start with the simplest experiments. Kill an entire AZ and watch what happens. Delete pods in a deployment and see how traffic redistributes. Restart a database replica and measure how long it takes for connections to recover. These are not exotic experiments. They are the failure modes that will actually happen to you in production, and you want to know how your system responds before the failure is unplanned.

Track everything in a shared document. What you ran, when you ran it, what you expected, what you got, what you changed as a result. The spreadsheet is not embarrassing. The spreadsheet is institutional memory.

Run experiments during business hours with the right people in the room. Not at 2am, not alone. Chaos engineering is not about being brave. It is about being prepared. You want your on-call engineers watching the blast radius in real time so they learn what the failure looks like before they have to respond to it for real.

Set a blast radius before you start. Know what you will do if the experiment goes wrong. Have a rollback plan that takes less than five minutes to execute. Chaos engineering is controlled failure, and the control is as important as the failure.

Also, test your incident management system. Run mock drills to simulate an incident, see who responds. Observe how communication is handled between teams and to external customers and improve on that.

The honest version Link to heading

Most teams skip chaos engineering because it feels risky, expensive, or like something only big companies do. The reality is that the risk of not doing it is higher. The ZK failover we found would have happened eventually in production. The AWS IP exhaustion bug would have hit us during a real AZ failure at the worst possible time. The experiments we ran did not prevent those failure modes from existing. They just made sure we found them first.

You do not need a platform. You need discipline, a shared document, and the willingness to deliberately break things before something else does it for you.