The occasional failure is inevitable in the cloud. Rather than fight this truth, Microsoft’s Azure team embraces it, writes Heather Nakama, software engineer for Azure Search.

That’s why the team plans for failure by designing its systems to be fault-tolerant, resilient and self-stabilizing. But how to verify those systems respond to failure as they should?

Enter Azure’s “Search Chaos Monkey.” Inspired by Netflix’s use of chaos engineering and the Netflix Simian Army, the Azure team created its own chaos monkey and let it loose in a test environment to smoke out problems.

“At Azure Search, chaos engineering has proven to be a very useful model to follow when developing a reliable and fault tolerant cloud service,” Nakama writes. “Our Search Chaos Monkey has been instrumental in providing a deterministic framework for finding exceptional failures and driving them to resolution as low-impact errors with planned, automated solutions.”