Chaos Engineering And Disciplined Experimenting with Casey Rosenthal

Casey Rosenthal is the co-founder and CEO of Verica and the former Engineering Manager of the Chaos Engineering team at Netflix. He is an author and thought leader in chaos engineering, “a discipline of experimenting with software systems in production in order to build confidence in the system's capability to withstand unexpected and turbulent conditions.” Casey was an early engineer and champion of chaos engineering, bringing together people from companies like Netflix, Google, Facebook, and Amazon to explore the field. In this week’s Unlearned Podcast, Casey and Barry O’Reilly talk about the chaos engineering domain and how to apply its principles to build high-performance teams and businesses.

Origins of Chaos Engineering
Netflix’s migration to the cloud, in particular the sudden outages and service disruptions that would occur, spurred the creation of a program they called Chaos Monkey. “So Chaos Monkey would for each service inside Netflix, every day it would randomly choose an instance and turn it off,” Casey tells Barry. The underlying principle was that once engineers knew a problem existed, they would fix it. “It changed their behavior by aligning the organization around the business problem that needed to be solved,” he remarks. He shares the early Chaos Community Days, bringing colleagues together from leading tech companies to build the discipline they would call chaos engineering.

Navigating Complexity
Chaos engineering assumes that you already have complexity in your system. “This is engineering to navigate it, or to surface it so that you’re aware of it,” Casey explains. Once you’re aware a problem exists, you can take steps to fix it. It’s a proactive approach to improving availability and security, which improves your system overall. He shares an example of how United Health Group was able to discover a system vulnerability they didn’t know they had, and allocate appropriate resources to strengthen their position. Barry comments, “The thing that’s very contrary with this is that it's not about people trying to predict the future, it's about them having the data to understand how the systems are performing and then taking action based on that.”

Relearning Leadership
“Unlearning management is relearning leadership”, Barry says. He asks Casey to share lessons he learned that he is bringing to his new company. Managers are creatures of habit, and that holds them back, Casey responds. “Most of us think we're making decisions when we're not; we're just following habit.” He tries to formulate his own management principles and strategies in his company, instead of following traditional ideas. He believes a manager’s job is to ensure their team has the context they need to make the right decisions. His litmus test is this: if your employees can explain why what they’re working on is the most important thing they could be working on for the company right now, then you are a successful manager.

Read the full show notes at BarryOReilly.com

Resources
Casey Rosenthal on LinkedIn | Twitter 
Verica.io