Ep. 196 - Harvard's Stefan Thomke, Author of Experimentation Works: The Surprising Power of Business Experiments

Hey listeners, before we start this week's episode, I wanted to let you know that we recorded a number of interviews before the Corona virus disruptions started, wanting to give some context before we jump into some of these shows. Thank you very much for listening, being part of the Inside Outside Innovation community, we look forward to talking more about the disruption of the Corona virus and other things.

On this week's episode of Inside Outside Innovation, we sit down with Stefan Thomke.  Stefan is a Harvard professor and author of the new book Experimentation Works:  The Surprising Power of Business Experiments. We talk about why experimentation matters, how to overcome the fear of failure and some of the latest trends that are driving companies to include a rigorous experimentation process into their business. 

Inside Outside innovation is the podcast that brings you the best and the brightest in the world of startups and innovation. I'm your host, Brian Ardinger, founder of InsideOutside.IO, a provider of research events and consultant services that help innovators and entrepreneurs build better products, launch new ideas, and compete in a world of change and disruption. Each week we'll give you a front row seat to the latest thinking tools, tactics, and trends and collaborative innovation. Let's get started. 

Brian Ardinger:  Welcome to another episode of Inside Outside Innovation. I'm your host Brian Ardinger, and as always, we have another amazing guest. Today we have Stefan Thomke. He is a Harvard professor and author of a new book that just came out called Experimentation Works: The Surprising Power of Business Experiments. Welcome to the show. 

Stefan Thomke: Thanks Brian.

Brian Ardinger: I'm excited to have you on the show because obviously in this corporate innovation space and even in the startup innovation space. Experimentation has gotten a lot more buzz as people try to understand how to navigate the world of uncertainty. I wanted to start the conversation by talking a little bit about why is experimentation so important.

Stefan Thomke: Brian, I mean that's a great question, and in fact, it's the uncertainty that makes experimentation so valuable. If you think about uncertainty, you can think about different types of uncertainties that companies face every single day. At one level there's R & D uncertainty. You know, I've had the pleasure of working with lots of R & D organizations over the years. I've been at this for more than 25 years. And there the question is, it could be a product, a service, or a new customer experience, does it work as intended? Another set of uncertainties is what I call scale up uncertainty. If I'm sitting on a scale upside where I have to scale up the service or scale of production, I worry about a different set of questions. I worry about whether something can be effectively made or scaled up.  You know, worry about can it be done at high quality, low cost, large volume, and so forth. And if I'm customer facing, I worry about yet another set of questions. And that is, does anybody want it? If they say they want it, do they really mean it and are they willing to pay for it. 

And then finally, if I'm running a business unit, of course, I need to make an investment decision, and the question here or the uncertainty here is the opportunity big enough? Does it justify the resource investment? And the problem, of course here is that the tools that we have, like, you know, calculating an ROI on net present value and also all these kinds of wonderful tools, they start breaking down, when you're dealing with a lot of uncertainty, when something is really novel, you know, how do you put a net present value on something that doesn't exist yet?  And so these are the sets of uncertainties that we face every single day.  My argument is that experimentation is really the best way to address it because it gives me information about cause and effect, which a lot of the other ways of approaching this problem don't do.

Brian Ardinger: Experimentation...A lot of people think about it as the scientific method and get scared from it. A lot of folks in business world are not necessarily scientists and that, talk a little bit about what are the major barriers to business folks understanding what experimentation means and then how to adapt that. 

Stefan Thomke: That makes sense to do a quick detour and ask ourselves, what do we really mean by experiments? And let me tell you what I don't mean often when people talk about experiments, what they're really saying is, I've tried something. It's often the way you use it in the English language or in companies what I've always run into is we've tried something, and it didn't work and therefore it must've been an experiment. Right? That's not really what I mean. I mean, a much more disciplined approach, like the scientific method, which by the way was essentially conceived 400 years ago, almost exactly 400 years ago in 1620 by Francis Bacon when he wrote the book Novum Organum, which was a new instrument for building an organizing knowledge. Now, it was done for science back then, but my argument, it's the same thing for knowledge about management and knowledge about behaviors and so forth. The experiment is at the heart of actually finding or building this new knowledge and organizing knowledge. 

Now, what is an experiment? Let me give you the gold standard. And then we can work from there. In an ideal experiment, you've got to test. And what you want to do in an ideal experiment is you want to separate what we call independent variables. This is the presumed cost, and it's something that you're trying to change from a dependent variable, which is the observed effect, while holding everything else, all the other potential changes constant. Here's an example, so imagine you've got a sales force. And you're coming in and you want to give them a bonus. The independent variable is the bonus. And the dependent variable, the observed effect, would be a lift in sales, for example. And what I want to do is I want to understand whether one causes the other to happen, so causality again. Without having the experiment polluted by a lot of other things that are changing. For example, you know, whether the salesperson doesn't appeal well that day or so on and so on.  

So that's really the gold standard that we're after, and in a real-world experiment as opposed to science where I can create a laboratory where I can control a lot of these other things. We randomly distribute all the other things that could influence the experiments evenly across the people that we're testing on. And we want to do is of course blind, so we don't know who we're experimenting on and they don't know that extra, they're being experiment on. But ff course, in the real world, there are some limitations in terms of what we can and cannot do. The hypothesis, of course, is at the heart of this, and your listeners may remember that the scientific method maybe from high school science days.

Brian Ardinger: Talk a little bit about how companies can start both identifying what to experiment on and the process to begin putting that into their culture. 

Stefan Thomke: The process of beginning putting that into their culture begins with an awareness that experimentation matters. That this is really important for the reasons that we just discussed. You know, the uncertainty. Once you're aware and you understand sort ...