Wednesday, December 17, 2014

The Discipline of Business Experimentation
•  Stefan Thomke
•  Jim Manzi



A critical, critical article. Experimentation is the underpinning of commercializing  innovation.

Soon after Ron Johnson left Apple to become the CEO of J.C. Penney, in 2011, his team implemented a bold plan that eliminated coupons and clearance racks, filled stores with branded boutiques, and used technology to eliminate cashiers, cash registers, and checkout counters. Yet just 17 months after Johnson joined Penney, sales had plunged, losses had soared, and Johnson had lost his job. The retailer then did an about-face.How could Penney have gone so wrong? Didn’t it have tons of transaction data revealing customers’ tastes and preferences? 
Presumably it did, but the problem is that big data can provide clues only about the past behavior of customers—not about how they will react to bold changes. When it comes to innovation, then, most managers must operate in a world where they lack sufficient data to inform their decisions. Consequently, they often rely on their experience or intuition. But ideas that are truly innovative—that is, those that can reshape industries—typically go against the grain of executive experience and conventional wisdom. 
Managers can, however, discover whether a new product or business program will succeed by subjecting it to a rigorous test. Think of it this way: A pharmaceutical company would never introduce a drug without first conducting a round of experiments based on established scientific protocols. (In fact, the U.S. Food and Drug Administration requires extensive clinical trials.) Yet that’s essentially what many companies do when they roll out new business models and other novel concepts. Had J.C. Penney done thorough experiments on its CEO’s proposed changes, the company might have discovered that customers would probably reject them… 
…To obtain that kind of knowledge—and ensure that business experimentation is worth the expense and effort—companies need to ask themselves several crucial questions: Does the experiment have a clear purpose? Have stakeholders made a commitment to abide by the results? Is the experiment doable? How can we ensure reliable results? Have we gotten the most value out of the experiment? Although those questions seem obvious, many companies begin conducting tests without fully addressing them. 
Checklist for Running a Business Experiment 
Purpose Does the experiment focus on a specific management action under consideration? What do people hope to learn from the experiment? 
Buy-In What specific changes would be made on the basis of the results? How will the organization ensure that the results aren’t ignored? How does the experiment fit into the organization’s overall learning agenda and strategic priorities? 
Feasibility Does the experiment have a testable prediction? What is the required sample size? Note: The sample size will depend on the expected effect (for example, a 5% increase in sales). Can the organization feasibly conduct the experiment at the test locations for the required duration? 
Reliability (remember, we start out to prove our hypotheses wrong--the scientific approach)
What measures will be used to account for systemic bias, whether it’s conscious or unconscious? Do the characteristics of the control group match those of the test group? Can the experiment be conducted in either “blind” or “double-blind” fashion? Have any remaining biases been eliminated through statistical analyses or other techniques? Would others conducting the same test obtain similar results? 
Value Has the organization considered a targeted rollout—that is, one that takes into account a proposed initiative’s effect on different customers, markets, and segments—to concentrate investments in areas where the potential payback is highest? Has the organization implemented only the components of an initiative with the highest return on investment? Does the organization have a better understanding of what variables are causing what effects?