What Makes Design of Experiments (DOE) Better than A/B Testing?


Watch the full interview: http://www.isixsigma.com/tools-templates/design-of-experiments-doe/mark-kiemele-interview/

Michael: A lot of companies in the startup world were focused on A/B testing. There are a whole slew of service providers online, like Optimizely.com or VisualWebsiteOptimizer.com; even Google Analytics allows companies to do A/B testing.

So, if I have a signup page on iSixSigma.com, for example, I can test how many people come to the page and click and sign up with a green button versus an orange button on the side. So that is AB testing, or one factor at a time testing. I change one factor and I measure it. What makes DOE different from these other testing techniques like one factor at a time, or educated guesses, or AB testing?

Mark: That is a great question, Mike, because that is the heart and soul of DOE. Why would I want to use a DOE when I was taught, in high school, to do one factor at a time testing? When my chemistry teacher says, “When you want to see the effect of temperature on this experiment, you got to hold everything else the same and you just change that one thing, and then you will see what effect it has on your yield or whatever you are measuring for a response or an output.” How misinformed are we, because we are teaching people at the very young stages of their life to do this AB, or one factor at a time, testing? There is different criteria or, what I should say, attributes of DOE, like replication, randomization, which we will talk about a little bit later when we get into the examples, but the one characteristic of DOE that distinguishes it from everything else is the concept of orthogonality. Orthogonality allows, and that is how we distinguish DOE. A DOE is a testing technique that has an orthogonal design or nearly orthogonal design. Now, what orthogonality buys is the ability to evaluate these factors, their effects, and the interactions independently from one another. That keyword is independent evaluation. And what does independent evaluation buy us? Well, independent evaluation buys us the ability to get to root cause. That is cause and effect relationships. And cause and effect relationships — root cause analysis today is tough stuff, Mike. People make light of it. It is hard because of those doggone combination effects.

Michael: Right.

Mark: Those interaction effects. So, if you have orthogonal designs, orthogonality buys you the ability to evaluate the effects independently. Independence allows you to buy the ability to get to root cause analysis. And of course, if you can find root causes of problems, guess what? Your decision-making is enhanced and, of course, the financial aspect will also be enhanced. So that is the orthogonality aspect.

Michael: Okay. And so, that all makes sense to me, Mark. I think most of the people that watch this show are educated like you and I are, and they can understand that you are saying orthogonality is this concept that allows you to evaluate the factors independently. So you do not see the bias. You do not have results that are confounded; is another word that means the results you are seeing are actually affected by something else that you are not anticipating. So that makes sense, but then I just go back to my real world example, where I have got a signup page on iSixSigma.com, and the output variable from what you were telling me earlier was I want people to actually sign up. Maybe it is put in an email address and click the signup button. Right? And so, AB testing would tell me, or my fifth grade science teacher would tell me, “Start off with an orange button. Change it to a green button. See how much that affects it. Then go from whichever one is better. That is now your base. Now change it from a green button with a sans serif font to a serif font, and so it feels more like a typewriter. Maybe that will affect people.” Every industry and every group of people is different, and you might say that the fortune one hundred, which come to visit iSixSigma.com to learn about concepts like DOE would be more attuned to a font that looks like a typewriter, let’s say, and so they are going to get a higher click-through rate on that. Will DOE allow me to say: “My output is signups on this page; now I want to look at all the factors that are involved,” and just solve it once, so I am not doing AB testing for the rest of my life?

Mark: Absolutely, Mike. That is the beauty and power of DOE. You can test multiple things simultaneously and still evaluate the effects independently. Orthogonality means balance in the design. That is — get back to your font and you color thing. You now home in on their orange or green, and then you start changing the fonts. There is a way to test those simultaneously so that you have balance in the design. And that balance will allow you to get that independent evaluation. We will talk a little bit about that in one of the examples that we do.

You May Also Like