A researcher wanted to investigate how frogs jump. He invented an experiment to determine the contribution of each leg. He needed a baseline, so he put the frog down and said “Jump frog jump!” Measuring the distance, he wrote in his journal “Frogs with 4 legs jump 3 feet.” Cutting off the right front leg, he put the frog down and said “Jump frog jump.” Measuring the distance he wrote in his journal “Right front leg contributes .5 feet to jump.” Cutting of the left front leg, he put the frog down and said “Jump frog jump!” Measuring the distance, he wrote. ” Left leg contributes .8 feet to jump.” Cutting of the right rear leg, he put the frog down and said “Jump frog jump.” Measuring the distance he wrote in his journal “Right rear leg contributes 1.7 feet to jump.” Cutting off the left rear leg, he put the frog down and said “Jump frog jump!” —- “Jump frog jump!” —- “Jump frog jump!” He wrote in his journal “Frogs with no legs are deaf”
We are all aware that it is easier to make good decisions when you have good data to back them up. Designing the affordable experiments to generate good data is where the difficulty starts to appear. This is especially true in a start up, where a lack of accumulated cash limits the number of pivots that can be done to recover from bad decisions. Even if you invent a timely affordable A/B test to guage your customer’s UX [User Experience], as this joke illustrates, many times convenient conclusions are applied to avoid the work that is actually required to provide what the customer is willing to buy.
A good example is a CAD (computer aided drafting) program. Back in the 90’s we still used drafting boards. By the 90’s this was a money issue related to the user experience of the CAD draftsman. The time, effort and cost of creating CAD drawings could not be recovered by the benefits. Since the downstream benefits of CAD drawings are fairly static, it all boiled down to adjusting the program so that making a drawing on the computer was quicker than using a drawing board. As is true during disruptive time, there were many players who saw the future benefits and wished to own the market. AutoCAD succeeded and displaced most other early players by incorporating effective feedback from thousands of CAD users. A/B testing established the CAD 2D drawing interface that has become dominate.
Ah, but the season changes. AutoCAD 2000 finally had all of the features that a large pool of their users ever wanted. This is not good news for a company with a business model based on selling a new version of their software every three years. Yes, there is other market segments that can take advantage of CAD drafting. Unfortunately cluttering the desktop with additional command icons alienated the existing user base. The A/B testing that enabled AutoCAD to achieve dominance, could not uncover a new look that would entice existing users to buy a new version… A good example of trying to extrapolate a convenient conclusion onto real test data. The actual conclusion is that it is necessary to allow the user to customize his desktop, so that it is possible to retain the look and function and minimized keystrokes of the earlier versions. Only then is the added functionality of new software versions interesting to current users. A/B testing will not invent a desktop look that is acceptable to everyone.
The same process applies to my www.mileagetrakker.com device. Our benchmarking against the competition indicated that UX is the battleground. Most business travelers put 12,000 business miles on a vehicle in a year. This deduction puts $2000 in their pocket provided that they have the correct tax records in the required format. Having spent a large part of Christmas vacation inputting the required mileage logs for many years in a row, my wife and I set out to find a better way. In start-up fashion we did A/B testing during the beta phase to discover whether the user wished to interface using a cell phone, texting, e-mail or a web site. I suspect that Mileage Trakker would have died if we simply taken the results to infer that it was possible to only offer the most popular method. In this mass customization era, the customers expect that they can choose the option that is best for them. Yes it does take more work on our part.
Mass customization is a paradigm shift. The reduction in computing cost has made it possible to offer your customer base some degree of choice. This reality on the ground will filter back into all of the activity and decision making methods within a business. As this example shows, it is important not to limit your chance for success with myopic vision