A Guide to Common Testing Challenges

To survive, companies must commit to optimization and testing.

It won’t be easy. Being good at something never is.

A few weeks ago I gave a keynote at the Shop.org online merchandising workshop and had an opportunity to chat with several online retailers of all sizes. Each shared the challenges it faces as it tries to adopt a culture of optimization.

I talked with Raul Vazquez, CEO of Walmart.com. Unlike many retailers, Walmart.com has a culture of optimization and embraces a process for improvement. Even with its process in place, however, it doesn’t do any A/B or any formal testing.

An executive from another multibillion dollar online retailer told me an all-too common tale of how it began testing with the help of a vendor. After seeing no lift and investing all the resources, the retailer cut the vender loose and let testing fall by the wayside. Like so many others in these tight times, it axed testing as a budget item. Can anyone afford to stop optimizing?

The fact that optimization and testing challenges are rampant shouldn’t surprise anyone. In its “Guide to Retail and Web Site Design,” Internet Retailer’s research unwrapped the fact that over 76 percent of online retailers don’t test.

It’s not that online markets have a violent opposition to testing. It’s a combination of things. Many don’t know where to start; others may have started and had limited or no success. Here are some other common challenges that online retailers and online marketers face:

• Simple slice-and-replace testing/optimization. This is the process of Web teams slicing and replacing an element, a page, or a portion of the visitor experience and finding little or no lift in the numbers. So they move on and try another. This is a challenge because this method has no way of determining if the original hypothesis was incorrect (the reason they made the change) or if it was an execution issues.
• Resources. Testing well is hard work and uses resources: time, effort, and cash. And because teams are unsure, they resist or don’t give their best effort.
• Platforms. Some site platforms make it difficult to test. Ultimately, you must determine what costs the company less: lost opportunities, platform upgrades, or changes to allow for easier and more efficient testing.
• Lack of a process. Most retailers test randomly, just selecting something in the conversion funnel or their least favorite page. Others perform large-scale tests and may break or fix many things along the way. In both cases, they have no method for gleaning insight so they can successfully repeat the process.
• Persuasion or brand perception issues. Often time the issues that keep visitors from converting aren’t as simple as a button color, an element’s position, or a form improvement. We recently worked with a big-name financial institution that wasn’t committed to taking the time to test these subtle issues. Instead, they went with another vendor that tested more simplistic conversion issues. Needless to say those efforts didn’t move the needle.

Most of these challenges are related to companies not setting a solid (i.e., more scientific) testing process in place. None of these challenges is a reason to give in.

If you have testing challenges, you should know two things before you throw your hands up. First, you aren’t alone. Second, don’t stop testing — instead stop testing badly by adopting a cycle of optimizationand a smart process for doing so.

I will be launching (and signing) my new book, “Always Be Testing,” at Search Engine Strategies San Jose on August 18. The book is based on my 10-plus years of experience helping companies that struggle with testing and giving them practical guidance and tools to deal with these challenges. If you plan on attending, let me know. I’d love to meet up with you and talk about your challenges.

What’s your favorite marketing tool or service? Which one made your campaign a success? We want to know! Nominate your choice in the 2008 ClickZ Marketing Excellence Awards. Nominations are open until August 14. Nominate now!

Meet Bryan at SES San Jose, August 18-22 at San Jose Convention Center.