Published:
The key to powerful growth hacking is to establish the discipline of A/B testing your efforts. An A/B test is really just testing one variation against the current option. You might A/B test an advertisement where you try a different call to action, say buy now versus buy today. Or you might try a new element to the user interface, in hopes of increasing the time spent on site. Every A/B test should have a clearly defined goal. What do you want to see happen, and what data do you need to measure to understand if the idea worked? With A/B testing, your ideas can be tested in real time.
Using some relatively simple tools, you can drive a fraction of your users to a different version of the page they were trying to access. There, you can measure their behavior and compare it to the existing solution. What's great about A/B testing is that you don't need to have equal groups of people. Instead, you need just enough volume that your results are statistically significant. This basically means the results you're seeing are unlikely to have happened by chance. To show you what I mean, let's look at an example of the results from a landing page test.
In Test A, we had 10,000 visitors and 50 conversions. This equals a .5% conversion rate. In Test B, we had 25,000 visitors and 150 conversions. This equals a .6% conversion rate. At a glance, you'd be inclined to call Test B a winner. For a marginal traffic increase, it looks like the conversions are higher, and the conversion rate is better. But we can use a statistical significance calculator to determine the truth behind this data. So here's a look at the output.
We're only 88% confident that Test B performs better than Test A. So we would need to let the test run longer, until we had enough data to determine it was really the winner. You're looking for a confidence value in the neighborhood of 95%. Now take a look at this same set of data, but this time we'll assume Test A had 80 conversions, and Test B still had 150. It's hard to tell at a glance. Test A has a higher conversion rate, but Test B has more conversions. When we plug the data into that calculator, we find that Test A converted 33% better than Test B, and our results have a 98% confidence rating.
Therefore, we would call Test A the winner. It's important to build A/B testing into your growth-hacking efforts. You can run A/B tests on your own, but if you need a little extra structure, take a look at the resources provided by Optimizely and Visual Website Optimizer. All A/B tests should start with a great hypothesis, and remember, a failed test isn't a bad thing. Learn from it and explore other alternatives.
Data will Paint the Picture
Confidence on result after longer run of Test will drive the decision
To be successful, having a high-level overview of your data isn't enough. You have to be willing to study the data at each point in the customer's journey. It's more than time on site and and total monthly visits. It's a quest to understand why they're doing what they're doing. The data will paint a picture, and it might not be obvious at first, but when you become extremely familiar with your data, it'll be a lot clearer. In fact, we should really start by determining how familiar you are with your data. Off the top of your head, can you indicate the most popular section of your website? How about the number one referral source for each of your user personas? What's the most common technology that your site is accessed with? Or on average, what's the most popular time for people to access your site? If you've got these nailed, that's great, but it doesn't mean you can stop digging.
Always push for more, because an important habit to form as a growth hacker is pursuing data relentlessly. Start every morning by looking over your metrics, familiarizing yourself with the information, exploring areas that are new and foreign, and trying to correlate things that might be seemingly unrelated. Look at everything from your social media metrics to your customer service email volumes. It all counts for something. And once you think you've learned enough, dig deeper. It's an incredibly valuable skill to hone and it separates good growth hackers from great growth hackers.
Over time, you'll learn where to spend your time, but don't worry about being terribly efficient as you learn. It's going to take a lot of practice and persistence. And if you're not naturally drawn to long spreadsheets full of numbers, it'll feel a little mundane. And I'm assuming you're running a lean operation here, which means you won't have access to a data sciences team or enterprise-level data mining software. This means you're the front lines for this information, all the more reason to spend your time researching. I'll share an example from Etsy to help solidify this topic.
I saw a presentation by Dan McKinley, a former engineer at Etsy.com. If you're unfamiliar with Etsy, they're a marketplace for handmade or vintage items. Members can build their own storefronts and sell products through their platform. Now they knew their users had a problem, the data said so. Users wanted more results per page, and they wanted them faster. So Etsy jumped in and spent five months developing infinite scrolling. The idea was that as you scrolled down the page, you'd get new results and never reach the bottom. They rolled it out and kept a close eye on the data.
They found that people were buying fewer things. The results were negative. They had to go back to how things were before, and that was an important learning experience. They heard the problem users were having, and they implemented a fix. If they didn't monitor their data diligently, they would've never realized the results were terrible. And they knew when they rolled back that they were still going to have the problem of not showing enough results, and it would be back to the drawing board as how to solve it again. Once you're really familiar with your data, start running tests. Try sending people through your funnel.
Give people unfamiliar with your site a specific goal. That might be to buy a specific product, or subscribe and cancel a free trial. Observe what happens. If you'd like to test this at scale, check out usertesting.com. Next, run the same test with your competitor or a product in a similar niche. You can even use usertesting.com for scale yet again. Finally, take the qualitative data and start mapping it to the quantitative. Do the issues outlined make sense based on what you're seeing? Do you have a problem that is larger than just one or two users complaining? If you notice that you have high abandonment after a user looks at a certain product, then you've got a starting point to analyze.
There's an endless amount of data you can interact with. The moral is to not trust your gut. Avoid leaning on instinct, and instead, let the data do the talking.
Handling failed experiments
It's inevitable. At some point an idea, a product feature, or even a growth strategy will fail. How you handle that failure is really important. It's really easy to sweep a failure under the rug, chock it up to series of bad decisions, and move on. And while moving on is great, if you don't take time to analyze the failure, you'll never truly learn from it. And as a growth hacker, learning from the data is a top priority. There's really no excuse not to learn. Data is so accessible now that every failure should get a thorough debrief.
Find the root of the problem even if you're working solo. You still need to walk through everything from start to finish. Not only do you learn from your mistakes but you'll also get better at recognizing the warning signs before failure happens. To get you thinking about how to approach failure, let's look at some common reasons things fail. One that I see often is that the data was misinterpreted. When analyzing a set of data, survey results, or even sales information, mistakes happen. Growth hacking is all about executing based on the data.
The data is the road map. And if you've got a bad set of numbers or an incorrect correlation, your map will send you in the wrong direction. This is often preventable by double-checking the data, reevaluating the collection technique, or checking for outliers that may have skewed everything. Another common point of failure is misunderstanding the customer. As marketers, we like to think we're pretty good at understanding our customer. And we probably do have a strong grasp of who they are and what they want. But it's really easy to introduce bias when we come up with ideas.
It's so easy to take something we want personally and project that need onto our customer. We'll make it something they need. And maybe it's something our customer actually asked for, but we didn't fully think about how it impacts everything collectively. More often than not, customers aren't really thinking about the big picture, either. They have a specific need, and it's your job to take a step back and explore how important it really is. Next, we have failure due to subpar execution. Execution is incredibly important.
The best ideas can only get so far. And if things break down in the execution, the idea will fail. Consider reviewing all the steps of how the project went. Where were the roadblocks? What in the execution failed? And how will you resolve that in the future? Failures in this area often identify areas that we need to grow either personally or as a team. Build into those weaknesses to prevent this hurdle from popping up in the future. Along with the theme of executing on a task is the issue related to time.
A lot of projects fail because there wasn't enough time. It's easy in hindsight to see the warning signs to biting off more than you or your team can chew, but when I map out ideas, I always double the time estimates provided to me. I'd rather be early than late. When deadlines are critical, be prepared to shift priorities and cut features to make it work. Another point of failure is the unwillingness to see things through. If you've done the research and you trust your data, don't pull the plug early. A lot of great projects get shut down just shy of their tipping point.
Stick to the plan. If you do fail, do it after the original time estimate has past. This way, you'll have a better understanding of what went wrong and you won't be left with the, "What, if ..." factor. And, finally, a huge contributing factor to failure is having the wrong attitude. It's okay to be skeptical about something. You can pursue an idea with positive skepticism. It's when you're convinced something is going to fail or an idea is terrible without fully vetting it out, that you run into a problem. If you're concerned an attitude will derail a concept then move to another idea until you've gathered enough support to try it out.
If you can take one thing away from all of this, it's that you're going to fail. So fail proudly, fail confidently, and fail often. And when you do fail, take time to understand why.
added on 03-Feb-2018