Why Is Website Test Frequency Important?

What Do We Mean by Test Frequency?

It’s pretty straightforward – test frequency is how often you conduct tests.

It could be once a week, once a month, or even once a year. However, the latter won’t do much to inform your CRO strategy!

At FABRIC8 we think having a high test frequency is important. This is probably because we take the ‘test and learn’ approach seriously. After all, the more you test, the more you learn, and the more actions you can take to improve user experience.

CRO testing is an ongoing process. The initial optimisation phase requires a significant amount of testing activity, but you need to keep it going for the life of your website if you’re to keep up with the changes in your business. These changes could be:

• Shifting business goals
• Increased volume of content
• New channels for bringing in leads
• Change in target audience
• General Trends – One day your audience might love hot-pink buttons, the next they may prefer gothic black. Well, that’s a little unlikely, but you get our drift – audience preferences change!

So, you should keep up with your testing to stay on top of your CRO. But what’s the process for this? Well, here’s a quick rundown:

The Format of a Standard CRO Test

First off, you have an idea. It might derive from data, personal insight or a gut feeling, but that idea will form the basis of the test.

Next, you create a hypothesis based on that idea. For example, ‘I believe that click-throughs will increase by 10% if we use a button rather than a link.’ It’s best to be as precise as possible here, so you can accurately measure your results.

The third stage is setting up your tracking methods. You can do this manually through Google Analytics, or other user-analytics software like Mixpanel, Kissmetrics and Amplitude that will do the work for you. Some of these even facilitate testing within their platform, so you can keep everything in one place.

Then, all you need to do is press ‘Go!’ and watch the results roll in. Afterwards, you analyse the results, see if the change was effective and decide whether to implement it. Simple!

From this style of testing, you learn two things (aka insights):

1. if your hypothesis was right
2. how much of a difference it makes

If the results lead to another test being carried out the next week, that means you gain four insights overall. And if things keep going in that way, the number of insights you have will just grow and grow.

So, after only three months you might have as much as a whopping 26 insights – go team!

Why Is This Important?

Well, the more insights you have, the more decisions you can make regarding your website. This means you can take more action to improve user experience and conversion rate, which ultimately results in more revenue. Sounds great, right?

Best of all, these insights can help you stay ahead of the competition. However, that’s only if they’re carrying out fewer tests than you …

In our example, we performed one test a week and gained a decent amount of insight after a three-month cycle. But, what if your competitor is carrying out two tests a week?

They'll know twice as much as you, so will be the ones with the competitive edge – eek, not good!

And what if they’re carrying out even more in-depth tests by tracking the impact of their tests on other activity, such as:

• click through
• drop-off rate
• number of conversions

Well, this means your competitor will have six insights for every test! But if this includes tracking acquisition paths and audience segments, you might need to whack that figure up to 120 insights per test!

That’s a whole lot more insights than you have … but, in reality, how valuable are these?

What Do You Need To Be Aware Of?

Trying to collect as much data/insight as possible is important, however it comes with its own problems.

If you have a high test frequency, you might find yourself overwhelmed by the sheer number of insights coming your way. Think about it, if you have 120 insights per week, how do you know where to start? Unless you have a handy AI platform that can do the changes for you, you’ll burn out pretty quickly.

It also introduces a common problem with multivariate testing, in that you’ll be unable to attribute the element that affected the KPI, as too many things are going on at once. If you have too high a test frequency, the results can be very hard to interpret.

On top of this, you need to consider that maintaining this test frequency might impact the quality of the data you collect. You may be unable to make decisions based on your tests, because you haven't allocated sufficient time to collect meaningful results.

The length of time you should allocate to a test is defined by traffic, business size and other elements, but two rules of thumb are:

• An average A/B test should run for approximately 14 days.
• It should reach a statistical significance of at least 95%.

If your test doesn’t meet these criteria, keep it running a bit longer.

Another common issue with testing is that many marketers do it just for the sake of it. They often ignore traffic and blame a page for low conversion rates/KPIs.

After all, if you send the wrong traffic to your site, a perfect page won’t make the slightest difference. You need to work hard at targeting the right visitors who will engage with your page and at coming up with a realistic, valuable hypothesis that will make a real difference to your site.

What’s The Best Approach to Testing?

If you’re aiming for a higher test frequency, your plan needs to be structured and well thought-out.

You must understand CRO, the testing process, your website visitors and how to implement changes on your site. If you don’t, you should probably enlist the help of a CRO agency to complete the tests and inform you of the findings.

You must also recognise that each test is part of an overall strategy. Conducting a test in isolation may work in the short term, but may actually have a negative effect in the long term. All your tests should work together to build a cohesive customer journey, so consider this in your planning.

To plan, it helps to have a clear goal at the outset of your testing cycle and at the start of each test. Otherwise, the waters may get a little muddy and you could end up testing something else entirely!

A data-driven approach to testing is always best. Collect quantitative CRO data from software and tools (e.g. conversion funnels and heatmaps) and qualitative data, from surveys and face-to-face interviews.

Finally, your approach needs to be ongoing and frequent, but measured and controlled. You need to be in charge of your data, not the other way around!

Put a system in place, and build up to a higher test frequency to ensure you and your site can cope with the changes implement. That way you’ll gain increasing insight into your customers, and become a true competitor in your online marketplace.