It is important to think about some crucial points of your business to quickly validate hypotheses:
Goals
Understanding your company's challenges and defining your company's priorities is crucial when deciding which experiments to carry out. By looking at business goals, hypotheses are raised with the aim of solving specific problems .
Metrics
Defining which metrics you will use and knowing how to differentiate finland phone number resource between them is very important. Primary metrics are geared towards your business, while secondary metrics are related to a specific experiment. Both represent a diagnosis of what is happening at a given moment, and one can influence the other.
Creative
The tip here is to always organize the available resources . In some cases, many creatives may be ready when you define what you are going to test, but in others they may not. Defining how much time and budget will be needed to produce the creatives is essential so that the testing strategy does not deviate from the overall business strategy.
Analysis
It is important to look at the results of your tests analytically, especially during the planning phase of your company. Among the different approaches to analyzing the results of A/B tests, frequentist and Bayesian are the most widely used in the market.
Frequentist analysis was widely used in science in the 20th century, but it has some limitations when the focus is on A/B testing, such as the need to define the duration of the experiment in advance and the impossibility of checking results before the expected period. The Bayesian approach , used by the CRO platform Croct , presents more detailed metrics, better statistical significance and greater speed for analyzing results, being the most used by those who understand the subject.
Structuring the A/B test
Defining a segment of your audience is the first step to structuring your A/B test. Experimenting across all of your website traffic can compromise your results, as we often hypothesize that a change in creative might have a big impact, but we can never be sure of that until we test it.
Let's say you randomly select 25% of your traffic to participate in the test. In this segment, the control group is characterized by those who do not see any variation, while group A represents those who will see a certain variation of the content being tested, and group B represents those who will see the second variation of that same content. When a user from this segment visits the site, he or she will be randomly assigned to one of these two groups.
There are several types of A/B testing, and for each of them the initial division of the audience is carried out in a different way. The different types are:
Common A/B Tests
The selected portion of traffic is divided randomly, without distinction between segments. In this model, only one hypothesis is validated at a time.
Segmented A/B Testing
The test is performed for a specific segment of the website's traffic. For example, sometimes we realize the need to validate a hypothesis only in relation to users who live in the city of São Paulo. We then select only this segment, and within it we allocate the control group and the variants. The result will determine the conversion rate for the variation that this segment is accessing.
Parallel A/B testing
In this case, two tests occur at the same time with different hypotheses. Parallel or simultaneous tests allow the validation of hypotheses to be obtained quickly. For example: one of the A/B tests will be targeted at residents of the city of Rio de Janeiro, while another, simultaneously, will be targeted at returning users.
Tip: There may be cases where simultaneous tests include users who fit both hypotheses. In this situation, it is important to create a rule that determines where this user will be allocated, thus preventing him or her from participating in two tests and the result from being distorted.