Read and interpret your experiment results
Check that your experiment computed correctly, that its groups are balanced, and why some users were left out.
Your goal
Trust, or catch a problem with, the groups an experiment produced, before you activate a campaign against them or draw a conclusion from comparing them.
Expected outcome
A clear read on whether the groups are balanced and large enough to measure on, or a diagnosed reason why they are not.
Find your experiment
The Experiments menu, located in the Measurement section, lets you view and manage all experiments created on your datamart. It displays a customizable table where the experiment name is shown by default. Clicking a name opens the experiment details.
Compute it, and follow its status
The experiment details page is displayed after you click Create Experiment. At that stage the experiment has not been computed and is marked Draft. To run it, click Compute.
Draft
The experiment has not yet been launched and can be edited.
In progress
The randomization is being computed. It can no longer be edited.
Uploading
The randomization has ended and the groups are being populated.
Completed
The groups are populated.
Error
Something went wrong. The experiment is not recorded and can be edited. Contact your Account Manager.
Read the results
The results of each experiment are detailed in the tabs below.
Overview
Groups details
Group size, both in number of UserPoint and in number of identifiers available, according to the identifiers selected in the creation form.
Representativeness details
A synthetic view of the randomization performed, used to validate group balance. Each criterion is analyzed through 2 graphs.
Strata details
Detailed results of the randomization: the strata created, and how each was distributed across the groups.
Unusable UserPoint
The UserPoint of the source segment that were not distributed to any group, and why.
Reading the representativeness graphs
The first graph, on the left, shows the distribution of UserPoint in your segment. It is for information only and says nothing about the randomization performed.
The second graph, on the right, shows how the segment's UserPoint were distributed between the groups. A histogram showing similar bars is proof that randomization went well and the groups are balanced.
Reading the strata details
If a stratum is too small to ensure balanced group distribution, it is excluded from the experiment entirely, and strata excluded for that reason are highlighted in red. The threshold is the acceptance threshold.
If a significant portion of UserPoint are eliminated because of small strata, it may indicate that:
the segment used for the experiment is too small, or
the experiment includes too many strata, which can result from using too many representativeness criteria, or criteria with too many distinct values.
Groups
This tab is dedicated to the segments created for the experiment. As many segments are created as there are groups configured in the creation form.
The name of each segment is established by default from the name of the experiment and a prefix of the form [group_type][test group number][size_expressed_as_a_percentage], with:
[group_type] = "Test" or "Control"
[test group number] = "1", "2", and so on
[size_expressed_as_a_percentage] = "90", "10", and so on
All segments created from experiments can be identified from the Segments menu using the Segment type = Experiment groups filter.
Why the groups hold fewer users than the source segment
It is very common, and completely normal, for a portion of the original segment's UserPoint to be excluded during distribution. The filters below cause it.
The UserPoint has no expected identifier
If you are planning an email campaign, for which you need an email address to send your message, but the segment you are working on is made up of UserPoint who do not have an email, then those have to be eliminated before computing the randomization, in order to guarantee both group size and balance. The identifiers concerned are the ones selected in the Addressability section.
The UserPoint is an outlier
Stratified randomization only. All UserPoint classified as outliers on at least one representativeness criterion are eliminated before the randomization is computed. Outliers are calculated via the Z-score and configured in Computing Console. See Outliers.
The UserPoint belongs to a stratum too small
Stratified randomization only. All UserPoint belonging to a stratum too small to guarantee group balance are eliminated before the randomization is computed. See Acceptance threshold.
Why a group size still moves afterwards
From a UserPoint point of view, the list assigned to a group by stratified randomization is set in stone at the time of calculation: the experiment does not redraw profiles.
You may still observe slight fluctuations at segment level, or during activation through your feeds, for the following reasons related to the identity engine.
Graph ID merges
Over time, different UserPoint may be merged if they are recognized as belonging to the same person. This leads to the technical deletion of obsolete UserPoint, some of which may belong to your experiment groups, causing the segment size to decrease slightly.
Identifier updates in feeds
The identifiers sent to your activation partners evolve at Graph ID level. Even though a UserPoint remains fixed within its group, the email addresses associated with it can be added, modified, or deleted over time. This is why you may see additions and deletions flowing through your feed.
Verify
Every group segment exists and, once the status is Completed, holds users.
For a stratified experiment, the two representativeness graphs per criterion show comparable bars across groups.
No stratum you expected to matter was silently excluded. Check the strata details if a criterion's balance looks off.
The delta between the segment size and the sum of the group sizes is explained by the filters above, and not by something unexpected.
Related pages
Last updated
Was this helpful?