For the complete documentation index, see llms.txt. This page is also available as Markdown.

Read and interpret your experiment results

Check that your experiment computed correctly, that its groups are balanced, and why some users were left out.

Your goal

Trust, or catch a problem with, the groups an experiment produced, before you activate a campaign against them or draw a conclusion from comparing them.

Expected outcome

A clear read on whether the groups are balanced and large enough to measure on, or a diagnosed reason why they are not.

Find your experiment

The Experiments menu, located in the Measurement section, lets you view and manage all experiments created on your datamart. It displays a customizable table where the experiment name is shown by default. Clicking a name opens the experiment details.

Compute it, and follow its status

The experiment details page is displayed after you click Create Experiment. At that stage the experiment has not been computed and is marked Draft. To run it, click Compute.

Status

Draft

The experiment has not yet been launched and can be edited.

In progress

The randomization is being computed. It can no longer be edited.

Uploading

The randomization has ended and the groups are being populated.

Completed

The groups are populated.

Error

Something went wrong. The experiment is not recorded and can be edited. Contact your Account Manager.

Groups, as segments, are created as soon as Compute is clicked. Although they are not populated at that stage, as long as the status is not Completed, feed configuration is already possible.

Read the results

The results of each experiment are detailed in the tabs below.

Overview

This tab is only available for experiments created using stratified randomization.

Section

Groups details

Group size, both in number of UserPoint and in number of identifiers available, according to the identifiers selected in the creation form.

Representativeness details

A synthetic view of the randomization performed, used to validate group balance. Each criterion is analyzed through 2 graphs.

Strata details

Detailed results of the randomization: the strata created, and how each was distributed across the groups.

Unusable UserPoint

The UserPoint of the source segment that were not distributed to any group, and why.

Reading the representativeness graphs

  • The first graph, on the left, shows the distribution of UserPoint in your segment. It is for information only and says nothing about the randomization performed.

  • The second graph, on the right, shows how the segment's UserPoint were distributed between the groups. A histogram showing similar bars is proof that randomization went well and the groups are balanced.

If you run an experiment with 3 representativeness criteria, this section displays 3 x 2 = 6 graphs.

Reading the strata details

If a stratum is too small to ensure balanced group distribution, it is excluded from the experiment entirely, and strata excluded for that reason are highlighted in red. The threshold is the acceptance threshold.

If a significant portion of UserPoint are eliminated because of small strata, it may indicate that:

  • the segment used for the experiment is too small, or

  • the experiment includes too many strata, which can result from using too many representativeness criteria, or criteria with too many distinct values.

Groups

This tab is dedicated to the segments created for the experiment. As many segments are created as there are groups configured in the creation form.

The name of each segment is established by default from the name of the experiment and a prefix of the form [group_type][test group number][size_expressed_as_a_percentage], with:

  • [group_type] = "Test" or "Control"

  • [test group number] = "1", "2", and so on

  • [size_expressed_as_a_percentage] = "90", "10", and so on

An experiment created with 2 test groups, each comprising 45% of the segment, and a control group comprising the remaining 10%, has its segments named as follows:

  • Test_1_45_experiment_name

  • Test_2_45_experiment_name

  • Control_10_experiment_name

All segments created from experiments can be identified from the Segments menu using the Segment type = Experiment groups filter.

Why the groups hold fewer users than the source segment

It is very common, and completely normal, for a portion of the original segment's UserPoint to be excluded during distribution. The filters below cause it.

Reason

The UserPoint has no expected identifier

If you are planning an email campaign, for which you need an email address to send your message, but the segment you are working on is made up of UserPoint who do not have an email, then those have to be eliminated before computing the randomization, in order to guarantee both group size and balance. The identifiers concerned are the ones selected in the Addressability section.

The UserPoint is an outlier

Stratified randomization only. All UserPoint classified as outliers on at least one representativeness criterion are eliminated before the randomization is computed. Outliers are calculated via the Z-score and configured in Computing Console. See Outliers.

The UserPoint belongs to a stratum too small

Stratified randomization only. All UserPoint belonging to a stratum too small to guarantee group balance are eliminated before the randomization is computed. See Acceptance threshold.

Why a group size still moves afterwards

From a UserPoint point of view, the list assigned to a group by stratified randomization is set in stone at the time of calculation: the experiment does not redraw profiles.

You may still observe slight fluctuations at segment level, or during activation through your feeds, for the following reasons related to the identity engine.

Phenomenon

Graph ID merges

Over time, different UserPoint may be merged if they are recognized as belonging to the same person. This leads to the technical deletion of obsolete UserPoint, some of which may belong to your experiment groups, causing the segment size to decrease slightly.

Identifier updates in feeds

The identifiers sent to your activation partners evolve at Graph ID level. Even though a UserPoint remains fixed within its group, the email addresses associated with it can be added, modified, or deleted over time. This is why you may see additions and deletions flowing through your feed.

Verify

  • Every group segment exists and, once the status is Completed, holds users.

  • For a stratified experiment, the two representativeness graphs per criterion show comparable bars across groups.

  • No stratum you expected to matter was silently excluded. Check the strata details if a criterion's balance looks off.

  • The delta between the segment size and the sum of the group sizes is explained by the filters above, and not by something unexpected.

Last updated

Was this helpful?