For the complete documentation index, see llms.txt. This page is also available as Markdown.

Create your test and control groups

Configure the groups, randomisation, and addressability of an experiment, and the datamart-level settings behind the form.

Your goal

Turn a segment into a set of test and control groups that are large enough, addressable enough, and, if the campaign outcome depends on it, balanced on the criteria that matter.

Expected outcome

An experiment in Draft status, ready to compute, with its groups configured the way your measurement needs them.

Configure the experiment

From the segment page, click More, then Create Experiment. The form is divided into the sections below.

1

General information

Name the experiment. The name helps you identify it in the experiment table, and is used by default in the name of the groups generated. A description is optional, and lets you comment on the experiment for yourself or your colleagues.

2

Groups

Specify the number and size of the groups. An experiment may have between 1 and 10 test groups, and may or may not have a control group. It must consist of at least two groups, test and/or control.

If a permanent control group is set up on your datamart and you wish to use it, the size of your control group is set by default with no possibility of modifying it. All UserPoint in your segment belonging to the permanent control group constitute the control group for this experiment.

3

Randomization

Decide between the types of randomization described below.

4

Addressability

Define the identifiers that will be used for activation, which in practice means selecting the identifiers you will later configure in your feeds. The aim is to retain only those UserPoint that have the required identifiers before randomization starts.

If several identifiers are selected, only UserPoint with all of them are retained.

Whatever identifiers are selected, randomization is calculated at the UserPoint level.

When all sections are complete, click Create experiment. You are redirected to the experiment details page, and the experiment is listed in the Experiments menu.

Randomization types

Type

Stratified randomization

Use this when you want to ensure a balance of groups on certain key criteria that may influence your campaign outcome. It involves configuring at least one representativeness criterion.

Simple randomization

A random draw using partitions. Each group is randomly assigned a number of partitions equivalent to its size, expressed as a percentage, configured in the previous section.

Which one to choose

The choice depends on your knowledge of your audience, data availability, and the level of rigor required for post-campaign analysis.

Stratified randomization is highly recommended when you know that specific criteria, such as age or purchase history, will heavily influence your objective. Besides guaranteeing balanced groups on those criteria, it significantly simplifies the post-campaign analysis phase: you only need to correct the exposure bias, meaning the fact that a portion of the test group did not actually see the ad. Unlike simple randomization, you do not also have to correct initial imbalances that occurred by chance. Stratification increases your statistical power: by eliminating random variation from the start, the test is much more capable of detecting the campaign's true impact.

Simple randomization is a basic random draw. It was historically chosen because stratified randomization was too complex to set up without a dedicated tool, but mediarithmics now makes both equally easy to configure. Opt for simple randomization primarily if:

  • You do not know which specific criteria strongly influence the objective you are trying to achieve.

  • You lack the necessary data to use as representativeness criteria.

  • The available data would require extensive pre-processing, for example through computed fields, making the setup too long or complex for your current project constraints.

Datamart-level configuration

The Experiments feature is configured through the Experiments menu in Computing Console. Configurations are done at datamart level, and they shape what every experiment creation form offers.

Default groups

Define the group configuration that appears by default in the experiment creation form. You are prompted to enter the most common configuration for your datamart.

Control group minimum size

Configure a minimum expected size for the control group, expressed in number of UserPoint, below which an alert appears in the form inviting the user to modify the configuration.

Whether the alert appears depends on the estimated size of the control group, calculated as: segment size in number of UserPoint, multiplied by the control group percentage entered in the form.

The aim is to prevent users from creating control groups that are too small for the incremental measurement to succeed. The alert is not blocking: the user can still create the experiment despite it.

Permanent control group

This option sets up a control group at datamart level. It is useful for limiting noise if you want to run several experiments in parallel, or if you want to assess the performance of a channel over an extended period of time.

When enabled, it adds a third option to the experiment creation form, so the choice becomes: no control group, customizable control group, or permanent control group. Used in an experiment, it defines the control group automatically from the UserPoint belonging to the permanent control group.

When you use the permanent control group in an experiment, the size of the control group is not modifiable, and approximates the size of the permanent control group configured on the datamart.

You can renew the permanent control group on demand with the dedicated button. The new one excludes by default the partitions used in the previous one.

A standard control group is calculated on the fly for one given experiment. The permanent control group is configured globally at datamart level, which is what makes it useful for evaluating a channel over the long term, or for running several A/B tests in parallel while limiting noise and overexposure.

Stratified randomization settings

Representativeness criteria

A representativeness criterion is data that can be used to balance groups in stratified randomization. To be usable, the data must come from the schema and carry a value at the UserPoint level. Criteria are used for balancing because they are supposed to influence the expected result of the campaign.

You are running a campaign with the objective of generating sales of a given product. You know that age, gender and purchase history on the category are 3 data strongly correlated to the probability of purchase, and you want the groups in your experiment balanced on those 3 dimensions. You therefore set up the following 3 data as representativeness criteria:

  • age

  • gender

  • sales on the category over the last 12 months

These rules govern what can be used:

A criterion must contain a limited number of distinct values. The default limit is 20.

A birthdate field cannot be used as it is. Prefer an age range field that returns values such as "18-24 yo", "25-34 yo", and so on. If a criterion does not exist in your schema, create it as a computed field, documented in the Setup and Monitor section under Collect and manage your data > Design your data model and schema > Computed fields.

Numeric fields containing more than 10 distinct values are automatically reprocessed to show only 3 values: low, medium, high. This is achieved by splitting the segment into 3 thirds.

Above 20 values, the criterion details are no longer displayed in the experiment overview.

How strata are defined

Strata are defined automatically by cross-referencing the different values of your representativeness criteria.

You configure two criteria: "Gender", with 2 values (Male, Female), and "Age Category", with 3 values (18-25, 26-40, 40+). The algorithm creates 6 distinct strata: Male 18-25, Female 18-25, Male 26-40, Female 26-40, Male 40+, Female 40+.

The platform then distributes the UserPoint of each stratum proportionally across your different test and control groups, to ensure balance.

Templates

A template is a set of representativeness criteria. Create templates when you tend to use the same criteria across your experiments. They are designed to:

  • guide non-expert users who may have difficulty defining the representativeness criteria to use

  • speed up the creation of your experiments

Acceptance threshold

The acceptance threshold defines the cases where a stratum that is too small is excluded from an experiment. Its default value is 1. If the distribution of all UserPoint in a given stratum deviates by more than 1 point from the distribution defined in the Groups section of the creation form, the stratum is excluded.

You created an experiment with 2 groups on a 50/50 distribution, and you left the acceptance threshold at its default value of 1.

A stratum containing 200 UserPoint splits into 100 and 100, which is exactly 50/50. It deviates by 0 points from your distribution, so it is kept.

A stratum containing 25 UserPoint cannot be split evenly. The closest it gets is 13 and 12, which is 52/48, so each group deviates by 2 points. That is more than the threshold, and the stratum is eliminated. Were it kept, its UserPoint would be over-represented in the test group and under-represented in the control group compared with the strata where the balance could be respected. A stratum of 5 UserPoint is eliminated for the same reason, by a wider margin: its closest split is 3 and 2, which is 60/40, or 10 points off.

Raise the acceptance threshold to 2 and the 25 UserPoint stratum is kept, because a deviation of 2 points is no longer more than the threshold. The 5 UserPoint stratum is still eliminated, because 10 points is.

Outliers

Defining outliers lets you exclude from your experiments the UserPoint that exhibit exotic behavior which, analyzed alongside other UserPoint, could distort results. A company placing an order for its 200 employees all at once is the classic case.

Outliers are defined only for numerical values, analyzed using the Z-score. The Z-score is the number of standard deviations by which the value of a UserPoint score is above or below the mean value of what is being observed:

z=x−μσ{\displaystyle z={x-\mu \over \sigma }}

where:

  • x is the score of a given UserPoint,

  • μ is the mean of the population,

  • σ is the standard deviation of the population.

You use sales over 12 months as a representativeness criterion, and you set outliers to be excluded as soon as their Z-score is equal to or greater than 5. Across your segment the mean is 100 and the standard deviation is also 100, because a handful of very large buyers stretch the distribution.

That threshold puts the cutoff at 100 + (5 x 100), so at 600.

A UserPoint with sales of 500 has a Z-score of (500 - 100) / 100 = 4, below the threshold, so it stays in the experiment. A UserPoint with sales of 1200 has a Z-score of (1200 - 100) / 100 = 11, far above it. That UserPoint is excluded, as is every UserPoint above 600.

Verify

  • The experiment is in Draft status after creation, and its groups are not yet populated.

  • Clicking Compute moves it through In progress and Uploading to Completed. See Read and interpret your experiment results.

  • The number of strata is consistent with the size of your segment. Many criteria, or criteria with many distinct values, multiply the strata and shrink each one.

The segment type this experiment produces is documented in Experiment groups segment. How the groups are named is on Read and interpret your experiment results. Partitions, which simple randomization draws from, are on Partition segment.

Last updated

Was this helpful?