Create your test and control groups
Configure the groups, randomisation, and addressability of an experiment, and the datamart-level settings behind the form.
Your goal
Turn a segment into a set of test and control groups that are large enough, addressable enough, and, if the campaign outcome depends on it, balanced on the criteria that matter.
Expected outcome
An experiment in Draft status, ready to compute, with its groups configured the way your measurement needs them.
Configure the experiment
From the segment page, click More, then Create Experiment. The form is divided into the sections below.
Groups
Specify the number and size of the groups. An experiment may have between 1 and 10 test groups, and may or may not have a control group. It must consist of at least two groups, test and/or control.
If a permanent control group is set up on your datamart and you wish to use it, the size of your control group is set by default with no possibility of modifying it. All UserPoint in your segment belonging to the permanent control group constitute the control group for this experiment.
When all sections are complete, click Create experiment. You are redirected to the experiment details page, and the experiment is listed in the Experiments menu.
Randomization types
Stratified randomization
Use this when you want to ensure a balance of groups on certain key criteria that may influence your campaign outcome. It involves configuring at least one representativeness criterion.
Simple randomization
A random draw using partitions. Each group is randomly assigned a number of partitions equivalent to its size, expressed as a percentage, configured in the previous section.
Which one to choose
The choice depends on your knowledge of your audience, data availability, and the level of rigor required for post-campaign analysis.
Stratified randomization is highly recommended when you know that specific criteria, such as age or purchase history, will heavily influence your objective. Besides guaranteeing balanced groups on those criteria, it significantly simplifies the post-campaign analysis phase: you only need to correct the exposure bias, meaning the fact that a portion of the test group did not actually see the ad. Unlike simple randomization, you do not also have to correct initial imbalances that occurred by chance. Stratification increases your statistical power: by eliminating random variation from the start, the test is much more capable of detecting the campaign's true impact.
Simple randomization is a basic random draw. It was historically chosen because stratified randomization was too complex to set up without a dedicated tool, but mediarithmics now makes both equally easy to configure. Opt for simple randomization primarily if:
You do not know which specific criteria strongly influence the objective you are trying to achieve.
You lack the necessary data to use as representativeness criteria.
The available data would require extensive pre-processing, for example through computed fields, making the setup too long or complex for your current project constraints.
Datamart-level configuration
The Experiments feature is configured through the Experiments menu in Computing Console. Configurations are done at datamart level, and they shape what every experiment creation form offers.
Default groups
Define the group configuration that appears by default in the experiment creation form. You are prompted to enter the most common configuration for your datamart.
Control group minimum size
Configure a minimum expected size for the control group, expressed in number of UserPoint, below which an alert appears in the form inviting the user to modify the configuration.
The aim is to prevent users from creating control groups that are too small for the incremental measurement to succeed. The alert is not blocking: the user can still create the experiment despite it.
Permanent control group
This option sets up a control group at datamart level. It is useful for limiting noise if you want to run several experiments in parallel, or if you want to assess the performance of a channel over an extended period of time.
When enabled, it adds a third option to the experiment creation form, so the choice becomes: no control group, customizable control group, or permanent control group. Used in an experiment, it defines the control group automatically from the UserPoint belonging to the permanent control group.
You can renew the permanent control group on demand with the dedicated button. The new one excludes by default the partitions used in the previous one.
Stratified randomization settings
Representativeness criteria
A representativeness criterion is data that can be used to balance groups in stratified randomization. To be usable, the data must come from the schema and carry a value at the UserPoint level. Criteria are used for balancing because they are supposed to influence the expected result of the campaign.
These rules govern what can be used:
Above 20 values, the criterion details are no longer displayed in the experiment overview.
How strata are defined
Strata are defined automatically by cross-referencing the different values of your representativeness criteria.
Templates
A template is a set of representativeness criteria. Create templates when you tend to use the same criteria across your experiments. They are designed to:
guide non-expert users who may have difficulty defining the representativeness criteria to use
speed up the creation of your experiments
Acceptance threshold
The acceptance threshold defines the cases where a stratum that is too small is excluded from an experiment. Its default value is 1. If the distribution of all UserPoint in a given stratum deviates by more than 1 point from the distribution defined in the Groups section of the creation form, the stratum is excluded.
Outliers
Defining outliers lets you exclude from your experiments the UserPoint that exhibit exotic behavior which, analyzed alongside other UserPoint, could distort results. A company placing an order for its 200 employees all at once is the classic case.
Outliers are defined only for numerical values, analyzed using the Z-score. The Z-score is the number of standard deviations by which the value of a UserPoint score is above or below the mean value of what is being observed:
where:
x is the score of a given UserPoint,
μ is the mean of the population,
σ is the standard deviation of the population.
Verify
The experiment is in Draft status after creation, and its groups are not yet populated.
Clicking Compute moves it through In progress and Uploading to Completed. See Read and interpret your experiment results.
The number of strata is consistent with the size of your segment. Many criteria, or criteria with many distinct values, multiply the strata and shrink each one.
Related pages
Last updated
Was this helpful?