> For the complete documentation index, see [llms.txt](https://docs.mediarithmics.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.mediarithmics.io/measurement-and-attribution/measure-and-optimize/set-up-an-a-b-test/create-your-test-and-control-groups.md).

# Create your test and control groups

Configure the groups, randomisation, and addressability of an experiment, and the datamart-level settings behind the form.

## Your goal

Turn a segment into a set of test and control groups that are large enough, addressable enough, and, if the campaign outcome depends on it, balanced on the criteria that matter.

## Expected outcome

An experiment in **Draft** status, ready to compute, with its groups configured the way your measurement needs them.

## Configure the experiment

From the segment page, click **More**, then **Create Experiment**. The form is divided into four sections.

{% stepper %}
{% step %}

### General information

Name the experiment. The name helps you identify it in the experiment table, and is used by default in the name of the groups generated. A description is optional, and lets you comment on the experiment for yourself or your colleagues.
{% endstep %}

{% step %}

### Groups

Specify the number and size of the groups. An experiment may have between **1 and 10 test groups**, and may or may not have a control group. It must consist of **at least two groups**, test and/or control.

If a [permanent control group](#permanent-control-group) is set up on your datamart and you wish to use it, the size of your control group is set by default with no possibility of modifying it. All UserPoint in your segment belonging to the permanent control group constitute the control group for this experiment.
{% endstep %}

{% step %}

### Randomization

Decide between the two types of randomization described below.
{% endstep %}

{% step %}

### Addressability

Define the identifiers that will be used for activation, which in practice means selecting the identifiers you will later configure in your feeds. The aim is to retain only those UserPoint that have the required identifiers before randomization starts.

{% hint style="info" %}
If several identifiers are selected, only UserPoint with **all** of them are retained.
{% endhint %}

{% hint style="info" %}
Whatever identifiers are selected, randomization is calculated at the **UserPoint** level.
{% endhint %}
{% endstep %}
{% endstepper %}

When the four sections are complete, click **Create experiment**. You are redirected to the experiment details page, and the experiment is listed in the Experiments menu.

## The two randomization types

<table data-search="false"><thead><tr><th width="220">Type</th><th></th></tr></thead><tbody><tr><td><strong>Stratified randomization</strong></td><td>Use this when you want to ensure a balance of groups on certain key criteria that may influence your campaign outcome. It involves configuring at least one <a href="#representativeness-criteria">representativeness criterion</a>.</td></tr><tr><td><strong>Simple randomization</strong></td><td>A random draw using partitions. Each group is randomly assigned a number of partitions equivalent to its size, expressed as a percentage, configured in the previous section.</td></tr></tbody></table>

### Which one to choose

The choice depends on your knowledge of your audience, data availability, and the level of rigor required for post-campaign analysis.

**Stratified randomization** is highly recommended when you know that specific criteria, such as age or purchase history, will heavily influence your objective. Besides guaranteeing balanced groups on those criteria, it significantly simplifies the post-campaign analysis phase: you only need to correct the *exposure bias*, meaning the fact that a portion of the test group did not actually see the ad. Unlike simple randomization, you do not also have to correct initial imbalances that occurred by chance. Stratification increases your **statistical power**: by eliminating random variation from the start, the test is much more capable of detecting the campaign's true impact.

**Simple randomization** is a basic random draw. It was historically chosen because stratified randomization was too complex to set up without a dedicated tool, but mediarithmics now makes both equally easy to configure. Opt for simple randomization primarily if:

* You **do not know which specific criteria** strongly influence the objective you are trying to achieve.
* You **lack the necessary data** to use as representativeness criteria.
* The available data would require extensive pre-processing, for example through computed fields, making the setup too long or complex for your current project constraints.

## Datamart-level configuration

The Experiments feature is configured through the Experiments menu in the Computing Console. Configurations are done at **datamart level**, and they shape what every experiment creation form offers.

### Default groups

Define the group configuration that appears by default in the experiment creation form. You are prompted to enter the most common configuration for your datamart.

### Control group minimum size

Configure a minimum expected size for the control group, expressed in number of UserPoint, below which an alert appears in the form inviting the user to modify the configuration.

{% hint style="info" %}
Whether the alert appears depends on the **estimated** size of the control group, calculated as: segment size in number of UserPoint, multiplied by the control group percentage entered in the form.
{% endhint %}

{% hint style="warning" %}
The identifiers selected in the Addressability section of the form are **not** analyzed for this estimate. The calculation assumes that all UserPoint in the experiment have the expected identifiers.
{% endhint %}

The aim is to prevent users from creating control groups that are too small for the incremental measurement to succeed. The alert is **not blocking**: the user can still create the experiment despite it.

### Permanent control group

This option sets up a control group at datamart level. It is useful for limiting noise if you want to run several experiments in parallel, or if you want to assess the performance of a channel over an extended period of time.

When enabled, it adds a third option to the experiment creation form, so the choice becomes: no control group, customizable control group, or permanent control group. Used in an experiment, it defines the control group automatically from the UserPoint belonging to the permanent control group.

{% hint style="info" %}
When you use the permanent control group in an experiment, the size of the control group is not modifiable, and approximates the size of the permanent control group configured on the datamart.
{% endhint %}

{% hint style="warning" %}
The permanent control group is created from partitions and is therefore **dynamic** by default: UserPoint are likely to move in and out of it over time. It cannot be built around representativeness criteria, and is therefore **not compatible with stratified randomization**.
{% endhint %}

You can renew the permanent control group on demand with the dedicated button. The new one excludes by default the partitions used in the previous one.

{% hint style="info" %}
A standard control group is calculated on the fly for one given experiment. The permanent control group is configured globally at datamart level, which is what makes it useful for evaluating a channel over the long term, or for running several A/B tests in parallel while limiting noise and overexposure.
{% endhint %}

## Stratified randomization settings

{% hint style="success" %}
Get in touch with your account manager if you want further information on how to enable this section.
{% endhint %}

### Representativeness criteria

A **representativeness criterion** is data that can be used to balance groups in stratified randomization. To be usable, the data must come from the schema and carry a value at the UserPoint level. Criteria are used for balancing because they are supposed to influence the expected result of the campaign.

{% hint style="info" %}
You are running a campaign with the objective of generating sales of a given product. You know that age, gender and purchase history on the category are 3 data strongly correlated to the probability of purchase, and you want the groups in your experiment balanced on those 3 dimensions. You therefore set up the following 3 data as representativeness criteria:

* age
* gender
* sales on the category over the last 12 months
  {% endhint %}

Two rules govern what can be used:

{% hint style="info" %}
A criterion must contain a **limited number of distinct values**. The default limit is **20**.

A `birthdate` field cannot be used as it is. Prefer an `age range` field that returns values such as "18-24 yo", "25-34 yo", and so on. If a criterion does not exist in your schema, create it as a computed field, documented in the **Setup and Monitor** section under *Collect and manage your data > Design your data model and schema > Computed fields*.
{% endhint %}

{% hint style="info" %}
**Numeric** fields containing more than **10** distinct values are automatically reprocessed to show only 3 values: low, medium, high. This is achieved by splitting the segment into 3 thirds.
{% endhint %}

Above 20 values, the criterion details are no longer displayed in the experiment overview.

### How strata are defined

Strata are defined automatically by cross-referencing the different values of your representativeness criteria.

{% hint style="info" %}
You configure two criteria: "Gender", with 2 values (Male, Female), and "Age Category", with 3 values (18-25, 26-40, 40+). The algorithm creates **6 distinct strata**: Male 18-25, Female 18-25, Male 26-40, Female 26-40, Male 40+, Female 40+.

The platform then distributes the UserPoint of each stratum proportionally across your different test and control groups, to ensure balance.
{% endhint %}

### Templates

A template is a set of representativeness criteria. Create templates when you tend to use the same criteria across your experiments. They are designed to:

* guide non-expert users who may have difficulty defining the representativeness criteria to use
* speed up the creation of your experiments

### Acceptance threshold

The acceptance threshold defines the cases where a stratum that is too small is excluded from an experiment. Its **default value is 1**. If the distribution of all UserPoint in a given stratum deviates by more than 1 point from the distribution defined in the Groups section of the creation form, the stratum is excluded.

{% hint style="info" %}
You created an experiment with 2 groups on a **50/50** distribution, and you left the acceptance threshold at its default value of **1**.

A stratum containing **200 UserPoint** splits into 100 and 100, which is exactly **50/50**. It deviates by 0 points from your distribution, so it is kept.

A stratum containing **25 UserPoint** cannot be split evenly. The closest it gets is 13 and 12, which is **52/48**, so each group deviates by 2 points. That is more than the threshold, and the stratum is eliminated. Were it kept, its UserPoint would be over-represented in the test group and under-represented in the control group compared with the strata where the balance could be respected. A stratum of **5 UserPoint** is eliminated for the same reason, by a wider margin: its closest split is 3 and 2, which is **60/40**, or 10 points off.

Raise the acceptance threshold to **2** and the 25 UserPoint stratum is kept, because a deviation of 2 points is no longer more than the threshold. The 5 UserPoint stratum is still eliminated, because 10 points is.
{% endhint %}

### Outliers

Defining outliers lets you exclude from your experiments the UserPoint that exhibit exotic behavior which, analyzed alongside other UserPoint, could distort results. A company placing an order for its 200 employees all at once is the classic case.

Outliers are defined only for numerical values, analyzed using the **Z-score**. The Z-score is the number of standard deviations by which the value of a UserPoint score is above or below the mean value of what is being observed:

$$
{\displaystyle z={x-\mu  \over \sigma }}
$$

where:

* x is the score of a given UserPoint,
* μ is the mean of the population,
* σ is the standard deviation of the population.

{% hint style="info" %}
You use sales over 12 months as a representativeness criterion, and you set outliers to be excluded as soon as their Z-score is equal to or greater than **5**. Across your segment the mean is **100** and the standard deviation is also **100**, because a handful of very large buyers stretch the distribution.

That threshold puts the cutoff at 100 + (5 x 100), so at **600**.

A UserPoint with sales of **500** has a Z-score of (500 - 100) / 100 = **4**, below the threshold, so it stays in the experiment. A UserPoint with sales of **1200** has a Z-score of (1200 - 100) / 100 = **11**, far above it. That UserPoint is excluded, as is every UserPoint above 600.
{% endhint %}

## Verify

* The experiment is in **Draft** status after creation, and its groups are not yet populated.
* Clicking **Compute** moves it through **In progress** and **Uploading** to **Completed**. See [Read and interpret your experiment results](/measurement-and-attribution/measure-and-optimize/set-up-an-a-b-test/read-and-interpret-your-experiment-results.md).
* The number of strata is consistent with the size of your segment. Many criteria, or criteria with many distinct values, multiply the strata and shrink each one.

## Related pages

<table data-view="cards" data-search="false"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Read and interpret your experiment results</strong></td><td>Checking balance, and what happens to excluded UserPoint.</td><td><a href="/measurement-and-attribution/measure-and-optimize/set-up-an-a-b-test/read-and-interpret-your-experiment-results.md">Read and interpret your experiment results</a></td></tr><tr><td><strong>Set up an A/B test</strong></td><td>What an experiment is, and why incrementality needs a control group.</td><td><a href="/measurement-and-attribution/measure-and-optimize/set-up-an-a-b-test.md">Set up an A/B test</a></td></tr></tbody></table>

{% hint style="info" %}
The segment **type** this experiment produces is documented in the **Audience & Activation** section under *Build your audiences > Segment reference > Segment types > Experiment groups segment*. How the groups are named is on [Read and interpret your experiment results](/measurement-and-attribution/measure-and-optimize/set-up-an-a-b-test/read-and-interpret-your-experiment-results.md#groups). Partitions, which simple randomization draws from, are on **Partition segment** in the same subtree.
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.mediarithmics.io/measurement-and-attribution/measure-and-optimize/set-up-an-a-b-test/create-your-test-and-control-groups.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.