> For the complete documentation index, see [llms.txt](https://docs.mediarithmics.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.mediarithmics.io/collaboration/prepare-your-environment/manage-your-datasets.md).

# Manage your datasets

Create a dataset in your Bunker, by type and by source, and follow its updates.

## Your goal

Bring a piece of your own data into your [Bunker](https://docs.mediarithmics.io/collaboration/plan-and-integrate/understand-data-collaboration-concepts/bunkers.md) as a dataset, ready to be shared in a Clean Room, and keep it up to date.

## Before you start

Check which [dataset types and sources](https://docs.mediarithmics.io/collaboration/plan-and-integrate/understand-data-collaboration-concepts/datasets.md#dataset-types-and-sources-availability-matrix) are available to you, and agree with your partner on the [matching keys](https://docs.mediarithmics.io/collaboration/plan-and-integrate/understand-data-collaboration-concepts/matching-keys.md) you will use. Once a dataset exists, see [Share your datasets](https://docs.mediarithmics.io/collaboration/prepare-your-environment/share-your-datasets.md) to make it available in a Clean Room.

## Create a Segment dataset

{% tabs %}
{% tab title="Datamart" %}
Only available if you use the CDP module.

{% stepper %}
{% step %}
Head to the **Datasets** feature
{% endstep %}

{% step %}
Click **Add Datasets**
{% endstep %}

{% step %}
Hover **Segment** and select **From your datamart**
{% endstep %}

{% step %}
Search for one or multiple segments by typing their first letters
{% endstep %}

{% step %}
Click **Validate source(s)**
{% endstep %}

{% step %}
Update the default dataset name and expiration date if needed
{% endstep %}

{% step %}
Click **Create dataset**
{% endstep %}
{% endstepper %}

{% hint style="info" %}
If the segment was calculated the previous day, the dataset is available within a minute. If the segment was created today, the dataset becomes available after the segment is calculated - usually by the next day.
{% endhint %}

A dataset created this way is automatically refreshed at regular intervals (usually every night).
{% endtab %}

{% tab title="File" %}
Best for a one-time or occasional import, or for a partner without access to the Core CDP module.

{% stepper %}
{% step %}
Head to the **Datasets** feature
{% endstep %}

{% step %}
Click **Add Datasets**
{% endstep %}

{% step %}
Hover **Segment** and select **From a file**
{% endstep %}

{% step %}
Drag and drop a CSV file to preview its content
{% endstep %}

{% step %}
For each property: select whether it is a matching key, pick its type (used to generate standard dashboards), and update the default dataset name and expiration date if needed. Identifiers of your own environment, such as an account ID or a device ID, are under **Private identifier** - see [Where each kind of identifier appears](https://docs.mediarithmics.io/collaboration/plan-and-integrate/understand-data-collaboration-concepts/matching-keys.md#where-each-kind-of-identifier-appears)
{% endstep %}

{% step %}
Click **Create dataset**
{% endstep %}
{% endstepper %}

The file holds a single column: the identifier of each user to put in the segment. For example, with hashed emails:

```csv
email_hash
6b3a55e0261b0304143f805a24924d0c1c44524821305f31d9277843b8a10f4e
e3b98a4da31a127d4bde6e43033f66ba274cab0eb7eb1c70ec41402bf6273dd8
```

To update, click **Import a new file** with a CSV in the same format (same attributes), choosing **Full update** (replaces all previously imported data) or **Incremental update** (adds to existing data). The file must meet the [file requirements](#file-requirements).
{% endtab %}
{% endtabs %}

## Create a Segment catalog dataset

{% tabs %}
{% tab title="Datamart" %}
Only available if you use the CDP module.

{% stepper %}
{% step %}
Head to the **Datasets** feature
{% endstep %}

{% step %}
Click **Add Datasets**
{% endstep %}

{% step %}
Hover **Segment catalog** and select **From your datamart**
{% endstep %}

{% step %}
Search for one or multiple segments by typing their first letters
{% endstep %}

{% step %}
Click **Validate source(s)**
{% endstep %}

{% step %}
Update the default dataset name and expiration date if needed
{% endstep %}

{% step %}
Click **Create dataset**
{% endstep %}
{% endstepper %}

{% hint style="info" %}
If the segments were calculated the previous day, the dataset is available within a minute. If a segment was created today, the dataset becomes available after the segment is calculated - usually by the next day.
{% endhint %}

A dataset created this way is automatically refreshed at regular intervals (usually every night).
{% endtab %}

{% tab title="File" %}
{% hint style="info" %}
Creating a segment catalog from a file is in Alpha. Contact your Account Manager to learn more.
{% endhint %}

A data provider, including one that does not use the CDP, can bring a whole catalog of segments into its Bunker in batch, from files, rather than segment by segment. The catalog can then be [matched against a partner's segment](https://docs.mediarithmics.io/collaboration/collaborate-with-a-partner/analyze-cross-partner-behavior.md#match-rate) and used to [enrich user profiles](https://docs.mediarithmics.io/collaboration/collaborate-with-a-partner/enrich-user-profiles-with-partner-segments.md).

{% stepper %}
{% step %}

#### Create the dataset

In **Datasets**, create a **Segment catalog** dataset sourced from a file, and set:

* **Name**
* **Expiration date**
* **Matching key type** - the identifier type that every membership file of this dataset carries, for example a hashed email. It applies to every import and cannot be changed afterwards.
  {% endstep %}

{% step %}

#### Upload the segment taxonomy

Upload the [segment taxonomy file](#the-segment-taxonomy-file). It is optional when you create the dataset: you can upload or update it later from the dataset settings. Each upload replaces the previous taxonomy entirely, so upload a new one whenever segments are added to or removed from the catalog.
{% endstep %}

{% step %}

#### Import the memberships

Import the [membership files](#the-membership-file) in **Incremental** mode: each row adds, updates or deletes a membership through the `operation` column. To start again from an empty dataset, contact your Account Manager: an authorized person resets the dataset, then you import your membership files again.

You can run the imports yourself, with a [document import](https://docs.mediarithmics.io/setup-and-monitor/collect-and-manage-your-data/import-data-in-batch.md#run-a-document-import) whose **Destination type** is **Collaborative dataset** rather than **Datamart**: its identifier type comes from the dataset's matching key type. Or agree the delivery of your files and the import cadence with your Account Manager, and the mediarithmics team runs them.
{% endstep %}
{% endstepper %}
{% endtab %}
{% endtabs %}

### The segment taxonomy file

The **segment taxonomy** maps each segment's technical name, the one used in the membership files, to the name shown in the interface. It is a comma-separated CSV file with a header row and one row per segment.

<table data-search="false"><thead><tr><th width="230">Column</th><th width="110">Mandatory</th><th>What it holds</th></tr></thead><tbody><tr><td><code>segment_technical_name</code></td><td>Yes</td><td>The segment's unique identifier, used to reference it in the membership files.</td></tr><tr><td><code>name</code></td><td>Yes</td><td>The label shown in the interface.</td></tr><tr><td><code>description</code></td><td>No</td><td>A short description of the segment.</td></tr><tr><td><code>path</code></td><td>No</td><td>The segment's category and subcategories.</td></tr><tr><td><code>expiration_in_minutes</code></td><td>No</td><td>How long, in minutes, a user stays in the segment by default. <code>0</code>, a negative value or an empty value means no expiration.</td></tr></tbody></table>

```csv
segment_technical_name,name,description,path,expiration_in_minutes
2104,Lawn mower intent,Users who intend to buy a lawn mower,Intent > Home & Garden > Lawn mower,43200
5793,Watches interest,Users who are interested in watches,Interest > Apparel > Watches,129600
```

### The membership file

A membership file says which identifiers belong to which segments. It is a CSV file with a header row and one row per identifier and segment. Every identifier is of the dataset's matching key type, so the file carries no identifier type of its own.

<table data-search="false"><thead><tr><th width="230">Column</th><th width="110">Mandatory</th><th>What it holds</th></tr></thead><tbody><tr><td><code>identifier_value</code></td><td>Yes</td><td>The identifier, of the dataset's matching key type.</td></tr><tr><td><code>segment_technical_name</code></td><td>Yes</td><td>The segment the identifier belongs to, as declared in the segment taxonomy.</td></tr><tr><td><code>expiration_ts</code></td><td>Yes, can be empty</td><td>A timestamp at which the membership expires, overriding the taxonomy's default for this row. A timestamp at or before 1970-01-01 is ignored, like an empty value.</td></tr><tr><td><code>expiration_duration</code></td><td>Yes, can be empty</td><td>A duration, in minutes, after which the membership expires, overriding the taxonomy's default for this row. <code>0</code> or a negative value is ignored, like an empty value.</td></tr><tr><td><code>operation</code></td><td>Yes</td><td><code>UPDATE</code> to add or update the membership, <code>DELETE</code> to remove it.</td></tr></tbody></table>

```csv
identifier_value,segment_technical_name,expiration_ts,expiration_duration,operation
6b3a55e0261b0304143f805a24924d0c1c44524821305f31d9277843b8a10f4e,2104,,,UPDATE
6b3a55e0261b0304143f805a24924d0c1c44524821305f31d9277843b8a10f4e,5793,,60,UPDATE
e3b98a4da31a127d4bde6e43033f66ba274cab0eb7eb1c70ec41402bf6273dd8,2104,,,DELETE
```

Every membership file carries the `expiration_ts` and `expiration_duration` columns, but a row can leave them empty to keep the taxonomy's default. Fill in at most one of them on a row: setting both is an error.

A row's expiration only overrides the taxonomy's default when it holds a real value. For a segment and a row:

<table data-header-hidden data-search="false"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td>Taxonomy <code>expiration_in_minutes</code></td><td>Row <code>expiration_duration</code></td><td>The membership</td></tr><tr><td><code>0</code> or empty</td><td><code>0</code> or empty</td><td>Never expires</td></tr><tr><td><code>0</code> or empty</td><td><code>60</code></td><td>Expires after 60 minutes</td></tr><tr><td><code>1440</code></td><td><code>60</code></td><td>Expires after 60 minutes: the row wins</td></tr><tr><td><code>1440</code></td><td><code>0</code></td><td>Expires after 1,440 minutes: the taxonomy's value applies</td></tr></tbody></table>

{% hint style="warning" %}
A row cannot make a membership permanent when the segment's taxonomy sets a positive duration. To keep a segment's members indefinitely, set its `expiration_in_minutes` to `0`, or leave it empty, in the taxonomy.
{% endhint %}

Rows are skipped, rather than imported, when:

* a mandatory value is missing, `operation` is invalid, or a value is in the wrong format;
* `segment_technical_name` is not in the segment taxonomy. Add the segment to the taxonomy, upload it again, then re-run the import.

## Create a User profile dataset

A User profile dataset is created from a file. Best for a one-time or occasional import, or for a partner without access to the Core CDP module.

{% stepper %}
{% step %}
Head to the **Datasets** feature
{% endstep %}

{% step %}
Click **Add Datasets**
{% endstep %}

{% step %}
Hover **User profile** and select **From a file**
{% endstep %}

{% step %}
Drag and drop a CSV file to preview its content
{% endstep %}

{% step %}
For each property: select whether it is a matching key, pick its type (used to generate standard dashboards), and update the default dataset name and expiration date if needed. Identifiers of your own environment, such as an account ID or a device ID, are under **Private identifier** - see [Where each kind of identifier appears](https://docs.mediarithmics.io/collaboration/plan-and-integrate/understand-data-collaboration-concepts/matching-keys.md#where-each-kind-of-identifier-appears)
{% endstep %}

{% step %}
Click **Create dataset**
{% endstep %}
{% endstepper %}

The file holds one column per property. For example, a hashed email with profile attributes:

```csv
Email_hash,Gender,Birth_date,Birth_year,Postcode,State,City,Country
dc3456efdfe497dfse4564,Female,1971-05-27,1971,78840,TEXAS,Del Rio,USA
aeff45adfef9878gdeae5,Male,1972-01-23,1972,H2H 1V3,QUEBEC,Montreal,CANADA
```

To update, click **Import a new file** with a CSV in the same format (same attributes), choosing **Full update** (replaces all previously imported data) or **Incremental update** (adds to existing data). The file must meet the [file requirements](#file-requirements).

## File requirements

Files you upload with **From a file**, to create or update a **Segment** or **User profile** dataset, must meet these requirements:

| Item                            | Restriction / limit       |
| ------------------------------- | ------------------------- |
| Format                          | CSV, comma delimiter only |
| Encoding                        | UTF-8                     |
| Minimum number of records       | 10                        |
| Maximum file size (file import) | 500 MB                    |

Enclose text values in quotation marks (`"`) so that a value containing a comma is not read as two properties: the comma is the delimiter. Write dates in `YYYY-MM-DD` format.

{% hint style="warning" %}
Header property names may only contain letters, numbers, or underscores; must start with a letter or underscore; and are capped at 50 characters. `_TABLE_`, `_FILE_`, `_PARTITION`, `_ROW_TIMESTAMP`, `__ROOT__`, and `_COLIDENTIFIER` cannot be used. Duplicate property labels are rejected even if the case differs, and no spaces are allowed in a header name.
{% endhint %}

To import a file larger than 500 MB, split it into several files smaller than 500 MB with the same columns and headers, import the first with **Full update**, then import the rest with **Incremental update** in sequence.

## Import a file from cloud storage

In the managed version only, a dataset can be fed automatically from a file in your preferred cloud storage - useful for datasets with periodic updates. Contact your Account Manager to check whether your provider is supported. Once set up, an update runs automatically according to the refresh schedule you configured.

## Manage and monitor a dataset

Click a dataset's name on the Datasets listing to see:

* **Metadata** - name, creation date, creator, expiration date, number of records or users, number of Clean Rooms it is shared in, and last updated date.
* **Overview** - every property in the dataset, its definition (matching key, type), a sample (except for identifiers), and its fill rate.
* **Usage** - where it is shared, for what purposes, and whether it is being matched with a partner's dataset.
* **Update history** - visibility into updates that occurred on the dataset.

### Dataset and update status

| Dataset status                | Meaning                                                                                                                                                         |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Synchronization pending       | Available within an hour of creation if the underlying segment was already calculated; otherwise typically by 9:00 AM CET the day after the segment is computed |
| Update in progress            | An update is running and will be active shortly                                                                                                                 |
| Active                        | The last update completed successfully; the dataset can be shared                                                                                               |
| Active (last update in error) | The dataset can still be shared and used for matching, using the last successful update                                                                         |
| Error                         | No active update exists; the dataset cannot be used                                                                                                             |

| Update status | Meaning                                                                                                        |
| ------------- | -------------------------------------------------------------------------------------------------------------- |
| In progress   | An update is running                                                                                           |
| Completed     | The update finished - records with a malformed attribute or a missing matching key may not have been preserved |
| Error         | The update failed; check the information tooltip for the cause                                                 |

## Troubleshooting

**Why is my dataset not created?** A formatting error can get your file misread or rejected. Check that:

* every column has a header in the first row, following the header rules in [File requirements](#file-requirements);
* commas are used consistently as the delimiter;
* every row has the same number of columns;
* commas and quotes inside text values are handled, for example by enclosing the value in quotation marks;
* values have no leading or trailing spaces;
* critical columns have no empty values;
* each column keeps a single data type;
* dates use the `YYYY-MM-DD` format.

For a correctly formatted file, see the examples in [Create a Segment dataset](#create-a-segment-dataset) and [Create a User profile dataset](#create-a-user-profile-dataset).

## Boundaries

* A dataset's definition (its properties) cannot be changed once created. Update its content by importing a new file, or create a new dataset if the definition itself changed.
* When a dataset reaches its expiration date, it is automatically archived: you can no longer share it into a Clean Room, but existing matching reports that used it remain accessible.

## Related pages

<table data-view="cards" data-search="false"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Datasets</strong></td><td>What a dataset is, its types and sources, and which combinations are available.</td><td><a href="https://docs.mediarithmics.io/collaboration/plan-and-integrate/understand-data-collaboration-concepts/datasets.md">Datasets</a></td></tr><tr><td><strong>Bunkers</strong></td><td>Where every dataset is stored.</td><td><a href="https://docs.mediarithmics.io/collaboration/plan-and-integrate/understand-data-collaboration-concepts/bunkers.md">Bunkers</a></td></tr><tr><td><strong>Share your datasets</strong></td><td>Share a dataset into a Clean Room.</td><td><a href="https://docs.mediarithmics.io/collaboration/prepare-your-environment/share-your-datasets.md">Share your datasets</a></td></tr></tbody></table>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.mediarithmics.io/collaboration/prepare-your-environment/manage-your-datasets.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
