> For the complete documentation index, see [llms.txt](https://docs.mediarithmics.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.mediarithmics.io/setup-and-monitor/collect-and-manage-your-data/import-data-in-batch.md).

# Import data in batch

Upload files of activities, profiles, segments, identifiers, and consent into your datamart.

## Your goal

Load data into the datamart from files rather than event by event - offline purchases, CRM profiles, email lists, identifier associations, suppression requests.

## Expected outcome

Imports that run to completion, with a success count you can check and an error file you can act on.

{% hint style="info" %}
If you need to track users in real time, use [Collect data in real time](/setup-and-monitor/collect-and-manage-your-data/collect-data-in-real-time.md) instead.
{% endhint %}

## What you can import

* **Offline activities** such as offline purchases and store visits.
* **User segments** such as email lists, cookie lists, user account lists.
* **User profiles** such as CRM data and scoring.
* **User associations**, such as CRM onboarding.
* **User dissociations**.
* **User suppression requests**, such as GDPR suppression requests and opt-out management.

## How it works

You upload files associated with a **document import definition**:

* **Files** represent the data.
* **Document imports** represent what the platform should do with the data.

Each line in the uploaded file is a **command to execute**. Which commands are available depends on the document import type.

```mermaid
flowchart LR
    D["Document import<br/>what to do with the data"]
    E1["Execution 1<br/>file uploaded"]
    E2["Execution 2<br/>file uploaded"]
    DM[("Datamart")]
    D --> E1 --> DM
    D --> E2 --> DM
```

{% stepper %}
{% step %}

### Create the document import definition

Tell the platform what you are importing.
{% endstep %}

{% step %}

### Upload files against it

Each uploaded file creates a new **document import execution**.
{% endstep %}
{% endstepper %}

{% hint style="success" %}
**Create a new document import for each new set of files.** If you upload CRM profiles every night, create a new "User profiles from CRM - *date*" document import every night rather than adding files to one permanent import. It keeps executions and errors attributable.
{% endhint %}

## Identifiers in imports

Imports need [user identifiers](/setup-and-monitor/collect-and-manage-your-data/design-your-data-model-and-schema/user-identity/user-identifiers.md) so your data attaches to the right UserPoint.

{% hint style="warning" %}
**Only one identifier is allowed per line.** Do not specify a user agent ID on a line that already uses an email hash.

You can, however, mix identifier types **across** lines of the same document: one line can use an account ID and the next an email hash.
{% endhint %}

## The document import object

| Field                     | Type    | Description                                                                                                                                                                                                                 |
| ------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `document_type`           | Enum    | `USER_ACTIVITY`, `USER_SEGMENT`, `USER_PROFILE`, `USER_CHOICE`, `USER_IDENTIFIERS_DELETION`, or `USER_IDENTIFIERS_ASSOCIATION_DECLARATIONS`.                                                                                |
| `mime_type`               | Enum    | `APPLICATION_X_NDJSON` or `TEXT_CSV`. It must match the file format. **`TEXT_CSV` can only be chosen for `USER_SEGMENT` imports.**                                                                                          |
| `encoding`                | String  | Encoding of the data, usually `utf-8`.                                                                                                                                                                                      |
| `name`                    | String  | The name of your import.                                                                                                                                                                                                    |
| `priority`                | Enum    | `LOW`, `MEDIUM`, or `HIGH`.                                                                                                                                                                                                 |
| `use_processing_pipeline` | Boolean | Whether the import goes through the [processing pipeline](/setup-and-monitor/collect-and-manage-your-data/collect-data-in-real-time.md#the-processing-pipeline) - activity analyzers, session aggregation. Default `false`. |
| `shuffle_lines`           | Boolean | Shuffles the lines of the file for better performance. Default `true`.                                                                                                                                                      |

```json
{
    "document_type": "USER_ACTIVITY",
    "mime_type": "APPLICATION_X_NDJSON",
    "encoding": "utf-8",
    "name": "<YOUR_DOCUMENT_IMPORT_NAME>"
}
```

{% hint style="info" %}
`use_processing_pipeline` is the setting people miss. By default an imported activity does **not** pass through activity analyzers or session aggregation, so it must already be correctly formatted. Set it to `true` when you want imported activities treated like tracked ones.
{% endhint %}

## Create and manage a document import

```
POST   https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports
GET    https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports
GET    https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports/:importId
PUT    https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports/:importId
DELETE https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports/:importId
```

```bash
curl -X POST \
  "https://api.mediarithmics.com/v1/datamarts/<DATAMART_ID>/document_imports" \
  -H 'Authorization: <YOUR_API_TOKEN>' \
  -H 'Content-Type: application/json' \
  -d '{
        "document_type": "USER_ACTIVITY",
        "mime_type": "APPLICATION_X_NDJSON",
        "encoding": "utf-8",
        "name": "<YOUR_DOCUMENT_IMPORT_NAME>"
      }'
```

The listing supports filters as query parameters: `keywords` matches import names and is **case sensitive**; `mime_type` filters on `APPLICATION_X_NDJSON` or `TEXT_CSV`; `document_types` filters on `USER_PROFILE`, `USER_ACTIVITY`, or `USER_SEGMENT`, several separated by commas; and `order_by=name` sorts by name instead of by ID. The listing is paginated.

## Upload a file, and follow the execution

A file upload **creates an execution**.

```
POST https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports/:importId/executions
```

```bash
curl --location --request POST \
  'https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports/:importId/executions' \
  --header 'Content-Type: application/x-ndjson' \
  --header 'Authorization: <YOUR_API_TOKEN>' \
  --data-binary '@/path/to/the/file.ndjson'
```

The response returns metadata including an `id` you use to track the execution.

### Execution statuses

```mermaid
stateDiagram-v2
    [*] --> PENDING: file uploaded
    PENDING --> RUNNING: import starts
    RUNNING --> SUCCEEDED: file imported
    PENDING --> CANCELED: cancel requested
```

| Status      | Meaning                                        |
| ----------- | ---------------------------------------------- |
| `PENDING`   | The execution has been created.                |
| `RUNNING`   | The import has started.                        |
| `SUCCEEDED` | The platform has finished processing the file. |

{% hint style="danger" %}
`SUCCEEDED` means the file was **processed**, not that every line worked. An execution can be `SUCCEEDED` with `total_success: 0` and `total_failure: 4`. Always read the result, not just the status.
{% endhint %}

### Reading the result

```
GET https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports/:importId/executions
GET https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports/:importId/executions/:executionId
```

The `result` object of an execution tells you what happened:

| Field                                             | What it tells you                                                                           |
| ------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| `total_success`                                   | Lines imported successfully.                                                                |
| `total_failure`                                   | Lines that failed.                                                                          |
| `error_file_uri`                                  | A CSV listing the errors - this is where you go to fix the data.                            |
| `possible_issue_on_identifiers`                   | Whether the platform suspects a problem with the identifiers used.                          |
| `top_identifiers`                                 | The identifiers seen most often, useful to spot a value repeated where it should be unique. |
| `num_tasks`, `completed_tasks`, `erroneous_tasks` | Progress and failure counts.                                                                |

The `error` object carries the messages, each with an error ID you can pass to support.

### Cancelling

```
POST https://api.mediarithmics.com/v1/datamarts/:datamartId/document_imports/:importId/executions/:executionId/action
```

With the body `{"action":"CANCEL"}`. Cancellation only works while the execution is `PENDING`.

## File size and performance

{% hint style="info" %}
For maximum performance:

* Keep each file to a maximum size of **100 MB**.
* Use a document import for multiple records when there will be more than **1,000 records per file**.
  {% endhint %}

To import more than 100 MB, split the file before uploading and call the API several times:

```bash
split -l <LINE_NUMBER> ./your/file/path
```

## In this section

<table data-view="cards" data-search="false"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Import activities</strong></td><td>Offline purchases, store visits, and other activities.</td><td><a href="/setup-and-monitor/collect-and-manage-your-data/import-data-in-batch/import-activities.md">Import activities</a></td></tr><tr><td><strong>Import profiles</strong></td><td>CRM data, with the update strategies.</td><td><a href="/setup-and-monitor/collect-and-manage-your-data/import-data-in-batch/import-profiles.md">Import profiles</a></td></tr><tr><td><strong>Import segments</strong></td><td>Lists of users to place in a segment.</td><td><a href="/setup-and-monitor/collect-and-manage-your-data/import-data-in-batch/import-segments.md">Import segments</a></td></tr><tr><td><strong>Import identifiers</strong></td><td>Deleting identifiers in bulk.</td><td><a href="/setup-and-monitor/collect-and-manage-your-data/import-data-in-batch/import-identifiers.md">Import identifiers</a></td></tr><tr><td><strong>Import user choices</strong></td><td>Consent decisions in bulk.</td><td><a href="/setup-and-monitor/collect-and-manage-your-data/import-data-in-batch/import-user-choices.md">Import user choices</a></td></tr><tr><td><strong>Data loader</strong></td><td>The same imports, without writing API calls.</td><td><a href="/setup-and-monitor/collect-and-manage-your-data/import-data-in-batch/data-loader.md">Data loader</a></td></tr><tr><td><strong>Enrich users with identifiers</strong></td><td>Declaring that identifiers belong together.</td><td><a href="/setup-and-monitor/collect-and-manage-your-data/import-data-in-batch/enrich-users-with-identifiers.md">Enrich users with identifiers</a></td></tr><tr><td><strong>Enrich user profiles</strong></td><td>Targeted updates inside a profile.</td><td><a href="/setup-and-monitor/collect-and-manage-your-data/import-data-in-batch/enrich-user-profiles.md">Enrich user profiles</a></td></tr></tbody></table>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.mediarithmics.io/setup-and-monitor/collect-and-manage-your-data/import-data-in-batch.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.