> For the complete documentation index, see [llms.txt](https://docs.mediarithmics.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.mediarithmics.io/setup-and-monitor/collect-and-manage-your-data/export-your-data/file-destination.md).

# File destination

Define where and how generated files are delivered - SFTP, S3, or Google Cloud Storage.

## Your goal

Declare a delivery target for the files the platform generates, so a feed can write to it - a partner's SFTP, your S3 bucket, a Google Cloud Storage bucket.

## Expected outcome

A validated destination with a token, ready to be referenced, and file paths and names that make the output easy to consume downstream.

## What a destination file is

A **destination file** defines **where and how** data files generated by the platform are delivered. It is configured in the Computing Console.

{% hint style="warning" %}
**On its own, a destination file does not export anything.** It becomes effective when a **feed** - through its stream-to-file capability - references it using its auto-generated **token**. The feed exports data chunks to the delivery file system, which aggregates them into a single file and delivers it to the destination.
{% endhint %}

Typical uses:

* Sending an audience segment's user identifiers to an external partner for activation.
* Supporting custom connectors that rely on file-based delivery.

Supported targets:

| Type                   | What it is                     |
| ---------------------- | ------------------------------ |
| `SFTP`                 | Secure file transfer over SSH. |
| `S3`                   | Amazon S3 buckets.             |
| `GOOGLE_CLOUD_STORAGE` | Google Cloud Storage buckets.  |

## Set one up

Setup is done in **Computing Console > File destinations**.

{% stepper %}
{% step %}

### Create the destination

Click **New file delivery destination**, fill in the configuration fields below, and save.
{% endstep %}

{% step %}

### Add the credentials

In the action menu of the destination, select **Credentials**. Copy the provided template - it differs per storage type - fill in the required fields, and save.

{% hint style="info" %}
Once saved, **you cannot read the credentials file again**. You can overwrite it.
{% endhint %}
{% endstep %}

{% step %}

### Validate the connection

In the action menu, select **Validate**.

{% hint style="info" %}
To validate, the integration user must have an **Editor** role on the object storage - or equivalent - with permission to **create and delete a temporary empty file** for verification.
{% endhint %}
{% endstep %}

{% step %}

### Use it

Reference the destination from a feed using its token.
{% endstep %}
{% endstepper %}

## Configuration fields

### General information

| Field              | Description                                                                                             |
| ------------------ | ------------------------------------------------------------------------------------------------------- |
| **Name**           | Human-readable name used in the interface.                                                              |
| **Technical name** | Internal identifier used by connectors. Referenced in plugin code, with no functional impact by itself. |
| **Type**           | `SFTP`, `S3`, or `GOOGLE_CLOUD_STORAGE`.                                                                |
| **Deduplicate**    | When enabled, removes *perfect duplicates* within a file - two strictly identical rows.                 |

### File information

| Field                 | Description                                                                                                                              |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| **File path macro**   | Path from the root of the destination where the file is written. Must **not start with `/`** and must **end with `/`**. Supports macros. |
| **File name macro**   | The file name to generate. Supports macros.                                                                                              |
| **Compression**       | `NONE`, `GZIP`, or `ZIP`.                                                                                                                |
| **Encoding**          | Character encoding. Recommended and supported: `UTF-8`.                                                                                  |
| **File header macro** | Optional header written at the top of the file, for instance column names. Can be empty. Supports macros.                                |

## Macros

Macros rely on **Freemarker** templating, and can be used in the file path, the file name, and the file header. They are what let you organise files dynamically by time, segment, or feed configuration.

| Macro                    | What it evaluates to                                                |
| ------------------------ | ------------------------------------------------------------------- |
| `DATE`                   | The date at which the file is generated.                            |
| `PROPERTY`               | A property defined in the plugin configuration.                     |
| `SEGMENT.ID`             | The segment ID.                                                     |
| `SEGMENT.TECHNICAL_NAME` | The segment technical name.                                         |
| `GROUPING_KEY`           | The `grouping_key` defined in the plugin code. Plugin context only. |
| `RANDOM_UUID`            | A random UUID, for uniqueness.                                      |

Freemarker date formatting structures folders:

```
data/partner/events/${DATE?string["yyyy/MM/dd"]}/
```

produces:

```
data/partner/events/2025/03/18/
```

<details>

<summary>A complete example</summary>

**File path macro**

```
data/exports/${SEGMENT.TECHNICAL_NAME}/${DATE?string["yyyy/MM"]}/
```

**File name macro**

```
events_${GROUPING_KEY}_${RANDOM_UUID}.csv
```

This produces monthly-partitioned files, uniquely identified per execution.

</details>

## The grouping key

The **grouping key** is used **exclusively in plugins**, during the `user_segment_update` phase, to control how records are aggregated. It defines **which records belong to the same logical file stream**: records sharing a grouping key value are written into the same file or files.

It is defined **in the plugin code**, not in the destination configuration - the destination references it through `${GROUPING_KEY}`.

| Strategy               | Effect                                                                                                                                                            |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Group by `segment_id`  | Accumulates records belonging to the same segment. Can mix data from **several plugin instances** targeting the same segment. Useful for segment-centric exports. |
| Group by `datamart_id` | Accumulates records from all plugin instances in the same datamart. Produces consolidated datamart-level exports.                                                 |
| Group by `feed_id`     | Accumulates records from **a single plugin instance only**. Ensures strict isolation between feeds.                                                               |

## Upload triggers

You must choose one trigger type. **A file is uploaded as soon as one of the active conditions is met.**

{% tabs %}
{% tab title="Size-based" %}

| Field                  | Description                                                                                                                                        |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| **File max size (KB)** | As soon as the aggregated data reaches this size, the file is uploaded immediately. Recommended value: **100,000 KB, about 100 MB**.               |
| **Min upload per day** | Controls the upload frequency - the number of upload windows per rolling 24-hour period. Acts as a secondary trigger when the size is not reached. |

**File max size is always the main limiting factor.** While records are written, reaching the size limit uploads the file immediately, regardless of time-based settings.

{% hint style="warning" %}
Despite its name, **min upload per day** should be read as *the number of upload windows per rolling 24-hour period*. It defines a time-based flush interval: even if the size limit is not reached, the file is uploaded at the end of each interval. The period is **rolling**, not aligned to clock boundaries.

```
interval = 24 hours / min_upload_per_day
```

{% endhint %}

**Example 1** - `File max size = 100000`, `Min upload per day = 1`

Files are uploaded when they reach 100 MB; otherwise at the end of a rolling 24-hour period. This does **not guarantee daily delivery** if the feed produces no data, and suits unpredictable volumes that may need more than one file per day.

**Example 2** - `File max size = 100000`, `Min upload per day = 24`

Files are uploaded when they reach 100 MB; otherwise at the end of a rolling 1-hour period. This caps delivery at **at most one file per hour**, and is **not suitable** if more than one file per hour is required.
{% endtab %}

{% tab title="Frequency-based" %}

| Field                  | Description                                                                                                                                                                       |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Max upload per day** | Divides a rolling 24-hour period into equal upload windows to cap the daily file count. Uploads are only delayed if the destination server is still processing the previous file. |

```
interval = 24 hours / max_upload_per_day
```

Files are then uploaded so as to respect the maximum number of files over the period.

{% hint style="info" %}
If a file with **exactly the same name** is already present on the destination server, the upload is delayed and the incoming data is aggregated into the next upload. This is a reason to include `${RANDOM_UUID}` or a timestamp in the file name macro.
{% endhint %}
{% endtab %}
{% endtabs %}

## Best practices

* Use **compression** for large exports, to reduce transfer and storage costs.
* Prefer **date-based partitioning** in paths - it simplifies downstream processing and retention.
* Make file names unique unless you deliberately want aggregation into an existing file.

## Verify

* **Validate** succeeds on the destination, which confirms the credentials and the write permission.
* A first delivery arrives at the expected path, with the expected name.
* Macros are resolved in the delivered path and name - a literal `${DATE}` means the macro was not evaluated.
* File sizes and delivery frequency match the trigger you configured.

## Related pages

<table data-view="cards" data-search="false"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Export your data</strong></td><td>Choosing between the export routes.</td><td><a href="/setup-and-monitor/collect-and-manage-your-data/export-your-data.md">Export your data</a></td></tr><tr><td><strong>Data warehouse export</strong></td><td>Using a file destination to feed a warehouse.</td><td><a href="/setup-and-monitor/collect-and-manage-your-data/export-your-data/data-warehouse-export.md">Data warehouse export</a></td></tr><tr><td><strong>Build a custom feed plugin</strong></td><td>The plugin side, including the grouping key.</td><td><a href="/setup-and-monitor/extend-the-platform/build-a-custom-feed-plugin.md">Build a custom feed plugin</a></td></tr></tbody></table>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.mediarithmics.io/setup-and-monitor/collect-and-manage-your-data/export-your-data/file-destination.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.