> For the complete documentation index, see [llms.txt](https://docs.mediarithmics.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.mediarithmics.io/setup-and-monitor/collect-and-manage-your-data/export-your-data/datamart-replication.md).

# Datamart replication

Stream every ingested change to your own cloud: a message queue, or a data warehouse.

## Your goal

Mirror the data the platform ingests into your own cloud, continuously, so your systems can react to changes without polling the API.

## Expected outcome

An active replication delivering one message per `update` or `delete` operation, for the object types you selected.

{% hint style="info" %}
This module is **not included in the default plan**. Contact your account manager to activate it.
{% endhint %}

## The two destination families

A replication has a **type**, and the type decides both where the data goes and the shape it arrives in.

| Family              | Destinations                                       | Delivery                                                                 | Documented on                                                                                                       |
| ------------------- | -------------------------------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- |
| **Message queues**  | Google Pub/Sub, Microsoft Azure Event Hubs (Alpha) | One message per operation, straight to the topic.                        | This page.                                                                                                          |
| **Data warehouses** | Google BigQuery, Snowflake                         | Files written to a cloud storage bucket, read into log and state tables. | [Data warehouse export](/setup-and-monitor/collect-and-manage-your-data/export-your-data/data-warehouse-export.md). |

The rest of this page describes the replication object itself, which both families share, and the message format of the message-queue family.

## What is replicated

Replication mirrors the `update` and `delete` operations on your datamart for these objects:

* `UserPoint`
* `UserAgent` - for datamarts with a `user_point_system_version` before `v202205`
* `UserDevicePoint`, `UserDeviceTechnicalId` - for `v202205`
* `UserActivity`
* `UserSegment`
* `UserProfile`
* `UserAccount`
* `UserEmail`
* User computed field, and user ML function values
* Parent UserPoint, when [UserPoints merge](/setup-and-monitor/collect-and-manage-your-data/design-your-data-model-and-schema/user-identity/userpoint-and-identity-graph.md#userpoint-merges)

The exact list depends on the family: a warehouse replication carries user events rather than the UserPoint and parent-UserPoint operations. See [Data warehouse export](/setup-and-monitor/collect-and-manage-your-data/export-your-data/data-warehouse-export.md#what-lands-in-the-warehouse).

{% hint style="warning" %}
When you create a replication you select which objects to replicate, and **you cannot change that selection afterwards**. Changing it means creating a new replication.
{% endhint %}

## Create a replication

{% stepper %}
{% step %}

### Create the object

In the Computing console, go to the **Data stream** menu and click **New Replication**. Select the **replication type** matching your destination, complete the configuration, and click **Save Replication**. It appears in the **Replications** subtab.
{% endstep %}

{% step %}

### Supply the credentials

A replication needs credentials for its destination: the access rights on the message queue service, or the editing rights on the object storage used by a warehouse replication.
{% endstep %}

{% step %}

### Activate it

A new replication is created **PAUSED**. Activate it to start delivery. If the platform cannot replicate when you activate, you get an error - almost always a credentials problem, so check the configuration and the credentials file first.
{% endstep %}
{% endstepper %}

## Versions

| Version                | Format      | Supported destination                      |
| ---------------------- | ----------- | ------------------------------------------ |
| **Version 1** (legacy) | JSON        | Google Pub/Sub, Microsoft Azure Event Hubs |
| **Version 2**          | Avro binary | Google Pub/Sub                             |

* Any **new** Google Pub/Sub replication is created in **Version 2**.
* You **cannot upgrade** a replication from Version 1 to Version 2.

The legacy JSON format was designed for a streaming architecture - Dataflow, Databricks - and has limitations with tools that need a schema, such as BigQuery. Version 2 introduces a schema to make integration easier.

## The message format

This is the format of a **message queue** replication. A warehouse replication writes Avro records instead, described in [Data warehouse export](/setup-and-monitor/collect-and-manage-your-data/export-your-data/data-warehouse-export.md#what-lands-in-the-warehouse).

Operations are converted to a standardised shape:

```
operation = {ts, doc_type, doc_id, op, value}
```

| Field      | Type                 | What it holds                                                                                                                                                                 | Versions  |
| ---------- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- |
| `ts`       | UNIX timestamp in ms | The mutation date.                                                                                                                                                            | All       |
| `doc_type` | Enumeration          | The object type: `UserActivity`, `UserProfile`, `UserSegment`, `UserAgent`, `UserAccount`, `UserEmail`, `UserPoint`, `UserPointParent`, `UserComputedField`, or `MlFunction`. | All       |
| `ctx_id`   | UUID                 | The UserPoint ID.                                                                                                                                                             | Version 2 |
| `doc_id`   | String               | The object's unique ID. The format varies per `doc_type`.                                                                                                                     | All       |
| `op`       | String               | `UPDATE` or `DELETE`.                                                                                                                                                         | All       |
| `value`    | JSON object          | The object value. The format varies per `doc_type`.                                                                                                                           | All       |

Messages carry metadata to help you filter the topic - attributes in Pub/Sub, properties in Event Hubs - with the key `doc_type` holding the message's document type.

### Identifiers and values per object type

| `doc_type`              | `doc_id`                                                                  | `value`                                                                           |
| ----------------------- | ------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| `UserPoint`             | `{{user_point_id}}`                                                       | Empty - the ID is already in `doc_id`.                                            |
| `UserAgent`             | `{{user_point_id}}:{{vector_id}}`                                         | Browser and device info.                                                          |
| `UserDevicePoint`       | `{{user_point_id}}:{{user_device_point_id}}`                              | Browser and device info.                                                          |
| `UserDeviceTechnicalId` | `{{user_point_id}}:{{user_device_point_id}}:{{user_device_technical_id}}` | Empty.                                                                            |
| `UserActivity`          | `{{user_point_id}}:{{user_activity_id}}`                                  | The detailed activity.                                                            |
| `UserSegment`           | `{{user_point_id}}:{{segment_id}}`                                        | Segment info.                                                                     |
| `UserProfile`           | `{{user_point_id}}:{{compartment_id}}:{{user_account_id}}`                | The detailed profile.                                                             |
| `UserAccount`           | `{{user_point_id}}:{{compartment_id}}:{{user_account_id}}`                | Empty.                                                                            |
| `UserEmail`             | `{{user_point_id}}:{{email_hash}}`                                        | The user's email hash.                                                            |
| `UserPointParent`       | The ID of the UserPoint **merged into** the oldest one - the one kept.    | A message stating `<current_user_point_id> merged with <the_kept_user_point_id>`. |
| `UserComputedField`     | `{{user_point_id}}:{{plugin_instance_id}}`                                | The computed field value.                                                         |
| `MlFunction`            | `{{user_point_id}}:{{ml_function_id}}:{{field_type_name}}`                | The ML function value.                                                            |

{% hint style="info" %}
`UserPointParent` is how you keep your own store consistent with merges. Without handling it, your copy keeps two users where the platform now has one.
{% endhint %}

<details>

<summary>Example messages, legacy JSON format</summary>

A new activity:

```json
{
   "ts": 1676627112685,
   "doc_type": "UserActivity",
   "doc_id": "XXXXXXX-XXXX-XXX-XXXXXXXX:XXXXXX-XXXXX-XXXX-XXXX-XXXXXXXXXX",
   "op": "UPDATE",
   "value": {
       "$type": "SITE_VISIT",
       "$source": "XXXX",
       "etc": "etc"
   }
}
```

A new device point:

```json
{
   "ts": 1676627112685,
   "doc_type": "UserDevicePoint",
   "doc_id": "4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893",
   "op": "UPDATE",
   "value": {
      "$os_family": "LINUX",
      "$brand": null,
      "$os_version": null,
      "$form_factor": "PERSONAL_COMPUTER",
      "$carrier": null,
      "$model": null,
      "$creation_ts": 0,
      "$browser_family": "FIREFOX"
   }
}
```

A new device technical ID - the value is empty because the identifier is in `doc_id`:

```json
{
   "ts": 1676627112685,
   "doc_type": "UserDeviceTechnicalId",
   "doc_id": "4700c85f-17e3-4304-aa7f-dc140173b08d:udp:-32453299893:mum:7231822539",
   "op": "UPDATE",
   "value": {}
}
```

</details>

## Replication status

| Status     | Meaning                                                                                                                                                                     |
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **ACTIVE** | All data processed by your datamart is replicated to your external solution.                                                                                                |
| **PAUSED** | No data is replicated.                                                                                                                                                      |
| **ERROR**  | The system can no longer replicate messages. Check your external solution - expired instance, invalid credentials. If nothing is wrong there, contact your account manager. |

## Initial synchronisation

An **initial synchronisation** replicates all existing documents selected in the replication and already stored in the platform into your cloud environment. It can be run for one or several **ACTIVE** replications, and not more than once a week.

{% tabs %}
{% tab title="One replication" %}

1. Find the replication in the list.
2. Click the caret-down icon and select **New Initial Synchronization**.
   {% endtab %}

{% tab title="Several replications" %}

1. Click the **New Initial Synchronization** button.
2. Select the replications to synchronise.
3. Click **Execute Initial Synchronization(s)**.
   {% endtab %}
   {% endtabs %}

Every selected replication receives a set of `UPDATE` operations representing all existing elements - every `UserProfile`, for example.

{% hint style="warning" %}
An initial synchronisation can produce a **very large volume of messages**, and processing them can be **expensive** depending on your cloud provider. Size the consumer and the budget before triggering one.
{% endhint %}

## Verify

* The replication status is **ACTIVE**.
* Messages arrive in the topic for the object types you selected, and only those.
* `doc_type` metadata is present, so your consumers can filter.
* Your consumer handles `UserPointParent` messages, so merges are reflected on your side.
* After an initial synchronisation, the message count is in the order of magnitude you expected.

## Related pages

<table data-view="cards" data-search="false"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Export your data</strong></td><td>Choosing between the export routes.</td><td><a href="/setup-and-monitor/collect-and-manage-your-data/export-your-data.md">Export your data</a></td></tr><tr><td><strong>UserPoint &#x26; Identity graph</strong></td><td>What a merge does, and why <code>UserPointParent</code> exists.</td><td><a href="/setup-and-monitor/collect-and-manage-your-data/design-your-data-model-and-schema/user-identity/userpoint-and-identity-graph.md">UserPoint &amp; Identity graph</a></td></tr><tr><td><strong>Data warehouse export</strong></td><td>The warehouse family: buckets, log and state tables, and the setup per warehouse.</td><td><a href="/setup-and-monitor/collect-and-manage-your-data/export-your-data/data-warehouse-export.md">Data warehouse export</a></td></tr></tbody></table>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.mediarithmics.io/setup-and-monitor/collect-and-manage-your-data/export-your-data/datamart-replication.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.