> For the complete documentation index, see [llms.txt](https://docs.mediarithmics.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.mediarithmics.io/setup-and-monitor/collect-and-manage-your-data/design-your-data-model-and-schema/data-planning-and-quality.md).

# Data planning and quality

Keep the data model, the collection, and the use cases in step - and catch the failures that are silent.

## Your goal

Write down what you intend to collect and why, then check that what actually arrives matches it - because most data-quality failures on this platform are silent rather than noisy.

## Expected outcome

A living data plan, and a small set of checks you run after every change to the schema, the tracking, or the privacy configuration.

## Why quality problems here are quiet

Nothing on this list produces an error you would notice by accident. Knowing the list is most of the job.

| Silent failure                                            | What actually happens                                                                                    |
| --------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| A property is missing from the schema                     | The data is still **stored**, but not indexed - so it is invisible to queries and segments.              |
| An indexed property is added later                        | Only **new** elements are indexed. `WHERE` queries run and return nothing for the data you already have. |
| `!` is used on a field that turns out to be null          | **The entire object** is not indexed.                                                                    |
| A `Date` field receives a timestamp                       | Queries on it throw an error, because there is no implicit conversion.                                   |
| A privacy wall is enforced before consent exists          | Incoming activities or profiles are **dropped**, and dropped data cannot be recovered.                   |
| An identifier arrives under an unknown registry           | The identifier is removed and the data may not be ingested properly.                                     |
| `$page_view` is used to record page history               | The events are dropped when the session closes.                                                          |
| An activity exceeds its size limit                        | Activities are limited to **8,000 characters**.                                                          |
| A custom event name is used where a predefined one exists | Built-in analytics, dashboards, and funnels ignore it.                                                   |

## What a data plan should contain

The plan is the single document that connects a use case to a field. For each entry:

| Column                     | Why it is there                                                                 |
| -------------------------- | ------------------------------------------------------------------------------- |
| **Use case**               | What the data is for. An entry with no use case is a candidate for removal.     |
| **Object and property**    | Where it lives in the model - `UserEvent`, `UserProfile`, a custom object.      |
| **Source**                 | Tag, tracking API, batch import, warehouse sync.                                |
| **Channel or compartment** | Which property or registration system it comes from.                            |
| **Predefined equivalent**  | The standard event name or property to use, if one exists.                      |
| **Indexed?**               | Whether it must be usable in a `WHERE` clause - this is the expensive decision. |
| **Retention**              | How long it must live, which becomes a cleaning rule.                           |
| **Owner**                  | Who to ask when the values look wrong.                                          |

For the per-channel event detail, the tracking plan is the operational half of this document - see [Define your tracking plan](/setup-and-monitor/collect-and-manage-your-data/collect-data-in-real-time/define-your-tracking-plan.md).

## Design decisions worth taking deliberately

{% stepper %}
{% step %}

### Index only what you query

Every indexed property costs index size and query performance. Fields such as first name or last name will never appear in a `WHERE` clause.
{% endstep %}

{% step %}

### Choose the string indexing mode per field

`text` for values you do not control and want broad, case-insensitive matching on; `keyword` for values you do control and want exact equality on. Getting this wrong is not visible in the schema - only in query results. See [Schema definition](/setup-and-monitor/collect-and-manage-your-data/design-your-data-model-and-schema/schema-definition.md).
{% endstep %}

{% step %}

### Prefer predefined names and properties

They are what the platform's automatic processing keys off.
{% endstep %}

{% step %}

### Decide retention before collecting

Expiration is stamped on write, and cleaning rules are not retroactive. See [Data retention](/setup-and-monitor/plan-and-integrate/set-up-the-environment/manage-data-retention.md).
{% endstep %}

{% step %}

### Decide the activity unique key convention

Datamart ID, UserPoint ID, timestamp, and unique key together identify an activity: identical values mean the new activity replaces the old one. If you generate keys yourself, agree the convention once. See [Collect data in real time](/setup-and-monitor/collect-and-manage-your-data/collect-data-in-real-time.md#the-activity-unique-key).
{% endstep %}
{% endstepper %}

## Checks to run after every change

{% tabs %}
{% tab title="Does it arrive?" %}
Track the number of events reaching each step of the ingestion pipeline: `JS_TAG`, `API`, `DOCUMENT_IMPORT`, `EVENT_RULES`, `ACTIVITY_ANALYZER`, `DOCUMENT_STORE`. A gap between two consecutive steps localises the problem - a rule that drops events, an analyzer that fails, a privacy wall that rejects.

See [Monitor data ingestion](/setup-and-monitor/operate-and-monitor/monitor-data-ingestion.md).
{% endtab %}

{% tab title="Is it indexed?" %}
Collection counts are retrieved every day, through OTQL queries, for each **indexed** collection. A collection is indexed if it is in your schema - volumes for non-indexed collections are not computed.

If a collection you expect shows no volume, the schema is the first place to look.

See [Monitor collection volume](/setup-and-monitor/operate-and-monitor/monitor-collection-volume.md).
{% endtab %}

{% tab title="Is it queryable as intended?" %}
Run one OTQL query per new indexed property, in the Query Tool, and check that it returns the population you expect for data collected after publication.

See [Query your datamart](/setup-and-monitor/operate-and-monitor/query-your-datamart.md).
{% endtab %}

{% tab title="Does one user look right?" %}
Open a known user and read the raw activities. This is the fastest way to find a property landing at the wrong path, a missing identifier, or an event rule doing something unexpected.

See [Explore a user profile](/setup-and-monitor/operate-and-monitor/explore-a-user-profile.md).
{% endtab %}
{% endtabs %}

## Two flags that indicate a modelling problem

The platform protects itself against UserPoints that accumulate too much, and both flags usually point at an identity or collection design issue rather than at unusual users.

**Hyper point** - a preventive flag set when too many identifiers are linked to one UserPoint. It is raised while merging UserPoints, when at least one of these conditions is met:

* More than 10 identifiers per registry ID in 2 days.
* More than 4 account IDs in 7 days.
* More than 4 email hashes in 7 days.
* More than 8 UserPoints merged into that UserPoint in the last 14 days.
* More than 100 user identifiers attached to that UserPoint.

Flagging keeps the UserPoint actionable in the platform.

**Quarantine** - a monthly job that flags any UserPoint with too many objects attached: more than 10,000 activities, or more than 5,000 segments. A quarantined UserPoint can still be looked up in Navigator, but nothing more is added to it. It can be freed by the same job once it drops below those limits, which typically happens when activities are deleted - through the API, or when cleaning rules apply.

If either flag appears on ordinary users, revisit [Resolve identities](/setup-and-monitor/collect-and-manage-your-data/design-your-data-model-and-schema/user-identity/resolve-identities.md) and your retention rules.

## Verify

* Every field in the schema traces back to a use case in the plan.
* Every use case in the plan traces to an indexed property, or is knowingly not queryable.
* Ingestion counts are consistent across pipeline steps for the channels you expect.
* Collection volumes are non-zero for every collection you index.
* A spot check on a real user shows the properties where the schema says they should be.

## Related pages

<table data-view="cards" data-search="false"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Define your tracking plan</strong></td><td>The per-channel, per-event half of the plan.</td><td><a href="/setup-and-monitor/collect-and-manage-your-data/collect-data-in-real-time/define-your-tracking-plan.md">Define your tracking plan</a></td></tr><tr><td><strong>Monitor data ingestion</strong></td><td>Watch events across the pipeline steps.</td><td><a href="/setup-and-monitor/operate-and-monitor/monitor-data-ingestion.md">Monitor data ingestion</a></td></tr><tr><td><strong>Schema definition</strong></td><td>The rules behind most of the silent failures above.</td><td><a href="/setup-and-monitor/collect-and-manage-your-data/design-your-data-model-and-schema/schema-definition.md">Schema definition</a></td></tr></tbody></table>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.mediarithmics.io/setup-and-monitor/collect-and-manage-your-data/design-your-data-model-and-schema/data-planning-and-quality.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.