> For the complete documentation index, see [llms.txt](https://docs.mediarithmics.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.mediarithmics.io/setup-and-monitor/operate-and-monitor/monitor-data-ingestion.md).
# Monitor data ingestion
Track events through each step of the ingestion pipeline to localise where they are lost.
## Your goal
Know how many events reach each step of the ingestion pipeline, so that when data goes missing you can say **where** it went missing rather than guessing.
## Expected outcome
A report you can run per datamart and per step, and a habit of checking it after every change to tracking, rules, analyzers, or privacy configuration.
## Why the steps matter
When activities are collected, the platform tracks the number of events that go through **each step** of the data pipeline. A gap between two consecutive steps localises the problem immediately.
```mermaid
flowchart LR
A["JS_TAG"] --> R["EVENT_RULES"]
B["API"] --> R
C["DOCUMENT_IMPORT"] --> R
R --> AA["ACTIVITY_ANALYZER"]
AA --> DS["DOCUMENT_STORE"]
```
| Step | What it measures |
| ------------------- | ---------------------------------------------------------------------------------------------------------- |
| `JS_TAG` | Events sent by the mediarithmics JavaScript tag. |
| `API` | Events sent through the ingestion API, `api.mediarithmics.com/v1/datamarts/{datamart_id}/user_activities`. |
| `DOCUMENT_IMPORT` | Events sent through `USER_ACTIVITY` imports with the document import feature. |
| `EVENT_RULES` | Events that **match** configured rules. |
| `ACTIVITY_ANALYZER` | Events that pass through activity analyzer plugins. |
| `DOCUMENT_STORE` | Events that are **stored** in the platform. |
{% hint style="info" %}
`DOCUMENT_STORE` is the number that matters for "did my data arrive". The steps before it tell you why it does not match what you sent.
{% endhint %}
## Read the report
The ingestion monitoring endpoint is a mediarithmics **data cube**, and works like the [activities analytics](/setup-and-monitor/operate-and-monitor/query-your-datamart/data-cube-activities-analytics.md) API.
```
POST https://api.mediarithmics.com/v1/platform_monitoring/data_ingestion
```
| Field | Type | Description |
| -------------------------- | ------ | -------------------------------------------------------------------- |
| `metrics` | Array | The metrics to retrieve. **Required.** |
| `dimensions` | Array | The dimensions to group metrics by. **Required.** |
| `dimension_filter_clauses` | Object | Filters applied on dimensions before calculating the metric. |
| `date_ranges` | Array | The periods to analyse - each with a `start_date` and an `end_date`. |
```json
{
"date_ranges": [
{
"start_date": "2021-10-10T00:00:00",
"end_date": "2021-10-25T23:59:59"
}
],
"dimensions": [
{ "name": "date_time" },
{ "name": "datamart_id" },
{ "name": "pipeline_step" }
],
"metrics": [
{ "expression": "event_count" }
]
}
```
{% hint style="success" %}
Relative date formats work: `"start_date": "now-7d/d"` is easier to script than a fixed date, and is what you want in a recurring check.
{% endhint %}
Group by `pipeline_step` to get the funnel above. Add `datamart_id` when your organisation has several.
## Retention
{% hint style="info" %}
These aggregated statistics are kept for **3 months**.
{% endhint %}
A shorter window than collection volumes or usage reports, so a year-on-year comparison of ingestion is not possible from this API - extract it if you need one. See [Data retention](/setup-and-monitor/plan-and-integrate/set-up-the-environment/manage-data-retention.md).
## Diagnosing a gap
| Gap | Likely cause |
| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Nothing at `JS_TAG` | The tag is not firing, the site token is wrong, or consent is never granted. See [Web tracking](/setup-and-monitor/collect-and-manage-your-data/collect-data-in-real-time/web-tracking.md). |
| Nothing at `API` | Authentication, or the payload is rejected. See [Server-side events](/setup-and-monitor/collect-and-manage-your-data/collect-data-in-real-time/server-side-events.md). |
| `EVENT_RULES` at zero when you expect matches | The rule does not match - a `$page_view`-only rule applied to differently named events, for instance. See [Event rules](/setup-and-monitor/collect-and-manage-your-data/collect-data-in-real-time/event-rules.md). |
| `ACTIVITY_ANALYZER` well below the step before | The analyzer is failing or timing out. See [Build an activity analyzer](/setup-and-monitor/extend-the-platform/build-an-activity-analyzer.md). |
| `DOCUMENT_STORE` well below `ACTIVITY_ANALYZER` | The analyzer is removing activities, or a **privacy wall** is dropping them because the users have no compatible choice. See [Manage consent & Processing activities](/setup-and-monitor/ensure-privacy-and-compliance/manage-consent-and-processing-activities.md). |
| Everything looks right but queries return nothing | The data arrived but is **not indexed** - a schema problem, not an ingestion problem. See [Monitor collection volume](/setup-and-monitor/operate-and-monitor/monitor-collection-volume.md). |
{% hint style="danger" %}
The single most common cause of a sharp `DOCUMENT_STORE` drop with healthy upstream steps is a privacy wall enforced before user choices existed. Check that before anything else - and remember dropped data cannot be recovered.
{% endhint %}
## When to check it
* After deploying or changing the tag.
* After creating or editing an event rule or an activity analyzer.
* **Before and after** enforcing a privacy wall.
* After a schema publication, together with collection volumes.
* On a schedule, so a slow decline is visible before someone reports it.
## Related pages
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://docs.mediarithmics.io/setup-and-monitor/operate-and-monitor/monitor-data-ingestion.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.