> For the complete documentation index, see [llms.txt](https://docs.mediarithmics.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.mediarithmics.io/setup-and-monitor/collect-and-manage-your-data/collect-data-in-real-time.md). # Collect data in real time Send events as they happen, and understand what the platform does to them before they are stored. ## Your goal Track what your users do on your digital touchpoints as it happens, and know exactly what the platform does to each event on its way into the datamart. ## Expected outcome Events arriving from every property in scope, aggregated into activities, correctly attributed to a channel and a user, and visible in queries. {% hint style="danger" %} If your need is to **mass import** activities, users, or CRM profiles, this is the wrong tool. See [Import data in batch](/setup-and-monitor/collect-and-manage-your-data/import-data-in-batch.md). {% endhint %} ## What you send, and what gets stored You send **user events**. The platform aggregates and encapsulates them into **user activities**. Some activities hold a single event, others hold many - when you send hits from the JS tag on a website, all the events of a session are aggregated into one activity. * A **UserActivity** is a collection of events done by a single user in a given period of time. A website activity is a session of no more than 30 minutes on a given site. An activity is also linked to its referrer, so a user who switches referrer and comes back to the same site starts a new activity. * A **UserEvent** is the most granular level of interaction, defined by a name and a set of custom properties. Events can be given any name and any properties, but [predefined names and properties](/setup-and-monitor/collect-and-manage-your-data/design-your-data-model-and-schema/standard-events.md) trigger specific treatment. For each user, activities are ordered in a timeline. Activities and events can be indexed and queried if their properties fit the object tree schema. ## The integrations
Web trackingThe JS tag on your websites, including AMP.Web tracking
App trackingMobile and CTV applications.App tracking
Server-side eventsThe tracking API, for server-to-server collection.Server-side events
Ads exposure trackingAd views and clicks from display campaigns.Ads exposure tracking
Email trackingEmail views and clicks.Email tracking
ConversionsRecording that a goal was met.Conversions
## The processing pipeline Sending an event puts it into the **User Activity Processing Pipeline**, where the platform does work on your behalf. Knowing the order matters, because it determines what can still change your data and what cannot. ```mermaid flowchart TD IN["Events sent
tag, app, API, import"] A["1. Activity created
at session close, or every minute if live"] B["2. Event rules
edit events, add events"] C["3. Activity analyzer
plugin transforms the activity"] D["4. Query-based conversions detected
pixel conversions skip this"] E["5. Automation triggers evaluated
'React to an event'"] F["6. UserChoice management
compatibility with the user's choices"] G[("Activity stored")] IN --> A --> B --> C --> D --> E --> F --> G ``` 1. At session closing - or every minute for `live` events - a user activity is created. 2. [**Event rules**](/setup-and-monitor/collect-and-manage-your-data/collect-data-in-real-time/event-rules.md) run. They edit the events in the activity, and add new ones based on their shape. 3. The [**activity analyzer**](/setup-and-monitor/collect-and-manage-your-data/collect-data-in-real-time/activity-analyzers.md) is applied. It is a plugin that runs your code to transform each activity. 4. **Query-based conversions** are detected. Conversions relying on pixels skip this step. 5. **Automation triggers** of type "React to an event" are evaluated. If the activity matches, the UserPoint enters the scenario. 6. [**UserChoice** management](/setup-and-monitor/ensure-privacy-and-compliance/manage-consent-and-processing-activities.md) ensures the data you ingest is compatible with the user's recorded choices. The activity is then stored. {% hint style="info" %} Everything before step 6 can modify your data. Anything you need *unmodified* - Edge-available properties, for instance - must not pass through an event rule or an analyzer. See [Edge](/setup-and-monitor/plan-and-integrate/set-up-advanced-features/edge.md). {% endhint %} ## Session aggregation A visit registered as `IN_SESSION` goes through the **session aggregation** step, where visits are aggregated into sessions based on the identifiers provided. A session is closed: * after **30 minutes of inactivity**, or * when a new event is recorded with a referrer from a **different domain** than the one previously recorded. Visits registered as `LIVE` do not create sessions. ## The UserActivity object The properties below are the ones you send. Base property names all begin with `$` - if you avoid that character in your own property names, you can always tell them apart. | Field | Type | Description | | --------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------- | | `$ts` | Long | The timestamp of the activity, in Unix epoch **milliseconds**. For a session it should be the start date. | | `$type` | String enum | The activity type - see the table below. | | `$session_status` | String enum | `NO_SESSION`, `IN_SESSION`, or `CLOSED_SESSION`. Updated automatically by the platform. | | `$ttl` | Integer | Time to live in minutes for the storage of this activity. `0` means no expiration. | | `$user_agent_info` | Object | Optional. Information about the device. | | `$user_identifiers` | List | All identifiers relative to the activity - several of each type are allowed. **Use this rather than the legacy properties.** | | `$origin` | Object | The activity origin. | | `$location` | Object | The activity location. | | `$events` | List | The user events attached to this activity. | | `$unique_key` | String | The unique key of the activity, formatted as a UUID v1. If empty, the platform generates one. | | *any custom property* | Any | The value of a custom property. | The legacy identifier properties - `$user_agent_id`, `$user_account_id` with `$compartment_id`, and `$email_hash` - still exist but should be replaced by `$user_identifiers`. Use one approach or the other, not both. {% hint style="warning" %} Two constraints worth designing around: * **Always prefer predefined properties** to custom ones when they exist. The platform automates a lot of actions based on them. * **Activities are limited to 8,000 characters.** {% endhint %} ### Activity types | `$type` | When it is used | | ----------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `SITE_VISIT` | Activities on a channel of type site. Additional properties: `$site_id` (the channel), `$session_duration` in seconds. | | `APP_VISIT` | Activities on a channel of type app. Additional properties: `$app_id`, `$session_duration` (optional). | | `CTV_VISIT` | Activities on a channel of type CTV. Additional properties: `$ctv_id`, `$session_duration` (optional). | | `DISPLAY_AD` | Activities involving ad view and ad click events. See [Ads exposure tracking](/setup-and-monitor/collect-and-manage-your-data/collect-data-in-real-time/ads-exposure-tracking.md). | | `EMAIL` | Users reading a mail or clicking a link inside it. See [Email tracking](/setup-and-monitor/collect-and-manage-your-data/collect-data-in-real-time/email-tracking.md). | | `TOUCH` | Other activities. | | `USER_SCENARIO_START` / `USER_SCENARIO_STOP` / `USER_SCENARIO_NODE_ENTER` / `USER_SCENARIO_NODE_EXIT` | Automation-specific activity types, emitted as a UserPoint moves through a scenario. | The channel referenced by `$site_id`, `$app_id`, or `$ctv_id` must exist - see [Channels](/setup-and-monitor/plan-and-integrate/set-up-the-environment/set-up-channels.md). ### Device information `$user_agent_info` describes the device: | Property | Type | Description | | ------------------ | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `$initial_ts` | Timestamp | When the device was seen. | | `$form_factor` | Enum | `PERSONAL_COMPUTER`, `SMART_TV`, `GAME_CONSOLE`, `SMARTPHONE`, `TABLET`, `WEARABLE_COMPUTER`, `OTHER`. | | `$os_family` | Enum | `WINDOWS`, `MAC_OS`, `LINUX`, `ANDROID`, `IOS`, `TIZEN`, `WEB_OS`, `ANDROID_TV`, `GOOGLE_TV`, `TV_OS`, `FIRE_TV`, `ROKU_TV`, `TITAN_OS`, `OTHER`. | | `$browser_family` | Enum | `OTHER`, `CHROME`, `IE`, `FIREFOX`, `SAFARI`, `OPERA`, `STOCK_ANDROID`, `BOT`, `EMAIL_CLIENT`, `MICROSOFT_EDGE`. | | `$browser_version` | String | The version of the browser. | | `$brand` | String | The brand of the device, for example "Apple". | | `$model` | String | The model, for example "Iphone 14". | | `$os_version` | String | The version of the operating system. | | `$carrier` | String | The service provider ensuring connectivity, for example "AT\&T". | | `$raw_value` | String | The raw user agent value. | | `$agent_type` | Enum | `WEB_BROWSER`, `MOBILE_APP`, or `CTV_APP`. | ### Activity location | Field | Type | Description | | ------------- | ---------------------- | ------------------------------------------ | | `$source` | String, optional | The location source: `IP`, `GPS`, `OTHER`. | | `$country` | String, optional | The country's name. | | `$region` | String, optional | The region's name. | | `$iso_region` | String, optional | The region ISO code (ISO 3166-2). | | `$city` | String, optional | The city's name. | | `$iso_city` | String, optional | The city code (UN/LOCODE). | | `$zip_code` | String, optional | The zip code. | | `$latlon` | Array\[Double, Double] | Latitude first, longitude second. | ## Activity origin The **activity origin** is the last digital channel leading to a user interaction. It is key information for analysing the results and performance of marketing activities. The interaction can be a **touch** - the user views a banner or an email - or a **visit** - the user visits a site or an app. In both cases the `$origin` object of the activity captures the originating channel. {% hint style="info" %} A user activity can only have **one** origin. If a user comes back from a different origin during a live session, the current session is closed and a new session is opened with the second origin. {% endhint %} ### How the origin is calculated | Activity | How the origin is determined | | ----------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | A visit on a site or an app | From the analysis of the referrer and the query parameters in the destination URL, such as the UTM parameters used by Google Analytics. | | Touch events from campaigns delivered by the platform | Calculated automatically from the campaign information. | | Touch events - pixel events in emails or banners | Calculated from the predefined properties provided in the event. | | An activity inserted through the API | The origin fields can be filled directly with the relevant data. | ### The origin object | Field | Description | | ------------------------------------------------------------ | ------------------------------------------------------------------------------ | | `$ts` | | | `$channel` | The communication channel, for example `cpc`, `newsletter`, `banner`, `video`. | | `$source` | The source of the traffic, for example `google.com`. | | `$campaign_name`, `$campaign_technical_name`, `$campaign_id` | The campaign. | | `$sub_campaign_technical_name`, `$sub_campaign_id` | The sub-campaign, equivalent to the ad group. | | `$message_id`, `$message_technical_name` | The message. | | `$keywords` | The keywords used in the search, for example `sport+shoes`. | | `$creative_name`, `$creative_technical_name`, `$creative_id` | The creative. | | `$engagement_content_id` | | | `$social_network` | The social network. | | `$referral_path` | The URL of the referral. | | `$log_id` | A custom unique identifier for the activity origin. | | `$gclid` | The unique identifier for a Google Ads click. | ### Detection from UTM parameters | Origin field | URL parameter | Example | | ---------------- | -------------- | ----------------------------------- | | `$channel` | `utm_medium` | `utm_medium = email` | | `$source` | `utm_source` | `utm_source = base loyalty program` | | `$campaign_name` | `utm_campaign` | `utm_campaign = back to school` | | `$creative_name` | `utm_content` | `utm_content = template 1` | | `$keywords` | `utm_term` | `utm_term = sport+shoes` | | `$gclid` | `gclid` | Google click identifier | ### Detection from the AT Internet `xtor` parameter The `xtor` parameter, structured as `xtor=A-B-C-D-E-F-G-H`, is analysed automatically: | Origin field | `xtor` field | Example | | ---------------- | ------------ | --------------------------------------------------- | | `$source` | A | `EPR` when `xtor=EPR-14234` | | `$campaign_name` | B | `7880` when `xtor=AD-7880` | | `$creative_name` | C | The creative version when `xtor=AD-3030-` | The source can also imply a marketing channel: | `xtor` source | Inferred `$channel` | | ------------------- | ------------------- | | `EPR`, `EREC`, `ES` | `$email` | | `AD` | `$rtb_display` | | `AL` | `$affiliation` | ### Declaring an origin from event properties When the event is declared through a tag, predefined event properties can declare the origin: | Origin field | Event property | Example | | ------------------------------ | ----------------- | ------------------------------------------------- | | `$source` | `$source` | `$source=crm_database` | | `$campaign_name` | `$campaign_name` | `$campaign_name=back to school` | | `$campaign_technical_name` | `$ctn` | `$ctn=DISPLAY-BACK-TO-SCHOOL` | | `$campaign_id` | `$caid` | `$caid=8989` | | `$sub_campaign_technical_name` | `$scatn` | `$scatn=STRATEGY-1` - the ad group technical name | | `$sub_campaign_id` | `$scaid` | `$scaid=8782` - the ad group ID | | `$creative_name` | `$creative_name` | | | `$creative_technical_name` | `$crtn` | `$crtn=special-banner` | | `$keywords` | `$keywords` | `$keywords = sport+shoes` | | `$social_network` | `$social_network` | | | `$referral_path` | `$referral_path` | | | `$log_id` | `$log_id` | A unique custom identifier | | `$gclid` | `$gclid` | Google click identifier | ## The activity unique key Each unique activity is stored under four accesses, in this order: **datamart ID, UserPoint ID, timestamp, and unique key**. When those four values are identical between two activities, **the new one cancels and replaces the old one**. If any one of them differs, a new activity is created. This is the mechanism to use for idempotent sends and for corrections. The unique key is a UUID. It can be generated by the platform or by you: | Source | Who generates it | | ---------------------- | ------------------------------------------- | | The mediarithmics tag | The platform. | | The mediarithmics APIs | You, or the platform if you leave it empty. | | Document import | You, or the platform if you leave it empty. | The best practice when you generate it is **UUID v1**, because it is not random: it is calculated from a timestamp and a unique string. Choose a key property - an order ID, a unique ID generated on your side - or a concatenation of properties that makes the string unique, and use the activity's own timestamp. ## In this section
Define your tracking planDecide what you send before you send it.Define your tracking plan
Event rulesReshape events as they arrive.Event rules
Activity analyzersRun your own code on each activity.Activity analyzers
Activities analyticsWhat the platform builds from your activities.Activities analytics
--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.mediarithmics.io/setup-and-monitor/collect-and-manage-your-data/collect-data-in-real-time.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.