> For the complete documentation index, see [llms.txt](https://docs.mediarithmics.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.mediarithmics.io/setup-and-monitor/collect-and-manage-your-data/design-your-data-model-and-schema/schema-definition.md). # Schema definition Write, validate, and publish the schema that defines what your datamart indexes. ## Your goal Declare the objects and properties of your datamart, index the ones you will query on, and publish the result without breaking what already works. ## Expected outcome A `LIVE` schema on your datamart in which every property your use cases need is present, and the properties used in `WHERE` clauses are indexed. ## What a schema is A **schema** is applied to a datamart and **defines what mediarithmics should index and make available through queries**. It defines the model according to which your data is stored and queried. Each schema is a **GraphQL schema** defining an object tree index, which is what allows fast [OTQL](/setup-and-monitor/operate-and-monitor/query-your-datamart/otql.md) queries over the whole graph. Being strongly typed is what unlocks the tooling around it. {% hint style="success" %} `UserPoint` is the root element of any mediarithmics schema, and only one index can be created. This may change in future releases. {% endhint %} Schemas can evolve: a new schema replaces the previous one without losing historical data, because the platform records all events it receives in a separate database. Changing the schema changes what is *indexed*, not what is *stored*. {% hint style="info" %} Where the datamart reads from a cloud data warehouse, the objects you declare here are backed by tables you name in that warehouse. The language and the directives are the same; what changes is that each object needs a source table. See [Map your warehouse tables to a schema](/setup-and-monitor/collect-and-manage-your-data/design-your-data-model-and-schema/map-your-warehouse-tables-to-a-schema.md). {% endhint %} ## Reading your current schema You do not have to work blind: * The **advanced segment builder** shows a simplified view of your schema containing only the indexed fields, which you can expand and collapse. * The schema itself is defined and amended in your [datamart settings](/setup-and-monitor/plan-and-integrate/set-up-the-environment.md). * The [mics CLI](#manage-schemas-with-the-mics-cli) can download the current `LIVE` schema as a `.gql` file. ## The default objects A default schema declares every available mediarithmics object with its standard properties. Start from it and add or remove properties based on your needs and the data you ingest. {% hint style="info" %} The number of referenced properties has an impact on query performance. Keep only the properties you need - do not simply copy all the default ones. {% endhint %} ```graphql type UserPoint @TreeIndexRoot(index:"USER_INDEX"){ id: ID! creation_ts: Timestamp! @TreeIndex(index:"USER_INDEX") creation_date: Date! @Function(name:"ISODate", params:["creation_ts"]) # User identifiers accounts: [UserAccount!]! emails: [UserEmail!]! devices: [UserDevicePoint!]! # User content activities: [UserActivity!]! events: [UserEvent!]! profiles: [UserProfile!]! choices: [UserChoice!]! # Technical objects scenarios: [UserScenario!]! segments: [UserSegment!]! } ``` The identifier objects are covered in [User identity](/setup-and-monitor/collect-and-manage-your-data/design-your-data-model-and-schema/user-identity.md); `UserActivity`, `UserEvent`, `UserProfile`, `UserChoice`, and `UserSegment` are defined in [How mediarithmics works](/setup-and-monitor/understand-mediarithmics/how-mediarithmics-works.md#information-attached-to-a-userpoint). ## Types and syntax ### The `!` operator `!` marks an element as mandatory - it is expected not to be null. ```graphql type MyType { user_account_id: String # does not necessarily have a user account user_account_id: String! # has a user account events: [UserEvent!]! # has a list of events, no element can be null events: [UserEvent!] # may have no list, but no element can be null } ``` {% hint style="warning" %} If you add `!` to a field that turns out to have null values, **the entire object is not indexed**. It is hard to guarantee a field always has a value across every ingestion method, so **do not use this operator for fields other than the predefined ones**. {% endhint %} ### Basic types `ID`, `String`, `Int`, `Boolean`, `Timestamp`, and `Date` are available. `ID` is treated as a keyword string, but marks data that is not meaningful to a human because it is an identifier. ### Timestamps and dates Import objects carrying dates as `Timestamp`, and generate a `Date` from the timestamp with the [`ISODate`](#function) function when you want it displayed as a date and time in queries or exports. ```graphql type UserActivity { ts: Timestamp @TreeIndex(index:"USER_INDEX") date: Date @Function(name:"ISODate", params:["ts"]) } ``` {% hint style="warning" %} There is **no implicit conversion** between timestamps and dates. Declaring a field as `Date` when the incoming data is a timestamp makes queries on it throw an error. {% endhint %} Both types work with date operators in queries. Use only one `@TreeIndex` directive when deriving a date from a timestamp - index the timestamp, not the function result. It saves index space, and both types have the same query capabilities. ## Directives Directives are how you tell the platform what to do with a property. | Directive | What it does | | ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `@TreeIndexRoot(index:"USER_INDEX")` | Marks the root element of the object tree index. Always `USER_INDEX`; multiple indexes are not currently supported. | | `@TreeIndex(index:"USER_INDEX")` | Makes a field available in `WHERE` clauses and in aggregation operations. Fields without it can still be retrieved in `SELECT`. | | `@Property(path:...)` / `@Property(paths:[...])` | Changes the path a property reads from. Several paths can be given: if the first is empty, the next is used. | | `@Mirror(object_type:..., filter:...)` | Creates a custom type based on a predefined one, optionally filtered. | | `@Function(name:..., params:[...])` | Declares a calculated field using a predefined function. | | `@ReferenceTable(type:"CORE_OBJECT", model_type:...)` | Gives query builders an autocomplete on names instead of IDs, for channels, compartments, and segments. | | `@EdgeAvailability` | Marks a property as usable in [Edge](/setup-and-monitor/plan-and-integrate/set-up-advanced-features/edge.md) segment queries. | | `@ComputedField(technical_name:...)` | Marks a property as calculated by a [computed field](/setup-and-monitor/collect-and-manage-your-data/design-your-data-model-and-schema/computed-fields.md) function. | | `@UpdateStrategyKey` | Enables targeted updates on objects that have an ID property inside a `UserProfile`. | | `@UpdateValueObject` | Enables targeted updates on objects **without** an ID property inside a `UserProfile`. | ### @TreeIndex, and how strings are indexed Do not mark every field with `@TreeIndex`. Fields such as first name or last name will never be used in a `WHERE` clause and would only make the index larger. {% hint style="warning" %} `@TreeIndex` is **mandatory** on some default properties. They already carry it in the default schema - removing it means your schema will not validate. {% endhint %} When indexing a `String`, choose how it is indexed: {% tabs %} {% tab title="text" %} ```graphql mystring: String @TreeIndex(index:"USER_INDEX", data_type: "text") ``` The value is treated as **a set of words**. `The QUICK brown fox JuMpS, over the Lazy doG.` becomes `the`, `quick`, `brown`, `fox`, `jumps`, `over`, `lazy`, `dog.` Two transformations happen: words are lowercased - so **all string operators are case-insensitive** on a `text` field - and the string is split, with the splitting characters removed. Characters that commonly trigger a split include space, `-`, `"`, `'`, `,`, `;`, `?`, `!`, and `/`. Characters that do **not** trigger a split include `.`, `_`, and apostrophes. Use `text` for full sentences such as page titles, URLs, lists of keywords, and similar content - generally when you do not fully control the value and want broad queries on it. {% endtab %} {% tab title="keyword" %} ```graphql secondstring: String @TreeIndex(index:"USER_INDEX", data_type: "keyword") ``` The value is treated as **a single word**, with no transformation applied. Use `keyword` for single values, IDs passed as text such as UUIDs, product IDs or category IDs, and any field whose values you already know - for instance because they come from a taxonomy. Generally: when you control the value and want exact equality in queries. {% endtab %} {% endtabs %} ### @Property By default the path of a property is its own name. `@Property` changes that - most usefully to lift properties out of a nested `$properties` object so they are easier to query. ```graphql type UserEvent { id: ID! ts: Timestamp! name: String! url: String @Property(path:"$properties.$url") referrer: String @Property(path:"$properties.$referrer") products: [Product] @Property(path:"$properties.$items") } type Product { id: String @TreeIndex(index:"USER_INDEX") @Property(path:"$id") name: String @TreeIndex(index:"USER_INDEX") @Property(path:"$name") } ``` Properties in the default schema already redefine their path - `creation_ts` on `UserPoint` points at `$creation_ts` without you declaring it. Properties that do not exist in the default schema, such as those of a custom `Product` object, do need the directive. Two more capabilities: * **Several paths.** `channel_id: String @Property(paths:["$site_id", "$app_id"])` - a user activity's channel ID is the site ID or the app ID depending on context. * **The `[parent]` token**, to go up the object tree: `@Property(path:"[parent].[parent].$origin.$creative_id")`. ### @Mirror ```graphql # ArticleView objects are UserEvents whose name is "navigation.article" type ArticleView @Mirror(object_type:"UserEvent", filter:"name == \"name.value\""){} ``` Used with a filter, `@Mirror` gives you typed collections per event kind: ```graphql type UserPoint @TreeIndexRoot(index:"USER_INDEX"){ basketviews: [BasketView] productviews: [ProductView] } type BasketView @Mirror(object_type:"UserEvent", filter:"name == \"$basket_view\""){} type ProductView @Mirror(object_type:"UserEvent", filter:"name == \"$page_view\""){} ``` Renaming a type without a filter is possible but should be avoided - it adds no capability. ### @Function | Function | What it does | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ISODate` | Creates a `Date` from a `Timestamp`: `creation_date: Date! @Function(name:"ISODate", params:["creation_ts"])`. | | `ThirdPartyCookieMappings` | Retrieves third-party cookie mappings for a device point. It works on device points carrying a device technical ID of type `MUM_ID`, translates that into a `vector_id`, and returns the attached partners' third-party cookies. | | `DeviceInfo` | **Legacy.** Extracts device information for an agent identifier. Only for datamarts still referencing the legacy `UserAgent` type; on datamarts using `UserDevicePoint`, use `ThirdPartyCookieMappings`. | ## Publishing a schema A schema has a status - `DRAFT`, `LIVE`, or `ARCHIVED` - and only one draft can exist at a time. ```mermaid stateDiagram-v2 [*] --> DRAFT: create, usually by cloning LIVE DRAFT --> DRAFT: upload content, validate DRAFT --> LIVE: publish LIVE --> ARCHIVED: superseded by a new publication ``` {% stepper %} {% step %} ### Create a draft, by cloning ``` POST /v1/datamarts/:datamartId/graphdb_runtime_schemas/:schemaId/clone ``` Cloning the current `LIVE` version is the **preferred method**: it keeps all the settings of the previous version, such as cluster versions and index sizes. Creating a draft without cloning is possible with `POST /v1/datamarts/:datamartId/graphdb_runtime_schemas`, but then the initial settings are not carried over - only do this if you know the settings you need and can set them before publishing. Either call fails if a draft already exists. {% endstep %} {% step %} ### Upload the content ``` PUT /v1/datamarts/:datamartId/graphdb_runtime_schemas/:schemaId/text ``` The schema goes in the raw body of the request. {% endstep %} {% step %} ### Validate ``` POST /v1/datamarts/:datamartId/graphdb_runtime_schemas/:schemaId/validation ``` Validation tells you whether the uploaded schema is valid and returns the errors if it is not - for instance a root type missing a required indexed field. {% endstep %} {% step %} ### Publish ``` POST /v1/datamarts/:datamartId/graphdb_runtime_schemas/:schemaId/publication ``` The selected schema goes `LIVE` and the current `LIVE` schema is archived. Publishing an unvalidated schema returns an error. {% endstep %} {% endstepper %} Reading schemas: `GET .../graphdb_runtime_schemas` returns all schemas and their status - archived ones included, so you keep a history - `GET .../graphdb_runtime_schemas/:schemaId` returns one, and `GET .../graphdb_runtime_schemas/:schemaId/text` returns its content. ## What happens when you change a schema This is the part worth reading twice. | Change | Effect | | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Add any property** | Immediately usable in the `SELECT` part of your OTQL queries, and with any operator that does not require indexing. | | **Add an indexed property** - a new property with `@TreeIndex`, or `@TreeIndex` added to an existing one | Only **new** elements entering the datamart are indexed. `WHERE` queries and operators needing indexing run, but return nothing for elements already stored. | | **Remove an indexed property** | You instantly stop being able to run `WHERE` queries and indexed operators on it. | {% hint style="success" %} You can ask your mediarithmics contact to start a complete reindexing of your datamart if you need past data to be covered by a new indexed property. {% endhint %} ## Manage schemas with the mics CLI The mics CLI shortens the workflow above. {% tabs %} {% tab title="Download" %} Retrieves the `LIVE` schema, or a specified one, and saves it as `schema--.gql`. ``` $ mics-cli schema:fetch DATAMARTID [SCHEMAID] ARGUMENTS DATAMARTID the ID of the datamart SCHEMAID [default: LIVE] the ID of the schema, or LIVE to get the live schema OPTIONS --stdout output the content of the file instead of saving it in a file ``` {% endtab %} {% tab title="Update" %} Creates a draft, uploads a file, validates it, and publishes - in one command. It handles status and cloning for you: * If a draft already exists, it is updated before publishing. * The latest `LIVE` schema is cloned if available, unless `--noClone` is set. * `--skipPublish` leaves the schema as a `DRAFT`. {% endtab %} {% endtabs %} ## Decorators: changing how the schema reads Decorators let you rename fields for display, or hide them from users, **without altering the technical schema**. They affect the Advanced Segment Builder and the Query Tool. They are defined as a CSV file with these columns: | Header | What it holds | | ------------- | ------------------------------------------------------------------- | | `OBJECT_NAME` | The object where the property is located. | | `FIELD_NAME` | The original technical name of the property. | | `HIDDEN` | `true` or `false` - whether the field is hidden in the interface. | | `LABEL` | The user-friendly name shown in the Segment Builder and Query Tool. | | `HELP_TEXT` | A short description shown as a tooltip. | | `LOCALE` | Currently must be `en-US`. | ``` OBJECT_NAME,FIELD_NAME,HIDDEN,LABEL,HELP_TEXT,LOCALE UserPoint,id,false,User ID,Unique identifier for the user point,en-US UserPoint,technical_hash,true,,,en-US UserEvent,subcategory,false,Subcategory,The specific sub-category of the event,en-US ``` Use a comma as separator, and double quotes around strings containing commas. {% hint style="warning" %} Normalise strings by removing accents. Characters such as `é`, `è`, or `à` are not supported in the CSV and must be replaced by their unaccented equivalents. {% endhint %} Manage them in **Settings > Datamart > Object View Configuration**, where you can upload new decorators - which overwrites the existing ones - download a blank template, download the active file, or delete it, in which case the schema reverts to raw technical names and hidden fields become visible again. Or by API: ```bash # Retrieve the current decorators curl -H "Authorization:$MICS_API_TOKEN" -X GET --location \ "https://api.mediarithmics.com/v1/datamarts/$DATAMART_ID/graphdb_runtime_schemas/$SCHEMA_ID/schema_decorators" # Replace them curl -H "Authorization:$MICS_API_TOKEN" -X PUT --location \ "https://api.mediarithmics.com/v1/datamarts/$DATAMART_ID/graphdb_runtime_schemas/$SCHEMA_ID/schema_decorators" \ --data-raw "$(cat path/to/schema_decorators.csv)" ``` ## Best practices * **Index the timestamp, not the derived date.** Put `@TreeIndex` on `creation_ts`, not on the `ISODate` result. * **Do not duplicate `UserEvent`.** Declaring events both on the `UserPoint` and inside `UserActivity` is only justified in specific scenarios - for instance using frequency directives on events while also querying several events within a single activity. Otherwise pick one. * **Index deliberately.** Every indexed property costs query performance and index size. * **Avoid `!` outside the predefined properties**, or one null value silently drops the whole object from the index. ## Verify * Validation returns no schema errors before you publish. * Every property used in a `WHERE` clause by your use cases carries `@TreeIndex`. * A test OTQL query on each new indexed property returns results for data arriving after publication. * If you need history on a new indexed property, a reindexing has been requested. ## Related pages
Standard eventsThe predefined names and properties to model against.Standard events
Computed fieldsDeclaring a calculated field in the schema.Computed fields
OTQLThe query language the index serves.OTQL
--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.mediarithmics.io/setup-and-monitor/collect-and-manage-your-data/design-your-data-model-and-schema/schema-definition.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.