For the complete documentation index, see llms.txt. This page is also available as Markdown.

Data planning and quality

Keep the data model, the collection, and the use cases in step - and catch the failures that are silent.

Your goal

Write down what you intend to collect and why, then check that what actually arrives matches it - because most data-quality failures on this platform are silent rather than noisy.

Expected outcome

A living data plan, and a small set of checks you run after every change to the schema, the tracking, or the privacy configuration.

Why quality problems here are quiet

Nothing on this list produces an error you would notice by accident. Knowing the list is most of the job.

Silent failure
What actually happens

A property is missing from the schema

The data is still stored, but not indexed - so it is invisible to queries and segments.

An indexed property is added later

Only new elements are indexed. WHERE queries run and return nothing for the data you already have.

! is used on a field that turns out to be null

The entire object is not indexed.

A Date field receives a timestamp

Queries on it throw an error, because there is no implicit conversion.

A privacy wall is enforced before consent exists

Incoming activities or profiles are dropped, and dropped data cannot be recovered.

An identifier arrives under an unknown registry

The identifier is removed and the data may not be ingested properly.

$page_view is used to record page history

The events are dropped when the session closes.

An activity exceeds its size limit

Activities are limited to 8,000 characters.

A custom event name is used where a predefined one exists

Built-in analytics, dashboards, and funnels ignore it.

What a data plan should contain

The plan is the single document that connects a use case to a field. For each entry:

Column
Why it is there

Use case

What the data is for. An entry with no use case is a candidate for removal.

Object and property

Where it lives in the model - UserEvent, UserProfile, a custom object.

Source

Tag, tracking API, batch import, warehouse sync.

Channel or compartment

Which property or registration system it comes from.

Predefined equivalent

The standard event name or property to use, if one exists.

Indexed?

Whether it must be usable in a WHERE clause - this is the expensive decision.

Retention

How long it must live, which becomes a cleaning rule.

Owner

Who to ask when the values look wrong.

For the per-channel event detail, the tracking plan is the operational half of this document - see Define your tracking plan.

Design decisions worth taking deliberately

1

Index only what you query

Every indexed property costs index size and query performance. Fields such as first name or last name will never appear in a WHERE clause.

2

Choose the string indexing mode per field

text for values you do not control and want broad, case-insensitive matching on; keyword for values you do control and want exact equality on. Getting this wrong is not visible in the schema - only in query results. See Schema definition.

3

Prefer predefined names and properties

They are what the platform's automatic processing keys off.

4

Decide retention before collecting

Expiration is stamped on write, and cleaning rules are not retroactive. See Data retention.

5

Decide the activity unique key convention

Datamart ID, UserPoint ID, timestamp, and unique key together identify an activity: identical values mean the new activity replaces the old one. If you generate keys yourself, agree the convention once. See Collect data in real time.

Checks to run after every change

Track the number of events reaching each step of the ingestion pipeline: JS_TAG, API, DOCUMENT_IMPORT, EVENT_RULES, ACTIVITY_ANALYZER, DOCUMENT_STORE. A gap between two consecutive steps localises the problem - a rule that drops events, an analyzer that fails, a privacy wall that rejects.

Collection counts are retrieved every day, through OTQL queries, for each indexed collection. A collection is indexed if it is in your schema - volumes for non-indexed collections are not computed.

If a collection you expect shows no volume, the schema is the first place to look.

Run one OTQL query per new indexed property, in the Query Tool, and check that it returns the population you expect for data collected after publication.

Open a known user and read the raw activities. This is the fastest way to find a property landing at the wrong path, a missing identifier, or an event rule doing something unexpected.

Flags that indicate a modelling problem

The platform protects itself against UserPoints that accumulate too much, and both flags usually point at an identity or collection design issue rather than at unusual users.

Hyper point - a preventive flag set when too many identifiers are linked to one UserPoint. It is raised while merging UserPoints, when at least one of these conditions is met:

  • More than 10 identifiers per registry ID in 2 days.

  • More than 4 account IDs in 7 days.

  • More than 4 email hashes in 7 days.

  • More than 8 UserPoints merged into that UserPoint in the last 14 days.

  • More than 100 user identifiers attached to that UserPoint.

Flagging keeps the UserPoint actionable in the platform.

Quarantine - a monthly job that flags any UserPoint with too many objects attached: more than 10,000 activities, or more than 5,000 segments. A quarantined UserPoint can still be looked up in Navigator, but nothing more is added to it. It can be freed by the same job once it drops below those limits, which typically happens when activities are deleted - through the API, or when cleaning rules apply.

If either flag appears on ordinary users, revisit Resolve identities and your retention rules.

Verify

  • Every field in the schema traces back to a use case in the plan.

  • Every use case in the plan traces to an indexed property, or is knowingly not queryable.

  • Ingestion counts are consistent across pipeline steps for the channels you expect.

  • Collection volumes are non-zero for every collection you index.

  • A spot check on a real user shows the properties where the schema says they should be.

Last updated

Was this helpful?