Data planning and quality
Keep the data model, the collection, and the use cases in step - and catch the failures that are silent.
Your goal
Write down what you intend to collect and why, then check that what actually arrives matches it - because most data-quality failures on this platform are silent rather than noisy.
Expected outcome
A living data plan, and a small set of checks you run after every change to the schema, the tracking, or the privacy configuration.
Why quality problems here are quiet
Nothing on this list produces an error you would notice by accident. Knowing the list is most of the job.
A property is missing from the schema
The data is still stored, but not indexed - so it is invisible to queries and segments.
An indexed property is added later
Only new elements are indexed. WHERE queries run and return nothing for the data you already have.
! is used on a field that turns out to be null
The entire object is not indexed.
A Date field receives a timestamp
Queries on it throw an error, because there is no implicit conversion.
A privacy wall is enforced before consent exists
Incoming activities or profiles are dropped, and dropped data cannot be recovered.
An identifier arrives under an unknown registry
The identifier is removed and the data may not be ingested properly.
$page_view is used to record page history
The events are dropped when the session closes.
An activity exceeds its size limit
Activities are limited to 8,000 characters.
A custom event name is used where a predefined one exists
Built-in analytics, dashboards, and funnels ignore it.
What a data plan should contain
The plan is the single document that connects a use case to a field. For each entry:
Use case
What the data is for. An entry with no use case is a candidate for removal.
Object and property
Where it lives in the model - UserEvent, UserProfile, a custom object.
Source
Tag, tracking API, batch import, warehouse sync.
Channel or compartment
Which property or registration system it comes from.
Predefined equivalent
The standard event name or property to use, if one exists.
Indexed?
Whether it must be usable in a WHERE clause - this is the expensive decision.
Retention
How long it must live, which becomes a cleaning rule.
Owner
Who to ask when the values look wrong.
For the per-channel event detail, the tracking plan is the operational half of this document - see Define your tracking plan.
Design decisions worth taking deliberately
Choose the string indexing mode per field
text for values you do not control and want broad, case-insensitive matching on; keyword for values you do control and want exact equality on. Getting this wrong is not visible in the schema - only in query results. See Schema definition.
Decide retention before collecting
Expiration is stamped on write, and cleaning rules are not retroactive. See Data retention.
Decide the activity unique key convention
Datamart ID, UserPoint ID, timestamp, and unique key together identify an activity: identical values mean the new activity replaces the old one. If you generate keys yourself, agree the convention once. See Collect data in real time.
Checks to run after every change
Track the number of events reaching each step of the ingestion pipeline: JS_TAG, API, DOCUMENT_IMPORT, EVENT_RULES, ACTIVITY_ANALYZER, DOCUMENT_STORE. A gap between two consecutive steps localises the problem - a rule that drops events, an analyzer that fails, a privacy wall that rejects.
Collection counts are retrieved every day, through OTQL queries, for each indexed collection. A collection is indexed if it is in your schema - volumes for non-indexed collections are not computed.
If a collection you expect shows no volume, the schema is the first place to look.
Run one OTQL query per new indexed property, in the Query Tool, and check that it returns the population you expect for data collected after publication.
See Query your datamart.
Open a known user and read the raw activities. This is the fastest way to find a property landing at the wrong path, a missing identifier, or an event rule doing something unexpected.
Flags that indicate a modelling problem
The platform protects itself against UserPoints that accumulate too much, and both flags usually point at an identity or collection design issue rather than at unusual users.
Hyper point - a preventive flag set when too many identifiers are linked to one UserPoint. It is raised while merging UserPoints, when at least one of these conditions is met:
More than 10 identifiers per registry ID in 2 days.
More than 4 account IDs in 7 days.
More than 4 email hashes in 7 days.
More than 8 UserPoints merged into that UserPoint in the last 14 days.
More than 100 user identifiers attached to that UserPoint.
Flagging keeps the UserPoint actionable in the platform.
Quarantine - a monthly job that flags any UserPoint with too many objects attached: more than 10,000 activities, or more than 5,000 segments. A quarantined UserPoint can still be looked up in Navigator, but nothing more is added to it. It can be freed by the same job once it drops below those limits, which typically happens when activities are deleted - through the API, or when cleaning rules apply.
If either flag appears on ordinary users, revisit Resolve identities and your retention rules.
Verify
Every field in the schema traces back to a use case in the plan.
Every use case in the plan traces to an indexed property, or is knowingly not queryable.
Ingestion counts are consistent across pipeline steps for the channels you expect.
Collection volumes are non-zero for every collection you index.
A spot check on a real user shows the properties where the schema says they should be.
Related pages
Last updated
Was this helpful?