Define the observations a monitoring report expects

Start with the intended measurement window

An expected observation contract specifies the model identity, version, period keys and fields required for a report. Write the expected dates independently of whatever happened to arrive. Otherwise a missing month disappears from both the received data and the definition of completeness, producing a reassuring percentage with no meaningful denominator.

The report example uses twenty explicit monthly dates from January 2023 through August 2024. They are period identifiers, not publication timestamps or evidence that a service ran on those days. A different instrument, reporting calendar or model use may require a different expected set; importing this fixture calendar into a real process would need justification.

Reconcile identities before interpreting counts

Consider a synthetic expected set containing January, February and March. A file containing January twice and March once has three rows, matching the expected count. It nevertheless omits February and duplicates January. Report expected count three, observed rows three, unique dates two, one missing date and one duplicated date.

An additional April row creates another problem even if the file still contains all three expected months. It changes the analysis window without permission. The engine reports unexpected dates separately and withholds calculations whose population or complete return window would become ambiguous. An operator should correct the scope or explicitly version a new contract, not silently trim inconvenient rows.

Separate field coverage from row coverage

A present row does not imply that every field is available. The feature, model return and benchmark return have separate null counts. A complete date set with one missing feature can still contain a complete aligned return history. Conversely, complete features cannot repair a missing benchmark return needed for the outcome comparison.

Suppose twenty dates are present, nineteen features are numeric, and every model and benchmark return is numeric. The report can calculate complete-window performance while labeling drift as an observed-subset diagnostic. If one model return becomes null, the performance block is withheld. These different consequences follow from each metric’s required inputs rather than a single blanket completeness score.

Publish the denominator and reason

W3C’s Data Quality Vocabulary treats a quality measurement as something with a metric and context. This guide applies that idea through explicit counts and definitions; it does not claim vocabulary conformance. A percentage is useful only when readers know which records belong in its denominator and which omissions it represents.

The example’s null shares use observed rows, including duplicates, as their denominator. That is a declared diagnostic convention, not a repair for duplicate identity. Retain both the share and the identity warning. For twenty observed rows with two null features, the null share is ten percent even if one row is duplicated and another expected month is absent.

Turn a failed contract into a specific action

An investigation should identify the missing keys, source handoff, last accepted generation and affected metrics. Decide whether the issue concerns delivery, an intentional calendar change or incorrect model identity. Preserve the original input and corrected input separately so a reviewer can see what changed and why the repaired report became calculable.

The interactive report validates a supplied teaching fixture; it does not discover a production inventory or independently verify upstream delivery. In a real deployment, expected sets and receipts would need durable cloud ownership. A complete local file is evidence about that file, while a continuing monitoring service requires separate execution, recovery and delivery evidence.

Keep in the decision record

• Define expected periods before examining received rows.
• Reconcile missing, duplicate and unexpected identities separately.
• Let each metric state its own input requirements.

Source notes

1. W3C — Data Quality Vocabulary, Working Group Note, 15 December 2016
2. W3C — Data on the Web Best Practices, Recommendation, 31 January 2017
3. W3C — PROV-O, Recommendation, 30 April 2013

Sources inform the stated methods and governance context. All example monitoring observations are synthetic. No regulatory approval or production monitoring certification is implied.

Continue the analysis