Methodology: an illustrative monitoring report

A frozen contract before any metric

The input identifies a model, model version, UTC as-of time, declared last-updated time and explicit expected period dates. Each observed row contains a date, feature value, model return and benchmark return. Returns are decimals for the stated period, not percentages entered as whole numbers. A separate baseline feature sample, fixed bin edges, smoothing value and example policy travel with the report.

The presets contain twenty expected monthly keys from January 2023 through August 2024. These dates label observation periods; they are not source publication timestamps. The publication uses invented values, including the declared delivery clock. It neither connects to a real model nor verifies that an unattended process ran at those times. Imported JSON is evaluated under the same explicit contract.

Quality first, with denominators preserved

Quality checks distinguish expected dates, observed rows, unique dates, missing dates, duplicates and unexpected dates. Null feature and return counts are reported separately. Null shares use observed rows as their denominator, even when duplicates exist; a raw row count is therefore not proof of complete unique coverage. Invalid dates, nonfinite values, unsupported types and inconsistent timestamps are rejected rather than silently repaired.

Declared freshness is the difference between as_of and last_updated in hours. It measures the age of supplied delivery metadata, not the age of the observation period or independently established cloud execution. Missing, duplicate or unexpected rows and return gaps trigger the illustrative escalation policy. A null feature or stale delivery time can require review without automatically invalidating an otherwise complete return series.

Fixed-bin drift is descriptive

Bins have left-inclusive, right-exclusive boundaries and infinite tails. With baseline counts b, current counts c, sample sizes N and M, K bins and positive smoothing alpha, baseline proportions are (b + alpha)/(N + alpha × K), and current proportions use (c + alpha)/(M + alpha × K). PSI is the sum of (current proportion − baseline proportion) × natural log(current proportion/baseline proportion).

Standardized mean shift is current mean minus baseline mean, divided by the baseline sample standard deviation using N − 1. Drift is withheld for duplicate or unexpected dates, fewer than ten usable current features, or zero baseline standard deviation. Missing expected rows or features can permit an explicitly labelled observed-subset diagnostic. Neither PSI nor mean shift is a significance test, an explanation of causality or a diagnosis of concept drift.

Complete-window outcome measurement

Performance requires one row for every expected date, no unexpected rows, complete aligned model and benchmark returns and at least two periods. Otherwise the complete-window block is withheld with its reason. Each cumulative return is the product of one plus its period returns, minus one. Period active returns are model minus benchmark; their arithmetic mean is distinct from the difference between the two cumulative returns.

Tracking error is the sample standard deviation of active returns multiplied by the square root of the declared periods per year. This is a conventional scaling assumption, not evidence that observations are independent. Wealth starts at one; drawdown is wealth divided by its running peak minus one, including that initial peak. The report adds no fees, slippage, Sharpe ratio, statistical significance or risk-adjusted alpha claim.

Interpret the example policy as a review queue

The default PSI levels are 0.10 for review and 0.25 for escalation, with comparisons including equality. The freshness review threshold is 48 hours; the drawdown review threshold is a 10% loss from the running peak. These values were chosen for the teaching fixture. They are not regulatory limits, statistically calibrated universal cutoffs or a provider's production controls.

PSI and drawdown comparisons allow a tiny floating-point rounding tolerance of 8 × Number.EPSILON × max(1, absolute value, absolute boundary) to preserve inclusive mathematical boundaries. This corrects numerical representation only; it is not a measurement-uncertainty allowance. Freshness still requires age strictly greater than its threshold. Unsupported numerical precision, including a nonconstant sample whose variance underflows, is rejected rather than interpreted as a stable result.

The status can be within_example_bounds, review or escalate. A low metric does not cancel a quality defect, and a withheld metric is not zero. Preserve the report, input, version and policy when changing a baseline or responding to an alert. The change register illustrates accountable decisions; the website does not send alerts, execute rollbacks or monitor the future.

Keep in the decision record

• Keep quality, drift and outcome interpretation separate.
• Withheld results remain visible and are never replaced by zero.
• Thresholds are explicit example decisions, not certification.

Source notes

1. NIST statistical handbook: measures of scale
2. CFA Institute: active return and tracking error
3. CFA Institute: portfolio risk and drawdown context

Sources inform the stated methods and governance context. All example monitoring observations are synthetic. No regulatory approval or production monitoring certification is implied.