"MLOps" cheat sheet

What to monitor

Record for every training run

code version      a94f3c2data version      churn-2026-03-01parameters        learning_rate=0.05 max_depth=6seed              42environment       image digest, library versions, hardwaremetrics           overall and per important groupartifacts         model file, evaluation report

Data checks before training

checks:
  - row_count: {min: 90000, max: 130000}
  - freshness: {column: event_date, max_age_hours: 30}
  - not_null: [customer_id, event_date]
  - range: {column: days_since_login, min: 0, max: 3650}
  - allowed_values: {column: plan_tier, values: [free, basic, pro, team]}
on_fail: stop_pipeline_and_page

Release steps

1. Register the model with run id, data version and metrics.
2. Package it in a pinned image and record the digest.
3. Gate: compare with production on a fixed set, including key groups.
4. Shadow on real traffic, then a small canary with quality checks.
5. Promote in steps. Keep the previous version ready.

Serving choices

Drift alert example

- alert: FeatureDriftHigh
  expr: model_feature_psi{model="churn",feature="days_since_login"} > 0.25
  for: 6h
  labels: {severity: ticket}

Debug a model bottom-up

1. What changed in the last week: data, feature code, model, image, traffic.
2. Inputs against training: null rate, range, new categories.
3. Served features against training features for the same record.
4. Replay a logged request with the same model version.
5. Only then the model, and last serving.

Commands for a model check

$ python check_inputs.py --window 48h --baseline training

$ python compare_features.py --sample 500 --serving-vs-training

$ python replay.py --request-id 7f3a --model churn-v14