The production distribution for a loan model is changing from training data.
Decide whether the model should be reevaluated.
Choose two appropriate monitoring practices.
Compare feature distributions and track performance over time and across groups to detect drift and its impact.
Detailed explanation
Output stability does not prove that input drift is harmless.
Repeatedly predicting the same class does not establish that input composition or error patterns are unchanged. Keep input checks separate from outcome-based quality checks rather than stopping input monitoring because outputs look stable.
The change may affect quality and different groups differently.
Match predictions to independently resolved outcomes and compare quality by period and group with case counts. An unchanged aggregate can conceal group differences. Record delayed or unobserved outcomes separately from the evaluated cases.
An aggregate can conceal deterioration in a particular group, so inspect group counts and quality as well.
Improvement in one group can offset deterioration in another, leaving aggregate accuracy unchanged. Inspect group counts, definitions, and observed-outcome coverage. Accuracy alone does not establish fairness across groups.
This detects changes in the input distribution.
Compare corresponding input features using consistent units and preprocessing. Inspect changes in proportions, missing values, and unexpected values to separate processing faults from population changes. Input shift can trigger reassessment without establishing prediction-quality loss.
Check official documentation
Compare the following points with the official documentation.
Compare Task 1.3 on MLOps with the official data-quality and model-quality explanations. For hypothetical input batches of 100 cases, compare category counts of 70/30 and 40/60. In a separate labeled evaluation with 100 cases in each group, compare matching-outcome counts of 90/90 and 95/85 for groups A/B and calculate aggregate and group-level agreement.
Official references
What to confirm
Do not infer prediction quality from an input-proportion change. In the labeled example, aggregate agreement remains 90% while group B changes from 90% to 85%. Distinguish evaluated cases from unresolved outcomes and avoid treating retraining as an automatic conclusion.
Key points
- Distribution shift
- Performance
- Continuous evaluation
Notes
- Environment: Reference review of official documentation and hypothetical counts. No actual lending decisions, AWS resource creation, model inference, or real user data are involved.
- The numbers are hypothetical teaching examples, not measurements of a real model or customers. Accuracy is one quality metric and does not alone establish business suitability or overall fairness.
- Evaluation limited to labels that have arrived can have coverage bias. Record period- and group-level counts, observed-outcome coverage, and unresolved outcomes rather than filling unknown results with guesses.
- The referenced Model Monitor documentation states that it is closed to new customers. This exercise reviews monitoring concepts and requires no new service enrollment or AWS operation.
Foundation review
Separate input change from quality
In a hypothetical example, 100 reference inputs contain category counts of 70/30 and 100 production inputs contain 40/60. Their proportions change from 70%/30% to 40%/60%. Those counts alone do not show whether predictions were wrong.
Look behind an aggregate
In a separate hypothetical evaluation, groups A and B each contain 100 labeled cases. Predictions match resolved outcomes in 90/90 cases at baseline and 95/85 cases in production. Aggregate agreement remains 180/200=90%, while group B changes from 90% to 85%. Report group definitions and counts.
Separate an alert from retraining
After detecting a distribution change, check missing values, units, preprocessing changes, and intended population changes. Reassess quality for periods with resolved outcomes, then investigate data repairs, further analysis, or retraining according to the cause and impact.