A customer-support classifier is missing more cases after customer behavior changes.
Decide whether reassessment or retraining is needed.
Choose two appropriate monitoring signals.
Monitor changes in input distribution and performance after ground-truth labels become available.
Detailed explanation
Input drift can indicate that production no longer resembles training data.
Changes in request topics, vocabulary, or user mix can differ from training or reference periods. Investigate transformations and label-definition changes too; changed inputs do not make retraining the only possible remedy.
Self-comparison without independent ground truth cannot measure actual missed cases.
Reusing predictions as correct labels prevents independent assessment of the model's errors. Match predictions with trusted outcomes such as verified support-case classifications.
Successful responses and correct classifications measure different things; the former cannot substitute for prediction quality.
A high HTTP 200 rate can coexist with incorrect class predictions. Monitor service errors and latency separately; use ground-truth-based performance to assess the missed-case and retraining concern here.
Accuracy, recall, or error can be measured once outcomes are known.
Detecting 18 of 20 actual target-class cases gives 90% recall; detecting 8 gives 40%. The remaining cases are misses in this example. Overall accuracy alone can obscure increased misses in a less common class.
Check official documentation
Compare the following points with the official documentation.
Review MLOps, monitoring, and performance metrics in Task 1.3 and the documented comparison of predictions with ground-truth labels. For hypothetical support classification, compare a period with 18 detections and 2 misses among 20 actual target-class cases with a period having 8 detections and 12 misses among 20 such cases.
Official references
- Official documentation 1
- Official documentation 2
- Official documentation 3
- Official documentation 4
- Official documentation 5
- Official documentation 6
What to confirm
Target-class recall changes from 18/20 = 90% to 8/20 = 40%. Separate this from HTTP success and self-generated labels, and check label definitions, classification thresholds, and evaluation periods before deciding whether reassessment or retraining is needed.
Key points
- Data drift
- Performance monitoring
- Reassessment
Notes
- Environment: Reference review of official documentation and hypothetical calculations. No AWS resources, inference runs, or real user data are required.
- These are hypothetical counts. Align labels, target class, evaluation period, and decision threshold; recall alone does not establish false-positive behavior or business impact.
- Production monitoring and retraining of this classifier belong to Task 1.3. Task 3.4 covers methods to evaluate foundation-model performance, rather than serving as a general operations-monitoring task.
- The referenced SageMaker Model Monitor documentation states that the service is closed to new customers while existing customers can continue using it (reviewed 2026-10-28). This question teaches monitoring principles; it does not assume that a new account can enable that service.
Foundation review
Target-class recall
The proportion of actual target-class cases correctly detected. It directly describes the missed-case concern in this scenario.
Consistent labels and decision conditions
Match independent ground-truth labels to predictions and record class definitions and decision thresholds. A change in these conditions is not proof of model deterioration alone.
Service success versus correct predictions
An HTTP 200 response can establish that a request received a successful response without establishing that its classification was correct. Measure operational health and prediction quality separately.