A demand-forecasting model is running in production.
Seasonal and customer behavior changes may reduce prediction quality.
Choose two appropriate continuous monitoring activities.
Production monitoring should cover input distribution changes and, when labels arrive, the resulting prediction performance.
Detailed explanation
For demand forecasting, outcome-based error such as MAE can be tracked after matched actual demand becomes available.
Demand forecasting predicts numeric values, so outcome-based error such as MAE provides quality evidence here. Forecasts of 10 and 14 units matched to actual demand of 12 and 10 units have absolute errors of 2 and 4 units, giving MAE of 3 units. Match the item and forecast horizon correctly.
Treating a prediction as ground truth cannot measure its real prediction error.
Comparing a prediction with itself produces an apparent zero error without establishing correctness. Mark unresolved outcomes as unevaluated and compare with actual observed demand once it becomes available.
Input change can trigger reassessment but does not itself prove an increase in outcome-based error.
Detecting drift flags potentially changed conditions. Reassess performance on resolved outcomes and investigate seasonality or processing changes before choosing a response. Detecting a warning and establishing performance loss are separate decisions.
Input changes can indicate data drift and trigger reassessment.
Seasonality can change product mix and demand levels. Record comparison periods and item mix, and investigate whether the change is expected or reflects missing data or changed units. Distribution change alone does not establish quality loss or justify immediate retraining.
Check official documentation
Compare the following points with the official documentation.
Review monitoring and retraining in Task 1.3 and the distinction between data quality and model quality. For hypothetical forecasts of 10 and 14 units and matched actual demand of 12 and 10 units for the same forecast horizon, calculate MAE and separate it from input-distribution signals available before outcomes arrive.
Official references
- Official documentation 1
- Official documentation 2
- Official documentation 3
- Official documentation 4
- Official documentation 5
- Official documentation 6
What to confirm
Absolute errors of 2 and 4 units give an MAE of 3 units. Explain why input drift can trigger reassessment without proving that outcome-based prediction error has worsened.
Key points
- Data drift
- Prediction performance
- Continuous monitoring
Notes
- Environment: Reference review of official documentation and hypothetical calculations. No AWS resources, inference runs, or real user data are required.
- The numbers are hypothetical teaching data, not measured model performance. When comparing MAE across periods, account for differences in item mix, forecast horizon, and units.
- Distinguish missing outcomes from incorrect predictions. Avoid evaluation biased toward labels that arrived early or joins based only on arrival order; report the target period and number of evaluable cases.
- The referenced SageMaker Model Monitor documentation states that the service is closed to new customers while existing customers can continue using it (reviewed 2026-10-28). This question teaches monitoring principles; it does not assume that a new account can enable that service.
Foundation review
Input change versus quality loss
Data drift describes a change in input distribution. It does not guarantee performance loss, and a small input change does not rule out a changed relationship with outcomes.
Forecast error and matching
Match each forecast to actual demand for the same item and target period. MAE averages absolute errors and, in this example, has the same unit as demand.
Outcome delay
Actual demand may remain unknown until the forecast period ends. Input and missing-value checks can run earlier, while prediction quality for unresolved outcomes remains unevaluated.