A missing-value rate rose from below one percent to five percent.
Notify operators about quality degradation even when the job itself succeeds.
Which monitoring approach is most appropriate?
Publish quality results as CloudWatch metrics and alarm on business-defined thresholds or normal ranges.
Detailed explanation
A successful execution can still produce invalid or incomplete data.
A successful execution can still produce invalid or incomplete data.
Missing and duplicate rates become time-series signals that can block publication and notify operators.
Missing and duplicate rates become time-series signals that can block publication and notify operators.
Deletion destroys evidence and does not measure the quality regression.
Deletion destroys evidence and does not measure the quality regression.
Object-write count does not describe missing columns in records.
Object-write count does not describe missing columns in records.
Check official documentation
Compare the following points with the official documentation.
AWS Glue Data Quality、CloudWatch custom metrics、alarmsの公式説明を確認する。
What to confirm
ジョブ成功と品質成功を別の指標で監視する必要性を説明できる。
Key points
- Quality metric
- Missing-value rate
- Alarm
- Quality gate
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Quality SLO
Define tolerated missing, duplicate, delay, and range-violation rates.
Response
Stop publication and identify the affected partition and upstream source.