Files are added to S3 every day and a Glue job processes them.
Avoid rereading all files while retaining a way to reprocess inputs after a failure.
Which automation approach is most appropriate?
Enable Glue job bookmarks and define reset, replay, and source-retention procedures for incremental processing.
Detailed explanation
Glue tracks processed input state so later runs can focus on new data.
Glue tracks processed input state so later runs can focus on new data.
Runtime and cost grow with the full dataset and the job is not incremental.
Runtime and cost grow with the full dataset and the job is not incremental.
Deleting source data removes a reliable basis for replay and audit.
Deleting source data removes a reliable basis for replay and audit.
TTL expires DynamoDB items and does not track S3 processing state.
TTL expires DynamoDB items and does not track S3 processing state.
Check official documentation
Compare the following points with the official documentation.
AWS Glue job bookmarks、enable/reset、incremental processingの公式説明を確認する。
What to confirm
増分追跡とファイル削除による擬似的な管理の違いを説明できる。
Key points
- Job bookmark
- Incremental processing
- Reset
- Source retention
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
State changes
Code or source-location changes can affect bookmark behavior and should be tested.
Reproducible replay
Retain raw data and reset state deliberately rather than deleting the evidence.