The same order event can arrive in S3 more than once after a retry.
Keep one record per event ID and prevent double counting in aggregates.
Which processing approach is most appropriate?
Use a stable event ID as an idempotency key and select one record according to an explicit rule.
Detailed explanation
Object versions do not deduplicate identical events stored in different files.
Object versions do not deduplicate identical events stored in different files.
An explicit duplicate key and a selection rule normalize retries to one record.
An explicit duplicate key and a selection rule normalize retries to one record.
IAM controls resource operations and does not inspect record contents.
IAM controls resource operations and does not inspect record contents.
Deleting an alarm does not transform data or remove duplicate records.
Deleting an alarm does not transform data or remove duplicate records.
Check official documentation
Compare the following points with the official documentation.
AWS Glueの重複排除、SparkのdropDuplicates、ウィンドウ関数の公式説明を確認する。
What to confirm
保存オブジェクトの版管理と、レコード内容の重複排除を区別できる。
Key points
- Deduplication
- Event ID
- Idempotency
- Replay
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
At-least-once delivery
Retries can deliver one event more than once, so consumers need a duplicate strategy.
Selection rule
First, latest, or highest-quality values must be chosen explicitly when duplicates differ.