Multiple measurements and duplicate event IDs arrive from each device.
Keep the one with the latest receive time per device before aggregation.
Which transformation is most appropriate?
Partition by device ID and rank rows with a window function, retaining the latest row per group.
Detailed explanation
IAM controls access and does not compare timestamps in records.
IAM controls access and does not compare timestamps in records.
Log retention does not deduplicate analytical data.
Log retention does not deduplicate analytical data.
A window ordering gives each device a deterministic ranking so only its latest record remains.
A window ordering gives each device a deterministic ranking so only its latest record remains.
Object-key order is not guaranteed to represent event time or duplicate identity.
Object-key order is not guaranteed to represent event time or duplicate identity.
Check official documentation
Compare the following points with the official documentation.
Apache Sparkのwindow、row_number、partitionBy、orderByの公式説明を確認する。
What to confirm
ファイル順序や保存期間ではなく、データ列のキーと時刻で重複を判定できる。
Key points
- Spark window
- row_number
- Deduplication
- Latest record
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Deterministic ordering
A complete ordering and tie-breaker ensure the same input produces the same surviving row.
Event time versus receive time
The selected timestamp must match the business rule for freshness and late arrivals.