Batch-process events and produce the latest state for each customer.
Keep the result stable when events have the same timestamp and when the job is replayed.
Which approach is most appropriate?
Use a window with row_number ordered by event time and a unique event ID, then keep rank one.
Detailed explanation
Removing history prevents replay and is not a state-selection query.
Removing history prevents replay and is not a state-selection query.
API audit records do not define the business event order.
API audit records do not define the business event order.
The window rank selects a complete latest row with a deterministic tie-breaker.
The window rank selects a complete latest row with a deterministic tie-breaker.
Deduplication does not guarantee that the retained row is the newest.
Deduplication does not guarantee that the retained row is the newest.
Check official documentation
Compare the following points with the official documentation.
Apache Spark Window、row_number、partitionBy、orderByの公式説明を確認する。
What to confirm
重複削除と最新順位選択の違いを説明できる。
Key points
- Window
- row_number
- Event time
- Determinism
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Late data
Late arrivals and watermarks affect when event-time state can be considered final.
Stable replay
A unique event ID prevents distributed execution order from changing the selected row.