A Spark Structured Streaming job stops during processing.
Restore input offsets and state after restart without counting already processed records twice.
Which configuration is most appropriate?
Persist checkpoint state in durable storage and design an idempotent output so offsets and state survive restart.
Detailed explanation
Offsets, state, and progress can be restored after restart while output avoids duplicate effects.
Offsets, state, and progress can be restored after restart while output avoids duplicate effects.
Permission grants do not store or restore offsets and state.
Permission grants do not store or restore offsets and state.
Node replacement or restart can erase the state and force unsafe reprocessing.
Node replacement or restart can erase the state and force unsafe reprocessing.
Removing state objects prevents the stream from resuming at the correct position.
Removing state objects prevents the stream from resuming at the correct position.
Check official documentation
Compare the following points with the official documentation.
Apache Spark Structured Streaming checkpointing、offset、state storeの公式説明を確認する。
What to confirm
ローカル状態・S3保持・IAM権限の役割を区別できる。
Key points
- Structured Streaming
- Checkpoint
- Offset
- State recovery
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Progress and state
Recovery needs offsets plus aggregations, watermarks, and other operator state.
Duplicate effects
Output keys, transactions, and write semantics determine whether replay creates a second result.