Join several terabytes of transactions with a store master of only a few megabytes.
Avoid shuffling the large side and reduce network transfer and stage time.
Which strategy is most appropriate?
Use a broadcast join when the small dataset fits executor memory, and monitor its size and memory limit.
Detailed explanation
This loses parallelism and concentrates memory and network work on one node.
This loses parallelism and concentrates memory and network work on one node.
Object-retention rules do not select a Spark join strategy.
Object-retention rules do not select a Spark join strategy.
Audit events are not the required store master and do not match the join key.
Audit events are not the required store master and do not match the join key.
Replicating the small side to executors can avoid shuffling the large transaction dataset.
Replicating the small side to executors can avoid shuffling the large transaction dataset.
Check official documentation
Compare the following points with the official documentation.
Apache Spark broadcast join、broadcast threshold、shuffleの公式説明を確認する。
What to confirm
データスキュー対策と小さい側のブロードキャスト、S3保存設定の違いを説明できる。
Key points
- Spark
- Broadcast join
- Shuffle
- Executor memory
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Join strategy
Dataset sizes, key distribution, and memory determine whether broadcast or shuffle is safe.
Capacity guard
Monitor the small side and fail or switch strategies if it no longer fits memory.