Join a multi-terabyte sales dataset with a dimension table of only a few megabytes in Spark.
Reduce a large-table shuffle while keeping the dimension within executor memory.
Which approach is most appropriate?
Broadcast the small relation only after checking its actual size, executor memory, skew, and the broadcast threshold.
Detailed explanation
Copying the small table to executors can avoid shuffling the large relation.
Copying the small table to executors can avoid shuffling the large relation.
This loses distributed execution and can exceed Lambda limits.
This loses distributed execution and can exceed Lambda limits.
Shuffling the small table too adds unnecessary network and compute work.
Shuffling the small table too adds unnecessary network and compute work.
Archiving does not change Spark join memory or network behavior.
Archiving does not change Spark join memory or network behavior.
Check official documentation
Compare the following points with the official documentation.
Apache Spark broadcast joins、Spark SQL autoBroadcastJoinThresholdの公式説明を確認する。
What to confirm
broadcast joinとshuffle joinのメモリ・ネットワーク特性を説明できる。
Key points
- Broadcast join
- Shuffle
- Executor memory
- Broadcast threshold
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Applicability
Measure relation size, skew, memory, and garbage collection before forcing a broadcast.
Fallback
When the dimension grows beyond the threshold, use another join plan and monitor it.