An EMR Spark join has many rows for one customer ID, leaving one task running much longer than the others.
Reduce partition imbalance and shorten the job completion time.
Which mitigation is most appropriate?
Identify skewed keys and consider salting or repartitioning based on the join and data size.
Detailed explanation
This removes parallelism and makes skew and processing time worse.
This removes parallelism and makes skew and processing time worse.
IAM policies grant access and do not control Spark partitioning or retries.
IAM policies grant access and do not control Spark partitioning or retries.
Splitting a heavily used key across processing units reduces concentration in one task.
Splitting a heavily used key across processing units reduces concentration in one task.
Object history does not change Spark's runtime partition distribution.
Object history does not change Spark's runtime partition distribution.
Check official documentation
Compare the following points with the official documentation.
Apache Sparkのデータスキュー、パーティション、salting、join戦略の公式説明を確認する。
What to confirm
権限設定やS3版管理ではなく、分散処理のパーティション設計が原因と対策になることを説明できる。
Key points
- Spark
- Data skew
- Salt key
- Repartition
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Straggler task
One oversized partition can determine the completion time of the entire job.
Join strategy
Salting, repartitioning, or broadcasting should be selected from data sizes and memory limits.