A few customers have far more events than others, making Spark tasks uneven.
Identify the hot keys before choosing salting or repartitioning.
Which quality approach is most appropriate?
Profile records per key, partition sizes, and task durations, then choose a meaning-preserving skew mitigation.
Detailed explanation
The concentrated values and their execution impact become measurable before redesign.
The concentrated values and their execution impact become measurable before redesign.
Randomization loses business meaning and breaks joins and aggregates.
Randomization loses business meaning and breaks joins and aggregates.
Data loss only hides the skew.
Data loss only hides the skew.
API activity does not show data-value concentration.
API activity does not show data-value concentration.
Check official documentation
Compare the following points with the official documentation.
Apache Spark data skew、partition size、task metricsの公式説明を確認する。
What to confirm
キーの集中を測定せずランダム変換する危険性を説明できる。
Key points
- Data skew
- Partition
- Task metrics
- Salting
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Meaning preservation
Remove salt during final aggregation and retain the original business key.
Measurement
Compare maximum and median partition size, duration, and garbage collection.