Several terabytes of S3 data must be joined and aggregated daily into Parquet.
Do not depend on one Lambda's execution time or memory; finish with distributed processing.
Which processing platform is most appropriate?
Run an Apache Spark job on Amazon EMR for the distributed join and aggregation.
Detailed explanation
Event selectors filter recorded API activity and do not aggregate arbitrary S3 data.
Event selectors filter recorded API activity and do not aggregate arbitrary S3 data.
An IAM role grants permissions but does not provide a distributed compute engine.
An IAM role grants permissions but does not provide a distributed compute engine.
EMR distributes Spark work across multiple nodes and supports large joins and aggregations.
EMR distributes Spark work across multiple nodes and supports large joins and aggregations.
Lambda execution time, memory, and temporary-storage limits make this unsuitable for terabyte-scale work.
Lambda execution time, memory, and temporary-storage limits make this unsuitable for terabyte-scale work.
Check official documentation
Compare the following points with the official documentation.
Amazon EMR、Apache Spark、分散処理、S3入力の公式説明を確認する。
What to confirm
分散ETLと単一関数の実行制約、権限と計算基盤の役割を区別できる。
Key points
- Amazon EMR
- Apache Spark
- Distributed processing
- Terabyte-scale data
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Parallel execution
Partitioning data and shuffling join keys allows a cluster to process large datasets concurrently.
Cluster operations
Node types, startup time, autoscaling, spot usage, retries, and output layout affect cost and reliability.