Process a large data set in S3 with Apache Spark.
Manage the cluster's compute resources while running distributed jobs.
Which service is most appropriate?
Use Amazon EMR to run Spark on a managed cluster designed for large-scale distributed data processing.
Detailed explanation
S3 Lifecycle transitions and expires objects and does not provision compute clusters.
S3 Lifecycle transitions and expires objects and does not provision compute clusters.
IAM grants permissions but does not execute a data transformation engine.
IAM grants permissions but does not execute a data transformation engine.
Route 53 is a DNS service and does not run distributed processing jobs.
Route 53 is a DNS service and does not run distributed processing jobs.
EMR provides managed clusters for Hadoop and Spark frameworks and their distributed jobs.
EMR provides managed clusters for Hadoop and Spark frameworks and their distributed jobs.
Check official documentation
Compare the following points with the official documentation.
Amazon EMRとSparkの公式説明を確認する。
What to confirm
Glue ETLとEMRクラスターの使い分けを処理要件から判断できる。
Key points
- Amazon EMR
- Apache Spark
- Distributed processing
Notes
- Environment: AWS公式EMRドキュメントの確認
- Confirm current AWS service behavior in the official documentation.
Foundation review
Framework runtime
EMR supplies the managed cluster environment while the team submits Spark jobs.
Permission versus execution
IAM authorizes access to data and services but is not a data-processing engine.