A Spark transformation runs for a few hours each night.
Avoid paying for an idle cluster and retain job-level logs and retry control.
Which automation approach is most appropriate?
Run the jobs on EMR Serverless and manage capacity, networking, logs, and retries from the workflow.
Detailed explanation
Lambda runtime, memory, and dependency limits are unsuitable for distributed Spark.
Lambda runtime, memory, and dependency limits are unsuitable for distributed Spark.
Intermittent workloads incur idle capacity and cluster administration cost.
Intermittent workloads incur idle capacity and cluster administration cost.
S3 Select is object-level selection and does not replace distributed Spark.
S3 Select is object-level selection and does not replace distributed Spark.
Resources are provisioned for the job without operating a continuously running cluster.
Resources are provisioned for the job without operating a continuously running cluster.
Check official documentation
Compare the following points with the official documentation.
Amazon EMR Serverless、Spark job run、CloudWatch logsの公式説明を確認する。
What to confirm
EMR Serverlessと常時稼働クラスター・Lambdaの適用範囲を説明できる。
Key points
- EMR Serverless
- Spark
- Job run
- Idle cost
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Capacity bounds
Set maximum compute and concurrent runs to protect account and downstream limits.
Retry isolation
Use an execution-specific input and output path so failed jobs can be replayed safely.