An ETL job uses several native libraries and processes multiple terabytes every day.
Tune CPU and memory per job and reproduce the execution image across the team.
Which design or service is most appropriate?
ECS or EKS containers can package dependencies in a reproducible image and tune resources for distributed ETL.
Detailed explanation
Metadata does not provide a reproducible execution environment or libraries.
Metadata does not provide a reproducible execution environment or libraries.
CloudTrail audits API calls; it is not a code execution environment.
CloudTrail audits API calls; it is not a code execution environment.
Runtime, package, memory, and concurrency limits do not fit terabyte-scale processing.
Runtime, package, memory, and concurrency limits do not fit terabyte-scale processing.
The image fixes dependencies and CPU and memory can be tuned per task or pod.
The image fixes dependencies and CPU and memory can be tuned per task or pod.
Check official documentation
Compare the following points with the official documentation.
Amazon ECSタスク定義、Amazon EKSワークロード、AWS Glue・EMRとの実行特性を確認する。
What to confirm
コンテナを使うべき処理量と、Lambdaや監査ログとの役割の違いを説明できる。
Key points
- ECS
- EKS
- Container image
- Resource design
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm the relevant DEA-C01 Skill and current AWS service documentation.
Foundation review
Execution boundary
Keep the image, scheduler, and data storage responsibilities distinct.
Scaling
Match parallel workers and resource limits to input size and transformation stages.