Read several hundred megabytes of configuration files, join them with an API master, and write to S3.
Avoid starting a Spark cluster for this small job and operate the Python process simply.
Which Glue job type is most appropriate?
Use a Glue Python shell job for a small Python transformation that does not require distributed Spark processing.
Detailed explanation
DynamoDB Streams records item changes and does not execute an arbitrary Python batch.
DynamoDB Streams records item changes and does not execute an arbitrary Python batch.
A Python shell job runs lightweight Python ETL without a Spark cluster.
A Python shell job runs lightweight Python ETL without a Spark cluster.
Object Lock protects objects from deletion or overwrite and does not run Python code.
Object Lock protects objects from deletion or overwrite and does not run Python code.
Cluster operations and cost are excessive for a small, infrequent transformation.
Cluster operations and cost are excessive for a small, infrequent transformation.
Check official documentation
Compare the following points with the official documentation.
AWS Glue Python shellジョブとSparkジョブの実行環境・制約の公式説明を確認する。
What to confirm
データ量・並列性に応じたGlueジョブ形式の選択を説明できる。
Key points
- Glue Python shell
- Small ETL
- Spark
- Runtime
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Runtime selection
Data volume, memory, duration, and parallelism determine whether Python shell or Spark is suitable.
Scale trigger
Monitor input size and runtime to define when a Spark-based job becomes necessary.