When new data arrives in S3, update the schema with a crawler and then run a Glue ETL job.
Manage the order inside Glue.
Which configuration is most appropriate?
Use a Glue Workflow and trigger that starts the ETL job after crawler success.
Detailed explanation
CloudTrail records API activity and does not define Glue dependencies.
CloudTrail records API activity and does not define Glue dependencies.
A trigger can use crawler success as a condition before starting the ETL job.
A trigger can use crawler success as a condition before starting the ETL job.
UNLOAD writes data out of Redshift and does not control Glue execution order.
UNLOAD writes data out of Redshift and does not control Glue execution order.
Object history does not start a job after a crawler finishes.
Object history does not start a job after a crawler finishes.
Check official documentation
Compare the following points with the official documentation.
AWS Glue Workflow、Trigger、Crawler、Jobの公式説明を確認する。
What to confirm
データ保存とGlueワークフローの実行制御を区別できる。
Key points
- Glue Workflow
- Crawler
- Trigger
- Dependency
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Metadata dependency
A crawler can refresh catalog information before a job reads the data.
Failure propagation
Define whether crawler failure stops ETL and how notification or retry occurs.