An external system places a daily file in S3 at an unpredictable time.
Wait in the DAG until the key exists, then start ETL without holding a worker unnecessarily.
Which configuration is most appropriate?
Use an Airflow S3KeySensor or deferrable sensor in MWAA with a timeout and polling policy.
Detailed explanation
The job can fail before the file exists and requires custom retry timing.
The job can fail before the file exists and requires custom retry timing.
Frequent calls increase cost and require custom waiting-state management.
Frequent calls increase cost and require custom waiting-state management.
The sensor waits for the key and can reduce worker occupation while the external condition is pending.
The sensor waits for the key and can reduce worker occupation while the external condition is pending.
Object Lock protects objects and does not notify Airflow of arrival.
Object Lock protects objects and does not notify Airflow of arrival.
Check official documentation
Compare the following points with the official documentation.
Amazon MWAA・Apache AirflowのS3KeySensor、Deferrable Operator、poke intervalを確認する。
What to confirm
定期ポーリングLambdaとAirflowセンサーの待機・運用差を説明できる。
Key points
- MWAA
- S3KeySensor
- Deferrable sensor
- External condition
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Condition versus schedule
A sensor waits for state; a schedule starts at a time regardless of whether the file exists.
Worker efficiency
Polling interval and deferral determine worker use, API calls, and detection latency.