An Airflow DAG calls an external SaaS API in parallel.
Only tasks using that API must respect its rate limit without slowing unrelated DAGs.
Which configuration is most appropriate?
Assign the API tasks to an Airflow Pool with an appropriate number of slots.
Detailed explanation
Unbounded calls can exceed the API rate limit and increase failures and retries.
Unbounded calls can exceed the API rate limit and increase failures and retries.
Credential lifetime does not directly control the number of simultaneous calls.
Credential lifetime does not directly control the number of simultaneous calls.
File layout does not directly limit Airflow task concurrency.
File layout does not directly limit Airflow task concurrency.
Pool slots limit concurrent use of the shared API while leaving other DAGs independent.
Pool slots limit concurrent use of the shared API while leaving other DAGs independent.
Check official documentation
Compare the following points with the official documentation.
Apache Airflow Pool、pool_slots、DAG concurrencyの公式説明を確認する。
What to confirm
データ配置・認証期限と、ワークフローの同時実行制御を区別できる。
Key points
- Airflow Pool
- Concurrency
- Rate limit
- MWAA
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Scoped throttling
A pool constrains the selected resource without globally slowing every DAG.
Backpressure
Queueing, retries, and polling should follow the downstream API capacity.