Several DAGs call the same external API and Redshift cluster.
Keep independent tasks parallel while limiting only the shared resource usage.
Which automation approach is most appropriate?
Use Airflow Pools and DAG or task concurrency limits to reserve slots for shared resources without serializing every DAG.
Detailed explanation
This loses independent retry boundaries and encounters Lambda execution limits.
This loses independent retry boundaries and encounters Lambda execution limits.
Audit-event volume is not a synchronized execution-control mechanism.
Audit-event volume is not a synchronized execution-control mechanism.
Pool slots limit shared-resource tasks while unrelated tasks can remain parallel.
Pool slots limit shared-resource tasks while unrelated tasks can remain parallel.
Object retention rules do not allocate workflow execution slots.
Object retention rules do not allocate workflow execution slots.
Check official documentation
Compare the following points with the official documentation.
Amazon MWAA Apache Airflow pools、concurrency、parallelismの公式説明を確認する。
What to confirm
全体直列化と共有リソースだけを制限する設計の違いを説明できる。
Key points
- Airflow Pool
- Concurrency
- Parallelism
- Shared resource
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Capacity matching
Set pool slots from API rate limits, Redshift WLM, and observed task duration.
Fair scheduling
Monitor priorities and queue times so one DAG does not starve others.