Three daily aggregates are missing from an otherwise running DAG.
Reprocess only those logical dates without disturbing the current schedule.
Which automation approach is most appropriate?
Use Airflow backfill or a logical-date parameter to limit reruns and make the output idempotent for each processing date.
Detailed explanation
Lifecycle manages storage transitions and expiration, not ETL recalculation.
Lifecycle manages storage transitions and expiration, not ETL recalculation.
Audit events do not recreate source data or DAG task results.
Audit events do not recreate source data or DAG task results.
Existing dates can be counted twice and downstream rows can be duplicated.
Existing dates can be counted twice and downstream rows can be duplicated.
Only the missing date range is rerun and its execution history is retained.
Only the missing date range is rerun and its execution history is retained.
Check official documentation
Compare the following points with the official documentation.
Apache Airflow backfill、logical date、catchupの公式説明を確認する。
What to confirm
全期間の無条件再実行と、対象日限定のbackfillの違いを説明できる。
Key points
- Backfill
- Logical date
- Catchup
- Idempotency
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Time semantics
Use the processing interval as a data key so delayed input can be recalculated.
Replacement boundary
Stage and replace one date's output instead of appending duplicate rows.