The same Glue workflow runs for each date partition.
Pass the processing date for both normal runs and backfills without duplicating job definitions.
Which implementation is most appropriate?
Use Glue job arguments and workflow run properties to parameterize the date and input or output prefixes.
Detailed explanation
Each run can supply a date and prefixes while reusing the same workflow definition.
Each run can supply a date and prefixes while reusing the same workflow definition.
An API operation name does not represent the workflow's logical processing date.
An API operation name does not represent the workflow's logical processing date.
Changing an input path alone does not parameterize transformation and backfill behavior.
Changing an input path alone does not parameterize transformation and backfill behavior.
Duplicated definitions increase maintenance work and the chance of inconsistent fixes.
Duplicated definitions increase maintenance work and the chance of inconsistent fixes.
Check official documentation
Compare the following points with the official documentation.
AWS Glue job arguments、getResolvedOptions、Workflow run propertiesを確認する。
What to confirm
ジョブ複製と、実行引数による再利用の違いを説明できる。
Key points
- Glue job arguments
- Parameterization
- Backfill
- Reuse
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Definition reuse
One tested workflow can serve normal runs and historical backfills.
Input contract
Date format, empty partitions, and output overwrite rules belong in the argument contract.