S3 data arrives daily and schema changes occur about once a month.
Avoid scanning every minute while keeping required metadata updates current.
Which schedule is most appropriate?
Schedule or trigger the crawler according to data arrival and schema-change frequency rather than running it continuously.
Detailed explanation
Repeated scans with no changes add cost and load without improving freshness.
Repeated scans with no changes add cost and load without improving freshness.
New partitions and schema changes would never be discovered automatically.
New partitions and schema changes would never be discovered automatically.
The scan frequency matches partition and schema changes and avoids unnecessary cost.
The scan frequency matches partition and schema changes and avoids unnecessary cost.
Storage class affects access characteristics, not crawler scheduling.
Storage class affects access characteristics, not crawler scheduling.
Check official documentation
Compare the following points with the official documentation.
AWS Glue Crawler schedule、event trigger、料金の公式説明を確認する。
What to confirm
自動更新の利便性と過剰スキャンのコストを説明できる。
Key points
- Crawler schedule
- Event trigger
- Scan cost
- Metadata freshness
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Freshness-cost trade-off
More frequent scans improve freshness only when the source changes often enough to justify them.
Targeted updates
Explicit partition registration can be cheaper than repeatedly crawling stable data.