New date partitions are added daily and the table has millions of partitions.
Avoid the cost of crawling and registering every partition while allowing Athena to calculate matching paths.
Which configuration is most appropriate?
Use Athena Partition Projection with partition ranges, formats, and locations defined as table properties.
Detailed explanation
Athena can calculate partition values from table properties instead of listing every catalog partition.
Athena can calculate partition values from table properties instead of listing every catalog partition.
VACUUM reorganizes Redshift storage and does not discover Athena partitions.
VACUUM reorganizes Redshift storage and does not discover Athena partitions.
Object versions do not generate partition values for a query.
Object versions do not generate partition values for a query.
Repeatedly scanning millions of partitions increases runtime and cost.
Repeatedly scanning millions of partitions increases runtime and cost.
Check official documentation
Compare the following points with the official documentation.
Amazon Athena Partition Projection、projection.*テーブルプロパティの公式説明を確認する。
What to confirm
Crawler登録とクエリ時投影、S3版管理の違いを説明できる。
Key points
- Partition Projection
- Partition range
- Table property
- Athena
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Catalog versus projection
Projection avoids individual catalog entries when partition values and locations are predictable.
Query pruning
Predicates must match the projected properties so Athena reads only relevant paths.