An Iceberg table is partitioned only by date, but regional queries are increasing.
Apply a new partition transform to future writes without rewriting every existing file immediately.
Which design is most appropriate?
Use Iceberg partition evolution to record old and new specifications in metadata and apply the new layout to future data.
Detailed explanation
Iceberg can manage old and new partition specifications in metadata without an immediate full file move.
Iceberg can manage old and new partition specifications in metadata without an immediate full file move.
A predicate does not change the physical partition specification or remove unrelated scans.
A predicate does not change the physical partition specification or remove unrelated scans.
Manual movement can break table metadata and cause a large, unnecessary rewrite.
Manual movement can break table metadata and cause a large, unnecessary rewrite.
A DynamoDB index does not change Iceberg file partitioning.
A DynamoDB index does not change Iceberg file partitioning.
Check official documentation
Compare the following points with the official documentation.
Apache Iceberg partition evolution、partition spec、snapshotの公式説明を確認する。
What to confirm
物理ファイルの手動移動と、メタデータ管理されたパーティション進化を区別できる。
Key points
- Partition evolution
- Partition spec
- Snapshot
- Compaction
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Gradual layout change
Engines must understand old and new specifications while data is migrated or compacted over time.
Measure pruning
Partition counts, file sizes, and query pruning show whether the new layout helps.