Streaming ingestion creates many small Parquet files in an Iceberg table.
Reduce metadata work and file-open overhead while preserving table consistency.
Which approach is most appropriate?
Run table-managed compaction and plan snapshot retention so files are rewritten consistently and old metadata is cleaned up.
Detailed explanation
Rewriting small files under Iceberg metadata improves reads without breaking table state.
Rewriting small files under Iceberg metadata improves reads without breaking table state.
Physical movement alone can leave Catalog and Iceberg metadata inconsistent.
Physical movement alone can leave Catalog and Iceberg metadata inconsistent.
TTL deletes DynamoDB items and does not compact files.
TTL deletes DynamoDB items and does not compact files.
Reading many small files and unnecessary columns worsens the overhead.
Reading many small files and unnecessary columns worsens the overhead.
Check official documentation
Compare the following points with the official documentation.
Apache Iceberg rewrite data files、compaction、snapshot expirationの公式説明を確認する。
What to confirm
手動移動とテーブル管理されたコンパクションの違いを説明できる。
Key points
- Compaction
- Small files
- Snapshot expiration
- Iceberg
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
File-size balance
Larger files reduce open and metadata overhead but should still support parallel reads.
Concurrent writes
Compaction must preserve snapshot consistency while writers continue to commit.