Daily data is added to S3 and a future feed will contain an additional column.
Keep existing date partitions usable and let queries handle a missing value for the new column.
Which design is most appropriate?
Treat the new column as a nullable, backward-compatible addition and manage Catalog and partition updates deliberately.
Detailed explanation
This creates duplicated data and extra management while making history harder to track.
This creates duplicated data and extra management while making history harder to track.
DynamoDB TTL deletes expired items and does not change Catalog schemas.
DynamoDB TTL deletes expired items and does not change Catalog schemas.
Changing a column's meaning breaks downstream types and contracts.
Changing a column's meaning breaks downstream types and contracts.
An optional column can be added while older records supply NULL and Catalog metadata is updated in a controlled way.
An optional column can be added while older records supply NULL and Catalog metadata is updated in a controlled way.
Check official documentation
Compare the following points with the official documentation.
AWS Glue Data Catalog schema evolution、partitions、Athena schemaの公式説明を確認する。
What to confirm
パーティション追加と論理スキーマ変更の影響を区別できる。
Key points
- Schema evolution
- Partition
- Nullable
- Data Catalog
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Additive versus breaking
Adding an optional field is often compatible; changing a type or meaning requires a migration plan.
Mixed partitions
Queries over old and new partitions should be tested for type, null, and completeness behavior.