Create a daily Parquet aggregate from raw S3 data.
Keep raw data and do not expose a partially written table if the aggregation fails.
Which automation approach is most appropriate?
Write CTAS or INSERT INTO output to a staging location, validate it, and publish the successful result through Catalog metadata.
Detailed explanation
A failed rebuild removes the source needed for recovery and replay.
A failed rebuild removes the source needed for recovery and replay.
SQL can transform and partition data, while staging prevents partial output from becoming the published table.
SQL can transform and partition data, while staging prevents partial output from becoming the published table.
Audit events are not the business data being summarized.
Audit events are not the business data being summarized.
Lifecycle transitions storage classes and does not convert file formats.
Lifecycle transitions storage classes and does not convert file formats.
Check official documentation
Compare the following points with the official documentation.
Amazon Athena CTAS、INSERT INTO、partitioning、Glue Data Catalogの公式説明を確認する。
What to confirm
形式変換SQLとS3 Lifecycle・監査ログの役割を区別できる。
Key points
- Athena CTAS
- Parquet
- Staging
- Publish
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Atomic publication
A new result should be complete before consumers see the updated table or view.
Replay
Raw data and processing-window identifiers allow failed outputs to be rebuilt.