S3 logs have added columns, changing types, and missing fields on different days.
Read the input in a Glue job and write a standard Parquet schema after transformation.
Which Glue data structure is most appropriate?
Use a DynamicFrame and resolve choices or apply mappings before writing the standardized schema.
Detailed explanation
Alarms monitor metrics and do not perform ETL transformations.
Alarms monitor metrics and do not perform ETL transformations.
DynamicFrame is designed to handle semi-structured input and schema inconsistencies before mapping to a standard schema.
DynamicFrame is designed to handle semi-structured input and schema inconsistencies before mapping to a standard schema.
Lifecycle policies manage storage class and deletion, not data-column types.
Lifecycle policies manage storage class and deletion, not data-column types.
The job can fail before it handles the varying types or missing fields.
The job can fail before it handles the varying types or missing fields.
Check official documentation
Compare the following points with the official documentation.
AWS Glue DynamicFrame、ResolveChoice、ApplyMappingの公式説明を確認する。
What to confirm
スキーマ揺れへの対応と、固定スキーマ変換の適用条件を説明できる。
Key points
- DynamicFrame
- ResolveChoice
- ApplyMapping
- Schema drift
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Input and output schemas
Flexible ingestion should end with an explicit schema contract for Athena or Redshift.
Data quality
Null handling, invalid values, and rejected records need a defined output or quarantine path.