Transform a multi-gigabyte text file line by line.
Keep memory usage bounded instead of loading the entire file at once.
Which implementation is most appropriate?
Use a file iterator or generator to process one record at a time and avoid a full in-memory list.
Detailed explanation
The complete file is held in memory and can exceed a job or Lambda limit.
The complete file is held in memory and can exceed a job or Lambda limit.
Values are produced incrementally, keeping memory use largely independent of total input size.
Values are produced incrementally, keeping memory use largely independent of total input size.
Logs are not a data store and create cost, search, and confidentiality problems.
Logs are not a data store and create cost, search, and confidentiality problems.
Object protection does not change Python's input-reading strategy.
Object protection does not change Python's input-reading strategy.
Check official documentation
Compare the following points with the official documentation.
Python generator、iterator、yield、AWS Lambda・Glueのメモリ制約を確認する。
What to confirm
遅延評価と全量読み込みのメモリ特性を説明できる。
Key points
- Python generator
- Iterator
- yield
- Memory
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Lazy evaluation
Only the next required value is held, reducing peak memory for large inputs.
Failure boundary
Decide whether one bad record is isolated or causes the complete job to fail.