S3 contains hundreds of thousands of files that need independent transformations.
Distribute input items to child workflows and control their history and concurrency.
Which configuration is most appropriate?
Use Step Functions Distributed Map with S3 item readers, child executions, and a concurrency limit.
Detailed explanation
S3 Select queries object content but does not distribute workflow executions.
S3 Select queries object content but does not distribute workflow executions.
Input and execution-history limits make Inline Map unsuitable for very large collections.
Input and execution-history limits make Inline Map unsuitable for very large collections.
Distributed Map scales large input processing as child executions and supports concurrency control.
Distributed Map scales large input processing as child executions and supports concurrency control.
You would need to build concurrency limits, recursion protection, and failure tracking yourself.
You would need to build concurrency limits, recursion protection, and failure tracking yourself.
Check official documentation
Compare the following points with the official documentation.
Step Functions Distributed Map、ItemReader、ResultWriter、MaxConcurrencyを確認する。
What to confirm
Inline Map・Distributed Map・再帰Lambdaの上限と運用差を説明できる。
Key points
- Distributed Map
- Child workflow
- Large input
- MaxConcurrency
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm current AWS service behavior in the official DEA-C01 documentation.
Foundation review
Large-scale orchestration
Input size, execution history, and downstream throttles determine whether distribution is needed.
Result aggregation
Collect child outputs and failures so selected items can be retried safely.