An image upload API receives short bursts of heavy traffic. Image conversion uses a long-running custom binary, and the API must respond without waiting for processing.
Scale ingestion and processing independently while absorbing queued jobs without losing them.
Which architecture is most appropriate?
Decouple upload intake from long-running image processing with Amazon SQS, then scale only the worker fleet from queue depth.
Detailed explanation
Kinesis is intended for ordered stream processing and requires shard and consumer capacity planning. It does not meet the requirement for queue-depth-based independent worker scaling.
Kinesis is intended for ordered stream processing and requires shard and consumer capacity planning. It does not meet the requirement for queue-depth-based independent worker scaling.
SNS is primarily for notification fanout, not a durable work queue that independently scales long-running processors. A fixed concurrency limit does not clear burst backlogs.
SNS is primarily for notification fanout, not a durable work queue that independently scales long-running processors. A fixed concurrency limit does not clear burst backlogs.
Synchronous calls couple the API to worker latency. Intake and processing cannot scale independently, and there is no durable backlog for bursts.
Synchronous calls couple the API to worker latency. Intake and processing cannot scale independently, and there is no durable backlog for bursts.
SQS separates intake and processing asynchronously and retains messages temporarily. The worker fleet can scale independently from metrics such as ApproximateNumberOfMessagesVisible.
SQS separates intake and processing asynchronously and retains messages temporarily. The worker fleet can scale independently from metrics such as ApproximateNumberOfMessagesVisible.
Check official documentation
Compare the following points with the official documentation.
Amazon SQSのデカップリングと、キューのメッセージ数を使ったコンシューマーの水平スケールを確認する。
Official references
What to confirm
取り込み側は処理完了を待たず、キューに蓄積されたジョブを処理ワーカーが独立した台数で消化する。
Key points
- Amazon SQS
- Asynchronous decoupling
- Queue-depth scaling
Notes
- Environment:
- Confirm current behavior in the official SAA-C03 guide and AWS service documentation.
Foundation review
Synchronous versus asynchronous
Long-running jobs can be placed on a queue so intake does not wait for processing and bursts can be absorbed.
Scaling boundary
Use a load signal for each side so only the component that needs capacity is scaled.