Continuously ingest application logs and store them in S3 with near-real-time delivery.
Avoid building a custom consumer and buffer-management process.
Which service is most appropriate?
Configure Amazon Kinesis Data Firehose with S3 as the destination so the service buffers and delivers the stream for you.
Detailed explanation
An archive class is not a near-real-time stream delivery pipeline.
An archive class is not a near-real-time stream delivery pipeline.
DNS log delivery is not a general-purpose stream ingestion and buffering service.
DNS log delivery is not a general-purpose stream ingestion and buffering service.
Firehose buffers streaming records and delivers them to S3 and other supported destinations as a managed service.
Firehose buffers streaming records and delivers them to S3 and other supported destinations as a managed service.
A custom implementation adds buffering, retry, and scaling work that Firehose already provides.
A custom implementation adds buffering, retry, and scaling work that Firehose already provides.
Check official documentation
Compare the following points with the official documentation.
Amazon Kinesis Data FirehoseのバッファとS3配信の公式説明を確認する。
What to confirm
Data StreamsとFirehoseの用途を、コンシューマー制御の有無で区別できる。
Key points
- Kinesis Data Firehose
- Buffering
- S3 delivery
Notes
- Environment: AWS公式Kinesis Firehoseドキュメントの確認
- Confirm current AWS service behavior in the official documentation.
Foundation review
Managed delivery
Firehose removes the need to operate a custom buffering consumer for this path.
Stream versus archive
Near-real-time ingestion is a pipeline concern, while an archive class is a storage concern.