An external order API limits each response and returns a nextCursor for the next page.
An hourly job must honor rate limits, resume from the last successful cursor after interruption, and store raw data in S3.
Which ingestion design is most appropriate?
Process the API's continuation token and persist a successful cursor checkpoint. Use backoff for rate limits and keep restartable raw data in S3.
Detailed explanation
A crawler discovers schemas in storage; it does not perform authenticated, paginated API retrieval.
A crawler discovers schemas in storage; it does not perform authenticated, paginated API retrieval.
The client can process nextCursor in order and persist the last successful position, limiting duplicate reads after a restart.
The client can process nextCursor in order and persist the last successful position, limiting duplicate reads after a restart.
Later pages are missed and a restart position is not retained, so the design cannot ingest all data reliably.
Later pages are missed and a restart position is not retained, so the design cannot ingest all data reliably.
The source can return 429 responses and create gaps or duplicates; request pacing and backoff are required.
The source can return 429 responses and create gaps or duplicates; request pacing and backoff are required.
Check official documentation
Compare the following points with the official documentation.
AWS GlueまたはLambdaのAPI取り込み実装、Secrets Manager、指数バックオフ、S3チェックポイントの公式資料を確認する。
Official references
What to confirm
データAPIのページング・レート制限・再開位置を、Crawlerや無制御並列処理と区別できる。
Key points
- Pagination
- Continuation token
- Checkpoint
- Rate limit
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm the relevant DEA-C01 Skill and current AWS service documentation.
Foundation review
Restartability
Persist a successful position and make repeated reads safe.
Source protection
Honor quotas and avoid uncontrolled concurrent requests.