A retrieval chatbot gives wrong answers.
Investigation shows that unrelated documents are retrieved before generation.
What should be evaluated first?
Evaluate whether retrieved documents are relevant, complete, and authorized before evaluating the generated answer.
Detailed explanation
If retrieval supplies poor evidence, generation cannot reliably answer from the source.
If retrieval supplies poor evidence, generation cannot reliably answer from the source.
UI styling does not measure retrieval quality.
UI styling does not measure retrieval quality.
Temperature does not fix irrelevant retrieval results.
Temperature does not fix irrelevant retrieval results.
Deletion removes the evidence needed for grounded generation.
Deletion removes the evidence needed for grounded generation.
Try it yourself
An example you can run in a temporary verification environment.
Amazon Bedrock Knowledge Bases公式の検索・取得結果と引用の説明を確認する。Expected result
取得段階と生成段階を分け、検索結果の関連性を評価する理由を説明できる。Key points
- Retrieval relevance
- Retrieval stage
- Generation stage
Notes
- Environment: AWS公式AIF-C01試験ガイドとAWS公式ドキュメントの確認
- Command output formatting can vary slightly by distribution or tool version.
- Run the example in a temporary directory or process when possible.
Foundation review
Read the scope first
Check whether the command acts on the current shell, a new process, an existing process, or a file.
Verify the observable result
Use the supplied command and compare the output with the expected result.