A FAQ model is updated.
Compare the old and new versions for accuracy, relevance, and safety.
Which evaluation-data design is most appropriate?
Create a representative fixed set containing questions, expected answers or acceptable behavior, evidence requirements, and prohibited outputs.
Detailed explanation
This cannot detect a quality or safety regression.
The same human-defined set enables a meaningful before/after comparison.
Self-generated text is not an independent standard for accuracy or safety.
One simple example cannot represent real use, edge cases, or refusal needs.
Check official documentation
Compare the following points with the official documentation.
Review Task 3.4 and the Bedrock model and RAG evaluation overview. Outline a before-and-after FAQ comparison using the same questions and criteria, separating expected evidence from prohibited outputs.
Official references
What to confirm
Distinguish evaluation data from training data and define accuracy, relevance, and safety criteria before comparing versions.
Key points
- Evaluation set
- Regression
- Accuracy and safety
Notes
- Environment: Reference review of the official AIF-C01 guide and AWS documentation; no AWS resources or inference are required.
- Keeping an evaluation set fixed does not mean adding its answers to training or fine-tuning data. Keep the comparison standard independent.
- Expected behavior need not be one exact string. Acceptable evidence, paraphrases, and refusal conditions can also be defined.
Foundation review
Independent evidence
The reference set should not be generated by the same system being evaluated.
Coverage
Examples should include normal cases, edge cases, and outputs that must be refused.