Compare answer quality after every model update.
Include ordinary, harmful, and refusal cases to detect regressions.
Choose two appropriate contents.
Keep representative questions and expected evidence fixed, include safety and boundary cases, and inspect severe individual failures in addition to averages.
Detailed explanation
There is no stable comparison baseline.
Small but high-impact failures can be missed.
Quality can be compared across versions.
Safety and failure regressions become visible.
Check official documentation
Compare the following points with the official documentation.
Read Task 3.4 on evaluating foundation models and their applications. Categorize ordinary requests, refusals, safety cases, and language or length boundaries, then outline a comparison that separates average quality from severe failures under stable criteria.
Official references
What to confirm
Maintain stable comparison questions and criteria while inspecting boundary cases and severe failures that averages can hide.
Key points
- Fixed set
- Regression
- Boundary case
Notes
- Environment: Reference review of the official AIF-C01 guide and AWS documentation; no AWS resources or inference are required.
- A fixed set can become unrepresentative as usage changes. Preserve a version for comparison while recording added cases and criterion changes.
- Do not let an average score cancel out severe safety failures. Check them separately against predefined acceptance criteria.
Foundation review
Golden set
A reviewed collection of inputs with expected answers, decisions, or evidence used as a reference.
Regression test
A test that checks whether a change broke behavior that previously met its requirements.