Compare generated summaries with reference summaries.
Use an automated metric for an initial evaluation.
Which metrics compare overlap with reference text?
ROUGE and BLEU compare generated text with reference text, but semantic quality and safety require additional evaluation.
Detailed explanation
These metrics use overlap with reference text for tasks such as summarization or translation.
These metrics use overlap with reference text for tasks such as summarization or translation.
CPU utilization describes resource use, not text quality.
CPU utilization describes resource use, not text quality.
A storage class describes storage characteristics, not model quality.
A storage class describes storage characteristics, not model quality.
Policy versioning manages permissions rather than generated text quality.
Policy versioning manages permissions rather than generated text quality.
Try it yourself
An example you can run in a temporary verification environment.
AWS公式AIF-C01ガイドのDomain 3.4で、モデル評価指標と自動評価の限界を確認する。Expected result
ROUGE・BLEUを参照テキストとの比較に使う指標として説明できる。Key points
- ROUGE
- BLEU
- Reference text
- Evaluation limits
Notes
- Environment: AWS公式AIF-C01試験ガイドとAWS公式ドキュメントの確認
- Command output formatting can vary slightly by distribution or tool version.
- Run the example in a temporary directory or process when possible.
Foundation review
Read the scope first
Check whether the command acts on the current shell, a new process, an existing process, or a file.
Verify the observable result
Use the supplied command and compare the output with the expected result.