An AI feature serves Japanese, English, and an underrepresented language.
Measure language differences.
Choose two evaluations.
Use language-specific representative, dialect, formatting, terminology, and refusal cases, and compare factuality, safety, fairness, latency, and comprehension.
Detailed explanation
Language differences remain.
English results support conclusions about the tested English data and conditions. They do not guarantee performance with other vocabulary, scripts, cultural context, or safety-filter behavior, so the target languages need their own evaluation.
Overall gaps are measured.
Factuality, safety, fairness, latency, and comprehension are different dimensions; report them separately by language. Record model and prompt versions, task difficulty, and the rubric so that differences in testing conditions are not mistaken for language capability gaps.
Language-specific failures are visible.
Use comparable task types and scoring criteria while including natural questions, dialects, script variations, and domain terms for each language. Direct translations of English cases can miss local failure modes. Include prohibited and legitimate requests to assess whether refusals are appropriate.
Impact is hidden.
If English has 9/10 correct answers, Japanese 8/10, and another language 1/2, pooled accuracy is 18/22, about 81.8%, while per-language accuracy is 90%, 80%, and 50%. Pooling hides the gap. Still, do not treat the two-case 50% as a stable ranking; collect more representative evidence.
Check official documentation
Compare the following points with the official documentation.
Review evaluation methods in Task 3.4, subgroup analysis in Task 4.1, Bedrock evaluations, and Guardrails language support. For a hypothetical factuality test with 9/10 correct English answers, 8/10 Japanese answers, and 1/2 answers in another target language, report each language's accuracy and sample count alongside pooled accuracy. Keep safety and latency in separate columns.
Official references
What to confirm
Per-language accuracy is 90% (10 cases), 80% (10 cases), and 50% (2 cases); pooled accuracy is 18/22, about 81.8%. Explain both the disparity hidden by pooling and why two cases cannot establish a stable performance ranking.
Key points
- Multilingual
- Coverage
- Language metrics
Notes
- Environment: Reference review of official documentation and a hypothetical aggregation example. No AWS resources, model invocations, or user-data collection are required.
- The numbers are hypothetical teaching data, not measured model performance or a guarantee for all languages. Pooled accuracy combines the same factuality test; it does not mix safety or latency into that score.
- Guardrails language support differs by policy and tier. A model's ability to generate a language does not establish that every safety filter supports it. Check the exact feature's official support table and test representative cases in the intended language.
- For automated evaluation or an LLM judge, check whether the evaluator can handle the language and rubric. Include review by people who understand the language, culture, and domain so that evaluator errors are not confused with model errors.
Foundation review
Per-language and pooled evaluation
Report scores and sample counts separately for each language using comparable tasks and rubrics, alongside pooled results. This prevents strong results in heavily sampled languages from hiding failures in sparsely sampled ones.
Representativeness and small samples
Include dialects, scripts, domain terms, and natural questions. One success in two cases is an observed 50%, but such a small sample cannot establish deployment performance or an inherent ranking of languages.
Appropriate refusal versus over-refusal
Measure blocking prohibited requests separately from rejecting legitimate ones. A high refusal rate alone does not establish safety or usefulness.