Human labels are used to train a moderation model.
Labeling instructions changed across teams.
Choose two controls.
Measure agreement and disagreement by subgroup and period, review instructions and sampling, and correct or document systematic label bias.
Detailed explanation
Systematic labeling differences become visible.
Systematic labeling differences become visible.
Bias is managed rather than hidden in the model.
Bias is managed rather than hidden in the model.
Human judgments can be inconsistent or biased.
Human judgments can be inconsistent or biased.
Different meanings are incorrectly combined.
Different meanings are incorrectly combined.
Try it yourself
An example you can run in a temporary verification environment.
AWS公式AIF-C01 Domain 4.1のラベルバイアス、公平性、ステークホルダーを確認する。Expected result
モデル入力だけでなく、正解ラベルの生成過程も公平性の対象と説明できる。Key points
- Label bias
- Agreement
- Sampling
Notes
- Environment: AWS公式AIF-C01試験ガイドとAWS公式ドキュメントの確認
- Command output formatting can vary slightly by distribution or tool version.
- Run the example in a temporary directory or process when possible.
Foundation review
Read the scope first
Check whether the command acts on the current shell, a new process, an existing process, or a file.
Verify the observable result
Use the supplied command and compare the output with the expected result.