Send a long Japanese document to a model.
Account for tokens rather than assuming characters equal tokens.
Choose two correct statements.
Models process token units whose mapping varies by tokenizer and language; long inputs must fit the context limit or be split or summarized.
Detailed explanation
Tokens may be words, subwords, or characters depending on the tokenizer.
Tokens may be words, subwords, or characters depending on the tokenizer.
The input cannot be processed unchanged beyond the limit.
The input cannot be processed unchanged beyond the limit.
The tokenizer determines the mapping.
The tokenizer determines the mapping.
Truncation or failure can discard information.
Truncation or failure can discard information.
Try it yourself
An example you can run in a temporary verification environment.
Amazon Bedrock公式モデル仕様とAIF-C01 Domain 2.2のトークン・コンテキスト制限を確認する。Expected result
トークン化、コンテキスト上限、長文対策を説明できる。Key points
- Tokens
- Context
- Long-input handling
Notes
- Environment: AWS公式AIF-C01試験ガイドとAWS公式ドキュメントの確認
- Command output formatting can vary slightly by distribution or tool version.
- Run the example in a temporary directory or process when possible.
Foundation review
Read the scope first
Check whether the command acts on the current shell, a new process, an existing process, or a file.
Verify the observable result
Use the supplied command and compare the output with the expected result.