Send a long Japanese document to a model.
Account for tokens rather than assuming characters equal tokens.
Choose two correct statements.
Models process token units whose mapping varies by tokenizer and language; long inputs must fit the context limit or be split or summarized.
Detailed explanation
Tokens may be words, subwords, or characters depending on the tokenizer.
The input cannot be processed unchanged beyond the limit.
Truncation or failure can discard information.
The tokenizer determines the mapping.
Check official documentation
Compare the following points with the official documentation.
Amazon Bedrock公式モデル仕様とAIF-C01 Domain 2.2のトークン・コンテキスト制限を確認する。
Official references
What to confirm
トークン化、コンテキスト上限、長文対策を説明できる。
Key points
- Tokens
- Context
- Long-input handling
Notes
- Environment: AWS公式AIF-C01試験ガイドとAWS公式ドキュメントの確認
- Compare the definition with the AIF-C01 objective and the service documentation before applying it.
- Keep the concept separate from a particular answer choice or implementation step.
Foundation review
Character count is different
The same number of characters can produce different token counts across languages and text.
Capacity planning
Input and output compete for the model's context capacity and can affect cost and latency.