A team creates versions 1 and 2 of a summarization prompt in Amazon Bedrock Prompt Management. The model and inference parameters are the same; only the instructions change.
Compare shows the configuration differences. Before switching production to version 2, check whether required summary items and the output format are preserved for ordinary documents, long documents, and ambiguous inputs.
Which comparison should the team perform next?
Configuration comparison and output evaluation have different roles. Use common representative inputs and criteria to assess ordinary and edge cases, associating results with their versions. Console test-variable values are not saved when the prompt is saved, so maintain evaluation inputs separately.
Detailed explanation
Different input difficulty and selective results make the effect of the prompt change difficult to assess.
Inputs of different difficulty and selectively chosen successes do not provide a consistent comparison. Common cases and criteria, with failures included, provide a more useful basis for the adoption decision.
Evaluating outputs under common conditions and criteria reveals quality effects that configuration differences alone cannot establish.
Start with the definition difference shown by Compare, then run the same documents through each version and assess required items and format. Preserve evaluation inputs separately from temporary console values and associate results with the version used.
Instruction changes can affect responses, so the configuration difference alone does not establish that output requirements are met.
Even a single instruction change can affect priorities or what gets summarized. Inspecting a JSON change does not establish that ordinary and edge-case outputs still meet requirements.
Instructions can change outputs even with the same model, so model identity is not a reason to skip evaluation.
Matching the model identifier is one comparison condition. Instructions are also inputs to the model, so model identity alone does not guarantee that summary and format requirements remain satisfied.
Check official documentation
Compare the following points with the official documentation.
Read AIF-C01 Task 3.2.5, Compare versions, and Test a prompt. Separate definition comparison, running with Test variables, and preserving evaluation inputs.
Official references
What to confirm
Explain why a JSON difference cannot establish quality, why both versions need common inputs and criteria, and why temporary test-variable values need separate records.
Key points
- Definition versus output
- Common inputs and criteria
- Temporary test values
Notes
- Environment: Reference-based review of current AIF-C01 Task 3.2 and Amazon Bedrock documentation using invented configurations; no AWS accounts, API calls, or model inference.
- Distinguish the JSON comparison from Run prompt with test variables. This is a documentation-based exercise and does not run model inference.
- Check the note that Test variables are not saved with a prompt. Do not assume Prompt Management automatically performs every evaluation and approval.
Foundation review
Configuration differences do not establish quality
Compare displays the JSON definitions of versions side by side to inspect configuration differences. To examine output responses, supply test-variable values, run the versions, and assess the results separately.
Use common inputs and criteria
For this instruction-only change, keep model and inference settings aligned and run both versions on ordinary and failure-prone inputs. Assess missing required items and format violations consistently rather than relying on one successful example.
Preserve evaluation inputs as records
Console Test variables are temporary and are not saved when the prompt is saved. Maintain inputs, version identifiers, outputs, and criteria in evaluation records instead of assuming that creating a prompt version also stores the evaluation set.