Prompt template
You are helping {{audience}}.
Task: {{one clear instruction}}.
Rules:
- at most 3 sentences
- if the answer is not in <documents>, say "not stated"
<documents>{{material}}</documents>
<question>{{question}}</question>
Return JSON: {"answer": "...", "source": "..."}Few-shot examples
<example>
<ticket>I was charged twice</ticket>
<answer>{"category": "billing"}</answer>
</example>Use 3 to 5 varied, realistic examples with identical format, including a borderline case and an "unknown" case.
Reasoning
Think step by step inside <thinking> tags.
Then give the final result in <answer> tags.import re
m = re.search(r"<answer>(.*?)</answer>", reply, re.S)
answer = m.group(1).strip()System prompt layout
<role>...</role>
<rules>...</rules>
<documents>...</documents>
<question>...</question>Long documents first, question last. Keep tag names consistent.
JSON schema
{"type": "object",
"properties": {"category": {"type": "string", "enum": ["billing", "account", "unknown"]}},
"required": ["category"], "additionalProperties": false}Methods: prompt only, forced tool use, schema-constrained output. Use enums and an explicit unknown value.
Validate and retry
for attempt in range(3):
reply = call_model(prompt, feedback)
try:
data = json.loads(reply)
validate(data, SCHEMA)
return data
except Exception as err:
feedback = "Rejected: " + str(err)
raise RuntimeError("failed validation")Version and test
prompts/triage/v7.txt # changed: added unknown categoryRun the evaluation set in CI, pin the model version, log the prompt version with each request, and release gradually.
Debug
python run_evals.py --case pii-014 --runs 10Reproduce, reduce the prompt, change one thing, re-run the whole set, and add the case to the tests.