Helpful is not the same as correct
Language models are eager to help, which is exactly the problem. Without constraints, an agent will answer a question it should have escalated, and it will sound confident doing it.
Build the evaluation set first
Before writing prompts, collect a few hundred real conversations and label the correct outcome for each: answer, ask a clarifying question, or hand off. This set becomes the test every change must pass.
Include the awkward cases on purpose: angry customers, ambiguous requests and questions just outside the agent's remit.
Ground every answer
Agents should answer from retrieved sources and live system data, not memory. If the source does not contain the answer, the correct behaviour is to say so and escalate.
Make hand-offs explicit
Write hand-off rules as plainly as a staff handbook:
- Refunds above a set value go to a person.
- Any mention of legal action, injury or safety escalates immediately.
- Two failed attempts to resolve a request trigger a hand-off.
Keep measuring after launch
Sample live conversations every week, score them against the same rubric and add new failure cases to the evaluation set. Quality drifts when policies and products change; measurement is how you notice.
Want help applying this to your team? Book a free AI workflow audit or talk to us about your project.
