Every support team knows the feeling of an inbox that grows faster than it drains. Tickets arrive without context, urgent issues hide among routine ones, and skilled agents spend the first minutes of each case just working out what it is about and who should handle it.
Triage is the part of support operations where AI helps most reliably. Reading a message, understanding what it is about, judging how urgent it is and sending it to the right place are classification tasks, which language models do well. Done carefully, triage automation shortens first response times, reduces misrouted tickets and frees agents to do the work that needs them.
Done carelessly, it misses an angry enterprise customer's outage report, mislabels a legal complaint as a billing question or sends a confident wrong answer to someone who needed a person. This guide shows how to build triage that is useful and safe: what to classify, how to design routing, how to draft replies responsibly, how to handle sensitive cases and how to measure results.
For the architecture questions behind this, see our guide to AI agents versus workflows. For where to place approvals, see human-in-the-loop design.
What triage actually involves
Triage is several decisions that are often bundled together. Separating them makes each one easier to build and test.
- Categorize. What is this about? Billing, access, bug report, feature request, cancellation, complaint, security concern.
- Prioritize. How urgent and how important is it? Consider impact, customer tier, deadlines and the sentiment of the message.
- Route. Which queue, team or specific person should handle it, including language and skill requirements?
- Enrich. What context should the agent see immediately? Account details, plan, recent orders, past tickets, product version.
- Detect special cases. Is this a duplicate, spam, a security issue, a legal threat, a request involving personal data or a possible outage?
- Draft. Can a useful first reply or internal summary be prepared?
Start with the first three, add enrichment next and treat drafting as a later, more careful step.
Design your taxonomy first
AI cannot rescue a muddled category list. If your agents disagree about what "Billing" and "Account" mean, the model will inherit the confusion.
Before building, do this groundwork.
Review a sample of recent tickets. Read a few hundred and note how they naturally cluster. Look at which categories agents actually use and which they avoid.
Keep the list short and mutually exclusive. Aim for a manageable number of top-level categories, each with a clear definition and examples. A small, clean set beats a long, overlapping one. You can add subcategories later.
Write definitions with examples and counterexamples. For each category, give a plain description, three typical examples and two near misses that belong elsewhere. These become part of your instructions to the model and your test set.
Include "unclear" and "multiple issues" as real options. Forcing a guess on an ambiguous message is how bad routing happens. An honest "needs human triage" outcome is a feature.
Decide which attributes are separate. Category, priority, language, sentiment and customer tier are different fields. Do not fold them into one label.
Prioritization needs more than sentiment
It is tempting to rank tickets by how angry they sound. That finds some urgent cases and misses others. A calm message reporting that checkout is failing for every customer is more urgent than a furious message about a typo.
Build priority from several signals combined by clear rules.
- Business impact. Is it a single user, a team or a system-wide problem? Is revenue or a deadline affected?
- Customer context. Plan, contract terms, service-level commitments and account health. These come from your systems, not from the model.
- Time sensitivity. Mentions of deadlines, expirations, launches or legal dates.
- Severity cues. Words and patterns that indicate outage, data loss, security exposure or safety.
- Repeat contact. Customers who have written several times about the same issue.
- Sentiment. A useful signal, but only one among several.
Let the model extract the signals from the text, and let deterministic rules combine them with account data into a priority. That keeps the logic auditable. If a customer's contract guarantees a response time, the rule enforces it, no matter what the model thinks about the tone of the message.
Routing that survives real life
Routing is the part customers feel most directly. A misrouted ticket adds delay, bounces between teams and frustrates everyone.
Route on skills and capacity, not only on topic. A billing question in Spanish should reach a billing agent who speaks Spanish. Build routing tables that include language, product area, tier and current workload.
Prefer the specific over the general. Send directly to the right specialist queue when confidence is high, and to a general triage queue when it is not.
Handle follow-ups. A reply to an existing ticket should stay with the same owner where possible, not be re-triaged as new.
Use fallbacks. If the model is unavailable, slow or returns an invalid answer, tickets should flow to a default queue and not stall. Design for failure from the start.
Limit automatic re-routing loops. A ticket bouncing between teams three times should be flagged for a coordinator.
Cases that need a human immediately
Some messages should never wait in a normal queue or receive an automated draft. Define these explicitly, and bias toward catching too many rather than too few.
- Security and privacy. Reports of compromised accounts, data exposure or requests to access, correct or delete personal data.
- Legal and compliance. Threats of legal action, regulator contact, subpoenas and formal complaints.
- Safety. Anything involving physical harm or emergencies, which should follow a defined escalation path.
- High-value accounts and outages. Reports that could indicate a widespread incident, or that come from your most critical customers.
- Vulnerable situations. Messages indicating distress, bereavement or financial hardship where tone and judgment matter.
- Press and public figures. Anything that could become a reputational issue.
For these, the AI's role is to detect and escalate quickly, with a short summary to help the person who picks it up. It should not draft a customer-facing reply or resolve the issue.
Drafting replies without creating risk
Reply drafting is where teams see the biggest time savings and where mistakes are most visible. A few design principles keep it safe.
Draft for the agent, not for the customer. In the early stages, every draft goes to an agent who edits and sends it. Automatic replies come later, if ever, and only for narrow, low-risk categories.
Ground answers in approved content. Retrieve from your help center, policies and macros, and have the model answer using only that material. Show the agent which sources were used. If no good source is found, the draft should say so and not improvise.
Never invent facts. Prices, dates, policy terms, refund eligibility and technical claims must come from retrieved content or from your systems. Instruct the model to leave a clear placeholder when it does not know, and make that placeholder impossible to miss so it cannot be sent by accident.
Do not make promises. Refunds, credits, timelines and exceptions should be decisions made by authorized people or by explicit policy rules, not by the model's phrasing.
Match your voice. Provide examples of your preferred tone, length and structure. Consistent style makes drafts easier to approve and customers more comfortable.
Respect privacy. Limit what customer data is sent to the model to what is needed. Check your vendor's data handling terms and your own privacy commitments, and avoid placing sensitive data in prompts unnecessarily.
Disclose automation honestly. If a customer is interacting directly with an automated assistant, do not pretend otherwise, and provide an easy route to a person.
Summaries and context for agents
Often the most useful thing AI does is not the reply but the briefing. For long threads, a short summary of the issue, what has been tried and what the customer wants saves minutes on every handoff.
A good internal summary includes the customer's goal, the key facts, the steps already taken, outstanding questions and any commitments made. Keep it factual and mark uncertainty. Summaries should be clearly labeled as generated, and agents should be able to see the original thread easily.
Building the evaluation set
You cannot know if triage works without measuring against known answers. Build an evaluation set before you launch.
Collect real tickets. Take a few hundred from different periods, channels, languages and categories. Include the unusual ones.
Label them carefully. Have experienced agents assign the correct category, priority and route. Where agents disagree, resolve it and refine your definitions. The disagreement is telling you something about your taxonomy.
Include hard cases. Short messages, multiple issues in one email, sarcasm, forwarded threads, attachments and messages in mixed languages.
Hold some back. Keep a portion untouched so you can test changes without fooling yourself.
Refresh regularly. Products, customers and language change. Add new examples, especially failures, every month.
Metrics that matter
Measure the system and the outcome, not just the model.
Classification quality. Accuracy per category, and precision and recall for the categories that matter most. For escalation categories like security and legal, recall is the priority: missing one is far worse than flagging a few extra.
Routing quality. Percentage of tickets reaching the right team first time, and number of reassignments per ticket.
Speed. Time to first response and time to resolution, broken down by priority.
Agent experience. How often drafts are used unchanged, edited or discarded, and how long agents spend per ticket. Ask them directly what is helping and what is not.
Customer outcomes. Satisfaction scores, reopen rates and complaints. Speed that harms quality is not an improvement.
Safety. Count escalation misses, drafts with incorrect facts caught in review and any incidents. Review every serious one.
Compare against a baseline from before the change, and watch for differences across languages, channels and customer tiers. A system that works well on English email but poorly on chat or other languages will quietly create unequal service.
Rollout in stages
- Shadow. The AI classifies in the background while humans triage as normal. Compare results and fix the taxonomy.
- Suggest. Show the suggested category, priority and route to the agent or coordinator, who accepts or changes it with one click.
- Auto-route the easy cases. For categories with consistently high accuracy and low risk, route automatically, with sampling to verify quality. Keep escalation detection active throughout.
- Assisted drafting. Introduce draft replies for a few well-understood categories, always reviewed by an agent.
- Expand carefully. Add categories and languages one at a time based on evidence.
At each stage, define what must be true to move forward and what triggers a step back.
Common mistakes
Starting with auto-replies. Begin with routing and agent assistance. Customer-facing automation is the highest-risk step.
Ignoring the taxonomy. Fix categories before blaming the model.
Using sentiment as priority. Combine impact, customer data and deadlines.
Letting the model state policies from memory. Ground every claim in approved content.
Forgetting non-English and non-email channels. Test each one separately.
Measuring speed only. Pair speed with quality and satisfaction.
No feedback loop. Capture agent corrections and use them to improve instructions and test sets.
Key takeaways
- Triage splits into categorizing, prioritizing, routing, enriching, special-case detection and drafting. Build and test them separately.
- Clean, short, well-defined categories matter more than model choice.
- Combine extracted signals with account data through deterministic rules to set priority.
- Detect and immediately escalate security, legal, safety, vulnerable and critical-account cases. Do not auto-draft for them.
- Draft for agents, ground answers in approved content and never let the model invent facts or make promises.
- Build an evaluation set from real tickets, track recall on escalation categories and roll out in stages.
When AI handles the sorting, agents spend their time on what people do best: understanding a customer's situation and solving the problem.
A note on sources
This is a framework article. It is built on common operating practice and does not cite outside statistics or studies. The templates and scoring methods here are starting points, so test them against your own data and adapt them before you rely on them. If you spot something that looks wrong, tell us.