System Design: Routing Models to Cut AI Cost in Half
Wellhound's app includes an AI care assistant that answers questions about a dog's results, explains biomarkers in plain language, and helps owners act on a vet's care plan. Margins on a preventive health membership are thin, so AI cost had to be designed, not discovered on the invoice.
The goal
Keep AI spend under about $1 per member per month without making the assistant worse.
One interface, many models
Every AI call in the app goes through a single internal LLM interface. The model for each task is set by configuration, not hard-coded. That one decision makes everything else possible: you can swap, test, and downgrade models per task without touching product code.
Routing by task
• Routine questions (what does this value mean, when is my next draw) go to Claude Haiku
• Complex, tool-using workflows (multi-step actions, reasoning across a dog's full record) go to Claude Sonnet
• Templated feedback that does not need a model at all, does not use one
• Care plan drafts, which a vet signs off on, stay on the strongest model
About 75% of requests route to Haiku, which cut estimated inference cost by about 50%.
Guardrails before cost
A cheaper model is only acceptable if it passes the same bar. Every task has a golden set (expected good answers) and a red-team set (things the assistant must refuse or route to a vet). A model is promoted to a task only after it passes both. The same process lets open-weight models be evaluated over time.
The assistant is also scoped. It explains, summarizes, and helps owners act, but it does not give veterinary advice. Clinical questions route to a vet, and care plans drafted by AI always carry the vet's final say.
What I took from it
• Put a seam between your product and your model provider on day one
• Route by task difficulty, not by habit
• Evaluation sets are what make cost-cutting safe
• In health, the guardrails are part of the product, not a layer on top