← Back

System Design: Routing Models to Cut AI Cost in Half

Wellhound's app includes an AI care assistant that answers questions about a dog's results, explains biomarkers in plain language, and helps owners act on a vet's care plan. Margins on a preventive health membership are thin, so AI cost had to be designed, not discovered on the invoice.

The goal

Keep AI spend under about $1 per member per month without making the assistant worse.

One interface, many models

Every AI call in the app goes through a single internal LLM interface. The model for each task is set by configuration, not hard-coded. That one decision makes everything else possible: you can swap, test, and downgrade models per task without touching product code.

Routing by task

• Routine questions (what does this value mean, when is my next draw) go to Claude Haiku
• Complex, tool-using workflows (multi-step actions, reasoning across a dog's full record) go to Claude Sonnet

• Templated feedback that does not need a model at all, does not use one

• Care plan drafts, which a vet signs off on, stay on the strongest model

About 75% of requests route to Haiku, which cut estimated inference cost by about 50%.

Guardrails before cost

A cheaper model is only acceptable if it passes the same bar. Every task has a golden set (expected good answers) and a red-team set (things the assistant must refuse or route to a vet). A model is promoted to a task only after it passes both. The same process lets open-weight models be evaluated over time.

The assistant is also scoped. It explains, summarizes, and helps owners act, but it does not give veterinary advice. Clinical questions route to a vet, and care plans drafted by AI always carry the vet's final say.

What I took from it

• Put a seam between your product and your model provider on day one
• Route by task difficulty, not by habit

• Evaluation sets are what make cost-cutting safe

• In health, the guardrails are part of the product, not a layer on top