LLM Routing: The Secret Architecture Cutting AI Costs by 60%

Imagine a world where every user request is sent to the perfect LLM for the task: GPT-4o mini for simple questions, Claude 3.5 Sonnet for complex analysis, Mistral Large for French-language tasks. Without the user noticing any difference.

That's exactly what LLM routing does. And companies adopting it are cutting costs by 50 to 70% with less than 2% quality loss.

TL;DR

• LLM routing = send each request to the best-suited model (optimal quality/cost ratio)
• 3 main strategies: complexity-based routing, cascade routing, consensus routing
• Typical savings: 50-70% reduction in LLM costs with < 2% quality degradation
• Tools: Martian, Unify, OpenRouter for managed routing; custom classifier for full control
• For RAG: combine LLM routing with prompt caching for > 80% savings

Why LLM Routing is Essential

The problem: one model for everything

Most companies use a single model for all requests:

All requests → GPT-4o → Responses
               $$$$$

The problem? 80% of requests are simple and don't need the power (and cost) of a GPT-4o.

Typical request distribution

The financial impact

For 100,000 requests/month:

The 3 Routing Strategies

Strategy 1: Complexity-based routing

The most common. A classifier analyzes the query and sends it to the right model.

                    ┌─────────────────┐
                    │   Classifier    │
                    │  (complexity)   │
                    └───┬──────┬──────┘
                        │      │
              ┌─────────┼──────┼──────────┐
              ▼         ▼      ▼          ▼
         ┌────────┐ ┌──────┐ ┌──────┐ ┌───────┐
         │Trivial │ │Simple│ │Medium│ │Complex│
         │(rule)  │ │(mini)│ │(std) │ │(pro)  │
         └────────┘ └──────┘ └──────┘ └───────┘
         $0         $0.001   $0.005   $0.03

Implementation:

DEVELOPERpython
from openai import OpenAI
import anthropic

client_openai = OpenAI()
client_anthropic = anthropic.Anthropic()

# Complexity classifier (itself a lightweight LLM)
CLASSIFIER_PROMPT = """Analyze the complexity of this user query.
Respond ONLY with one of the following levels:
- TRIVIAL: greetings, thanks, confirmations
- SIMPLE: direct factual question, basic info
- MEDIUM: comparison, synthesis, multi-faceted question
- COMPLEX: analysis, long-form writing, multi-step reasoning
- EXPERT: complex creative task, data analysis, legal

Query: {query}
Level:"""

async def classify_complexity(query: str) -> str:
    """Classify complexity with a lightweight model."""
    response = client_openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "user", "content": CLASSIFIER_PROMPT.format(query=query)}
        ],
        max_tokens=10,
        temperature=0,
    )
    return response.choices[0].message.content.strip()

# Complexity → model mapping
MODEL_ROUTING = {
    "TRIVIAL": None,  # Template response, no LLM
    "SIMPLE": {"provider": "openai", "model": "gpt-4o-mini"},
    "MEDIUM": {"provider": "anthropic", "model": "claude-3-5-haiku-20241022"},
    "COMPLEX": {"provider": "openai", "model": "gpt-4o"},
    "EXPERT": {"provider": "anthropic", "model": "claude-sonnet-4-20250514"},
}

TRIVIAL_RESPONSES = {
    "hello": "Hello! How can I help you?",
    "thanks": "You're welcome! Don't hesitate if you have more questions.",
}

async def route_and_generate(query: str, context: str) -> dict:
    """Route the query to the right model and generate a response."""
    complexity = await classify_complexity(query)
    config = MODEL_ROUTING[complexity]

    if config is None:
        # Trivial request → template response
        for keyword, response in TRIVIAL_RESPONSES.items():
            if keyword in query.lower():
                return {"response": response, "model": "template", "cost": 0}

    if config["provider"] == "openai":
        response = client_openai.chat.completions.create(
            model=config["model"],
            messages=[
                {"role": "system", "content": f"Context:\n{context}"},
                {"role": "user", "content": query},
            ],
        )
        return {
            "response": response.choices[0].message.content,
            "model": config["model"],
            "cost": estimate_cost(response.usage, config["model"]),
        }
    elif config["provider"] == "anthropic":
        response = client_anthropic.messages.create(
            model=config["model"],
            max_tokens=2048,
            system=f"Context:\n{context}",
            messages=[{"role": "user", "content": query}],
        )
        return {
            "response": response.content[0].text,
            "model": config["model"],
            "cost": estimate_cost_anthropic(response.usage, config["model"]),
        }

Strategy 2: Cascade routing

Start with the cheapest model. If confidence is too low, escalate to a more powerful model.

Query → GPT-4o mini → Confidence > 80%? → Yes → Response ✓
                           │
                           No
                           ↓
              Claude 3.5 Sonnet → Confidence > 80%? → Yes → Response ✓
                                       │
                                       No
                                       ↓
                              GPT-4o → Response (forced) ✓

Implementation:

DEVELOPERpython
import re

CASCADE_MODELS = [
    {"provider": "openai", "model": "gpt-4o-mini", "threshold": 0.8},
    {"provider": "anthropic", "model": "claude-sonnet-4-20250514", "threshold": 0.7},
    {"provider": "openai", "model": "gpt-4o", "threshold": 0.0},  # Fallback
]

CONFIDENCE_PROMPT_SUFFIX = """

After your response, add on the last line:
CONFIDENCE: X.XX (between 0 and 1, your confidence in the answer)"""

async def cascade_route(query: str, context: str) -> dict:
    """Cascade routing with confidence evaluation."""
    for config in CASCADE_MODELS:
        response, confidence = await generate_with_confidence(
            query, context, config
        )

        if confidence >= config["threshold"]:
            return {
                "response": response,
                "model": config["model"],
                "cascade_level": CASCADE_MODELS.index(config) + 1,
            }

    # Fallback: last response
    return {
        "response": response,
        "model": CASCADE_MODELS[-1]["model"],
        "cascade_level": len(CASCADE_MODELS),
    }

async def generate_with_confidence(query, context, config):
    """Generate a response and extract the confidence score."""
    full_query = query + CONFIDENCE_PROMPT_SUFFIX

    if config["provider"] == "openai":
        resp = client_openai.chat.completions.create(
            model=config["model"],
            messages=[
                {"role": "system", "content": f"Context:\n{context}"},
                {"role": "user", "content": full_query},
            ],
        )
        text = resp.choices[0].message.content
    else:
        resp = client_anthropic.messages.create(
            model=config["model"],
            max_tokens=2048,
            system=f"Context:\n{context}",
            messages=[{"role": "user", "content": full_query}],
        )
        text = resp.content[0].text

    # Extract confidence
    confidence_match = re.search(r"CONFIDENCE:\s*([\d.]+)", text)
    confidence = float(confidence_match.group(1)) if confidence_match else 0.5
    clean_response = re.sub(r"\nCONFIDENCE:.*$", "", text).strip()

    return clean_response, confidence

Strategy 3: Consensus routing

For critical queries: query multiple models and take the majority response or combine them.

Query → ┌─ GPT-4o mini ──────────┐
        ├─ Claude 3.5 Haiku ─────┤→ Comparator → Final response
        └─ Mistral Small ────────┘

Implementation:

DEVELOPERpython
import asyncio

async def consensus_route(query: str, context: str) -> dict:
    """Query 3 models and combine responses."""
    models = [
        {"provider": "openai", "model": "gpt-4o-mini"},
        {"provider": "anthropic", "model": "claude-3-5-haiku-20241022"},
        {"provider": "openai", "model": "gpt-4o-mini"},
    ]

    # Parallel calls
    tasks = [generate(query, context, m) for m in models]
    responses = await asyncio.gather(*tasks)

    # Combine responses
    combined = await synthesize_responses(query, responses)

    return {
        "response": combined,
        "models_used": [m["model"] for m in models],
        "individual_responses": responses,
    }

async def synthesize_responses(query, responses):
    """One model synthesizes the others' responses."""
    synthesis_prompt = f"""Here are 3 responses to the same question.
Synthesize the best answer by combining the strengths of each.
If responses contradict each other, favor the consensus.

Question: {query}

Response 1: {responses[0]}
Response 2: {responses[1]}
Response 3: {responses[2]}

Synthesis:"""

    resp = client_openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": synthesis_prompt}],
    )
    return resp.choices[0].message.content

Routing Solution Comparison

Managed routing tools

Martian: The ML router

DEVELOPERpython
import requests

def route_with_martian(query: str, context: str) -> dict:
    """Routing via Martian - automatic best model selection."""
    response = requests.post(
        "https://api.withmartian.com/v1/chat/completions",
        headers={"Authorization": "Bearer MARTIAN_API_KEY"},
        json={
            "messages": [
                {"role": "system", "content": context},
                {"role": "user", "content": query},
            ],
            "model": "router",  # Martian chooses the model
            "max_tokens": 1024,
            "route_params": {
                "max_cost": 0.01,  # Max budget per request
                "min_quality": 0.85,  # Minimum quality
            }
        }
    )

    result = response.json()
    return {
        "response": result["choices"][0]["message"]["content"],
        "model_used": result["model"],
        "cost": result["usage"]["total_cost"],
    }

OpenRouter: The universal aggregator

DEVELOPERpython
from openai import OpenAI

# OpenRouter is OpenAI SDK compatible
client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key="OPENROUTER_API_KEY",
)

def route_with_openrouter(query: str, context: str) -> dict:
    """Routing via OpenRouter with price-based selection."""
    response = client.chat.completions.create(
        model="openrouter/auto",  # Automatic routing
        messages=[
            {"role": "system", "content": context},
            {"role": "user", "content": query},
        ],
        max_tokens=1024,
        extra_body={
            "route": "cost",  # Optimize for cost
            # Options: "cost", "quality", "balanced"
        }
    )
    return {
        "response": response.choices[0].message.content,
        "model_used": response.model,
    }

Unify: The benchmarker

DEVELOPERpython
import unify

client = unify.Unify(api_key="UNIFY_API_KEY")

def route_with_unify(query: str, context: str) -> dict:
    """Routing via Unify - benchmark-driven."""
    response = client.generate(
        messages=[
            {"role": "system", "content": context},
            {"role": "user", "content": query},
        ],
        model="router",
        routing_strategy="lowest_cost",  # or "highest_quality", "balanced"
    )
    return {
        "response": response,
        "model_used": client.last_model_used,
    }

LLM Routing for RAG

RAG architecture with routing

┌────────────────────────────────────────────────────────┐
│                  RAG + LLM ROUTING                      │
├────────────────────────────────────────────────────────┤
│                                                         │
│  User query                                             │
│       │                                                 │
│       ▼                                                 │
│  ┌──────────────┐                                      │
│  │  Classifier  │  ← Analyze complexity + intent        │
│  │  (GPT-4o     │                                      │
│  │   mini)      │                                      │
│  └──────┬───────┘                                      │
│         │                                               │
│    ┌────┴────┐                                          │
│    ▼         ▼                                          │
│  Simple    Complex                                      │
│    │         │                                          │
│    ▼         ▼                                          │
│  ┌──────┐  ┌──────────┐                                │
│  │Light │  │  Full    │                                │
│  │ RAG  │  │  RAG     │                                │
│  │(top3)│  │  (top10  │                                │
│  └──┬───┘  │  +rerank)│                                │
│     │      └────┬─────┘                                │
│     ▼           ▼                                      │
│  GPT-4o     Claude 3.5                                  │
│  mini       Sonnet                                      │
│  ($0.001)   ($0.015)                                    │
│     │           │                                      │
│     └─────┬─────┘                                      │
│           ▼                                             │
│      Final response                                     │
└────────────────────────────────────────────────────────┘

Combining routing with prompt caching

The ultimate cost reduction combo:

Metrics and Monitoring

Routing KPIs

DEVELOPERpython
# Essential metrics to track
routing_metrics = {
    # Distribution
    "requests_per_model": "Counter per model",
    "complexity_distribution": "Histogram of levels",

    # Quality
    "user_satisfaction_per_model": "Average score per model",
    "cascade_escalation_rate": "% of escalated requests",
    "consensus_disagreement_rate": "% of disagreements",

    # Costs
    "cost_per_request_avg": "Average cost per request",
    "cost_savings_vs_single_model": "Savings vs single model",
    "classifier_cost_overhead": "Classifier overhead cost",

    # Performance
    "routing_latency_p50": "< 50ms",
    "total_latency_p50": "< 2s",
    "cache_hit_rate": "> 70%",
}

Typical dashboard

LLM Routing ROI

ROI calculator

Implementation time

Ailog and LLM Routing

Ailog's RAG pipeline natively integrates intelligent routing:

• Automatic classifier that analyzes the complexity of each question
• Multi-model with automatic fallback on errors
• Prompt caching combined for maximum savings
• Dashboard tracking costs and quality per model

Also discover prompt caching to go further with optimization, or explore RAG caching strategies to reduce latency.

FAQ

Does LLM routing add latency?

Yes, but very little. The classifier typically adds 30-80ms (one GPT-4o mini call with 50 tokens). For a total latency budget of 2-3 seconds, it's negligible. The trick: use a rule-based classifier (regex, length) for trivial cases, and an LLM classifier only for ambiguous cases.

How do I measure quality to adjust routing?

Three complementary approaches: (1) user feedback (thumbs up/down), (2) LLM-as-judge that scores responses out of 10, (3) RAG metrics (faithfulness, relevance) with frameworks like RAGAS. Compare scores per model to adjust routing thresholds.

Should I use a managed router or build custom?

To start, a managed router (OpenRouter, Martian) is sufficient: set up in 1 day, no maintenance. When your volume exceeds 500K requests/month or you have specific needs (private models, ultra-low latency), switch to a custom classifier.

Does routing work with streaming?

Yes. The classifier decides on the model before generation. The response is then streamed from the chosen model as usual. The only constraint: cascade routing with confidence evaluation is incompatible with pure streaming (you need the complete response to evaluate confidence).

What's the risk of quality degradation?

With well-calibrated routing, degradation is minimal: < 2% on average. The main risk is a poorly trained classifier sending complex queries to a lightweight model. Solution: start conservative (high thresholds) and gradually lower them while monitoring quality.

---

Ready to cut your AI costs by 3x? Create your Ailog account and benefit from intelligent LLM routing integrated into your RAG pipeline, hosted in France.