Stop sending every AI request to your most expensive model. See how intelligent routing can cut cost and latency without sacrificing much quality.

Most production AI agents still send every LLM call to the same expensive frontier model. Classification steps, simple tool calls, progress checks, and hard reasoning all hit the same endpoint. The result is unnecessary cost and latency. NVIDIA NeMo Switchyard solves this.

It is an open-source routing layer (proxy + library) that sits between your agent and the models. It decides, request by request or turn by turn, which model should handle the work. In this tutorial, we'll build a working two-model router and gradually move from random routing to content-aware routing. So, let's get started.

What Exactly Does Switchyard Do?

A normal LLM application might look like this:

Application

|

v

GPT / Claude / Local LLM

Switchyard adds a routing layer:

Application

|

v

Switchyard

/ \

v v

Cheap Powerful

Model Model

The application does not need to know which upstream model ultimately serves the request. Switchyard selects the actual target and forwards the request. Let's see this practically.

Step 1: Installing Switchyard

For the CLI/server path, the project documentation provides a uv installation route:

uv tool install "nemo-switchyard[cli,server]"

Verify the installation:

switchyard --version

Output:

switchyard 0.2.0

nemo-switchyard v0.2.0

Alternatively, the native Rust server can be installed directly with Cargo:

cargo install --locked switchyard-server

For this tutorial, we'll route models through OpenRouter, so export your API key:

export OPENROUTER_API_KEY="your-key-here"

Do not store the API key directly in the configuration file.

Step 2: Understanding a Switchyard Configuration

Let's start with the simplest possible setup: two models and random routing.

Create a YAML file named routes.random.yaml

and add:

defaults:

base_url: https://openrouter.ai/api/v1

api_key: ${OPENROUTER_API_KEY}

routes:

ab-test:

type: random_routing

strong:

model: openai/gpt-4o

weak:

model: openai/gpt-4o-mini

strong_probability: 0.3

rng_seed: 42

fallback_target_on_evict: weak

The key setting is:

strong_probability: 0.3

Switchyard interprets this as roughly:

30% -> strong model

70% -> weak model

Random routing is not intelligent routing, but it is useful for A/B tests and for validating the proxy before introducing a classifier. fallback_target_on_evict

is required for this route type and refers to a tier ID such as strong

or weak

.

Step 3: Starting the Routing Server

Start Switchyard with:

switchyard serve \

-c routes.random.yaml \

--host 127.0.0.1 \

--port 4000

There is no --dry-run

option in the tested serve

CLI. Starting the server is effectively the validation step: an invalid routing bundle fails during startup. You can verify that the proxy is alive with:

curl -s http://127.0.0.1:4000/health

Output:

{"status":"ok"}

Step 4: Sending a Request Through the Router

Now send an OpenAI-compatible request:

curl http://localhost:4000/v1/chat/completions \

-H "Content-Type: application/json" \

-d '{"model":"ab-test","messages":[{"role":"user","content":"Explain gradient descent in simple terms."}]}'

Notice this field:

"model": "ab-test"

Your client is not asking for a specific model (gpt-4o or gpt-4o-mini). Switchyard chooses the actual model. For this example, the request landed on the weak tier:

"model": "openai/gpt-4o-mini",

"usage": { "prompt_tokens": 14, "completion_tokens": 247, "cost": 0.0001503 }

Response:

Gradient descent is a method used in optimization to find the minimum of a function. Imagine you're on a hilly landscape, and your goal is to get to the lowest point in the valley. Here's how it works, step by step:

1) Start at a Random Point: You begin at a random location on the hill.

2) Find the Slope: You look around and determine the steepness of the hill (the gradient) at your current location. This tells you which direction is downhill.

3) Take a Step Downhill: You take a step in the direction that goes down the steepest slope. The length of your step is called the "learning rate" — if you take small steps, you're cautious, while larger steps will get you there faster but might lead you off course.

4) Repeat: You keep repeating this process, recalculating the slope and stepping down until you can't go any lower — that's the bottom of the valley or the minimum of the function.

In simple terms, gradient descent is about marching down the hill step by step until you reach the lowest point. It's widely used in machine learning to adjust models so they make better predictions.

Step 5: Upgrading to Intelligent Routing

Random routing is great for experiments, but suppose we want this behavior:

Simple request — cheap model

Hard request — strong model

Switchyard provides a classifier

route for exactly this purpose. The classifier estimates whether the weaker model can solve the task, then applies a configured threshold. Create routes.smart.yaml

and write this configuration:

defaults:

base_url: https://openrouter.ai/api/v1

api_key: ${OPENROUTER_API_KEY}

routes:

smart:

type: deterministic

classifier:

model: openai/gpt-4o-mini

strong:

model: openai/gpt-4o

weak:

model: openai/gpt-4o-mini

profile: general

session_affinity: true

fallback_target_on_evict: weak

And start it:

switchyard serve \

-c routes.smart.yaml \

--host 127.0.0.1 \

--port 4000

Now there are three roles:

classifier

|

| predicts weak-model capability

v

+-------------------+

| Should weak solve?|

+-------------------+

/ \

/ \

yes no

| |

v v

weak strong

The classifier produces a structured estimate containing a value called p_solve

: an estimate of the probability that the weak model can successfully complete the request.

Step 6: Testing the Smart Route

Try an easy question:

curl http://localhost:4000/v1/chat/completions \

-H "Content-Type: application/json" \

-d '{

"model": "smart",

"messages": [

{

"role": "user",

"content": "What is 15% of 200?"

}

]

}'

Output:

To find 15% of 200, you can multiply 200 by 0.15:

200 × 0.15 = 30

So, 15% of 200 is 30.

Then try a harder one:

curl http://localhost:4000/v1/chat/completions \

-H "Content-Type: application/json" \

-d '{

"model": "smart",

"max_tokens": 1500,

"messages": [

{

"role": "user",

"content": "Find the race condition in a distributed job queue where workers acquire leases using non-transactional Redis operations, then propose a failure-safe redesign."

}

]

}'

Output:

In a distributed job queue system using Redis to manage and lease

jobs to workers, race conditions can occur if multiple workers

attempt to acquire a lease for the same job simultaneously using

non-transactional operations. This can lead to multiple workers

incorrectly believing they have successfully acquired the lease,

resulting in duplicate processing of the same job.

Typical Race Condition Scenario

...

By incorporating these redesign elements into the distributed job

queue architecture, race conditions can be significantly reduced

and job leases can be handled more reliably and safely.

We didn't hard-code the model selection here. Instead, the classifier determines the appropriate tier for each prompt and routes the request accordingly. If you look at the logs, you can see which model was ultimately selected for each request.

| Prompt | Served Model | Tier | Latency |

|---|---|---|---|

| "What is 15% of 200?" | openai/gpt-4o-mini |

weak | 1,428 ms |

| Redis race-condition redesign | openai/gpt-4o |

strong | 4,475 ms |

Step 7: Routing Coding Agents Based on Their Progress

Prompt difficulty is not the only useful routing signal.

Consider a coding agent working for 30 turns. It may spend early turns exploring files, debugging failures, and reasoning about architecture. Later turns may simply apply an established plan or make repetitive edits. Using the strongest model for every turn wastes inference budget. Switchyard's stage_router

is designed for this kind of multi-turn workload. It uses conversation and tool-result signals to decide whether a turn should go to a capable or efficient tier.

You can create a configuration like this:

routes:

stage:

type: stage_router

strong:

model: openai/gpt-4o

weak:

model: openai/gpt-4o-mini

picker: efficient_first

confidence_threshold: 0.5

signal_recent_window: 3

fallback_target_on_evict: weak

The idea is:

Agent turn

|

v

Recent progress / failure signals

|

v

Is extra capability useful now?

/ \

/ \

weak strong

Here, the router looks for signals associated with things such as errors, repeated unproductive behavior, exploration, and recent productive changes. The goal is to reserve the stronger model for turns where extra capability appears useful.

Step 8: Escalating Only After the Weak Model Struggles

Another strategy is to avoid predicting difficulty up front.

Let the cheap model try first, then escalate when evidence of sustained trouble appears. The flow becomes:

Request

|

v

Weak model

|

v

Judge result

/ \

okay struggling

| |

v v

stay strong model

Switchyard calls this escalation routing. You can create a configuration like this:

routes:

agent:

type: escalation_router

strong:

model: openai/gpt-4o

weak:

model: openai/gpt-4o-mini

judge:

model: openai/gpt-4o-mini

confirmations: 2

recent_turn_window: 28

window_message_chars: 500

fallback_target_on_evict: weak

This is conceptually different from up-front deterministic classification. Deterministic/capability routing asks:

How difficult does this request appear?

Escalation routing asks:

Is the weak model actually getting into trouble?

This makes escalation useful for long-running agent sessions where task difficulty can change over time.

Step 9: Measuring Whether Routing Is Actually Helping

A router is only useful if it improves the quality-cost trade-off. Switchyard exposes Prometheus metrics and statistics around requests, errors, latency, tokens, and routing behavior. The project also supports structured request telemetry and optional routing logs.

You can get server metrics with:

curl -s http://localhost:4000/metrics | head

and aggregate JSON statistics:

curl -s http://localhost:4000/v1/stats | python3 -m json.tool

For experiments, compare at least three runs:

| Configuration | Purpose |

|---|---|

| Always strong | Quality ceiling and cost baseline |

| Always weak | Cheap baseline |

| Switchyard router | Test whether routing captures most strong-model quality at lower cost |

The more useful question is not whether the router was 85% accurate, but how much of the strong model's quality did routing preserve, and how much cost and latency did it reduce? For example:

Strong-only:

$20

92% task success

Weak-only:

$5

71% task success

Router:

$9

89% task success

This tells you whether routing is economically useful.

Final Thoughts

As LLM systems become more agentic, the question is shifting from:

Which model should I use?

to:

Which model should I use for this request, at this point in the workflow, under this cost budget?

Switchyard is NVIDIA's attempt to turn that decision into reusable infrastructure.

For a first experiment, don't jump directly into stage routing or complex agent escalation.

Start with two models.

Measure them independently.

Use weighted random routing to verify your setup.

Then introduce capability-based routing and measure whether it preserves most of the strong model's quality while moving a meaningful percentage of requests to the cheaper tier.

This experiment gives you something far more useful than another LLM benchmark:

a quality-versus-cost curve for your actual workload.

And that is ultimately what intelligent model routing is trying to optimize.

Kanwal Mehreen is a machine learning engineer and a technical writer with a profound passion for data science and the intersection of AI with medicine. She co-authored the ebook "Maximizing Productivity with ChatGPT". As a Google Generation Scholar 2022 for APAC, she champions diversity and academic excellence. She's also recognized as a Teradata Diversity in Tech Scholar, Mitacs Globalink Research Scholar, and Harvard WeCode Scholar. Kanwal is an ardent advocate for change, having founded FEMCodes to empower women in STEM fields.