Blog
Choosing & integrating LLM APIs: a practical guide
Making your first LLM API call is easy: with most provider SDKs, it's about five lines of code. Keeping that call fast, affordable, and reliable once real users show up is where the actual engineering happens: costs can compound as conversations grow, rate limits arrive at the worst moment, and many stateless LLM endpoints forget conversation state after each request. This guide covers what LLM APIs provide and how they're priced, how to choose a provider and access model, and how to build the caching, retry, and memory layer your app needs around those calls.
What LLM APIs do & how they're priced
To understand the engineering around an LLM call, start with what the API provides and how the provider charges for it. LLM APIs are inference-as-a-service: you send a prompt over HTTP, the provider runs the model on their hardware, and you typically pay per token, although provisioned, tool, runtime, and self-hosted pricing also exist. Beyond the core chat/completions endpoint, most providers offer vector embeddings for similarity search, function calling for structured tool invocation, vision and multimodal input, streaming, and batch processing for async workloads.
Providers generally meter token usage and commonly quote rates per million input and output tokens, billed separately:
- OpenAI's gpt-5.6-sol runs $5.00 per million input tokens and $30.00 per million output, per its published rates.
- Anthropic's Claude Sonnet 5 runs $2 per million input tokens and $10 per million output, per its pricing docs.
- Batch processing cuts costs for workloads that can wait: Anthropic's Batch API charges 50% of standard rates.
Across these providers, output tokens cost more than input, and long conversations raise input costs as they grow; the session-state section covers trimming and summarization.
How to choose an LLM API provider
What really shapes user experience is speed: how fast the first token arrives, how smoothly tokens stream after that, and how the API holds up under load. A fast average response time isn't enough on its own, either. Tail latency (those occasional slow responses) can frustrate users even when your averages look fine, and it varies widely between providers.
Quality matters too, but it's harder to pin down than a single score. A model that tops a public leaderboard can still fall short on your specific task, so the reliable test is how it does on your own prompts and data, not how it ranks on a benchmark someone else designed. Weigh that against speed and cost, because the three pull against each other: pushing for the highest quality usually means a slower or pricier model, and optimizing hard for speed and cost tends to give up some quality. You can usually get two of the three, rarely all three at once.
Build fast, accurate AI apps that scale
Get started with Redis for real-time AI context and retrieval.Finally, think about data handling. Providers differ on how long they retain your inputs and outputs, where that data lives, and what compliance guarantees they offer. If you're working with regulated data, these details matter more than benchmark scores.
Comparing LLM API access models
Once you've shortlisted models, you still have to decide how to reach them. Providers often offer the same model through several doors, and four access models cover most setups. The trade-off is operational control versus provider convenience:
- Direct proprietary APIs (OpenAI, Anthropic): pay-as-you-go per token with minimal ops burden. You get frontier models, but the weights stay proprietary, so switching costs tend to be highest here.
- Cloud-mediated access (Bedrock, Azure OpenAI, Vertex AI): cloud platforms bill models through your existing cloud account and provide cloud identity and access management (IAM), residency controls, and provisioned throughput options.
- Open-model inference APIs: serverless per-token access to open-weight models, often with routing across multiple providers.
- Self-hosted open-weight models: full control over data and weights, in exchange for fixed GPU costs whether the hardware is busy or idle, plus real machine learning operations (MLOps) staffing.
A practical split is a frontier API for reasoning-heavy steps and cheaper open-model access for classification and retrieval. That split can reduce cost while preserving stronger models for the tasks that benefit from them.
Integrating an LLM API into your app
Whichever access model you pick, the integration mechanics look similar. Providers authenticate requests with a bearer token or API key, which you should store securely and separate by environment. Interactive apps typically stream tokens over server-sent events (SSE), so users see output as it's generated instead of waiting for a full response.
For structured outputs, lean on the provider SDKs. Most support schema-based parsing (like Pydantic in Python or Zod in JavaScript), which keeps your code types and JSON schemas in sync. Tool calling follows a simple loop: you define the tools, the model chooses one, your app runs it, and you send the result back. Treat any model-generated tool call like user input and validate it before execution.
Errors need a bit of classification before you retry. Rate limits (429) and server errors (5xx) are usually temporary, so retry them with exponential backoff and jitter. Bad requests (400) and auth failures (401) won't fix themselves, so retrying just wastes money and quota.
Cutting LLM API cost & latency with caching
Retries keep you running; caching can meaningfully reduce how much you spend on repeated inference. Two caching layers work especially well with LLM APIs, and they stack—provider-side prompt caching and semantic caching.
Prompt caching reuses the model's work on a repeated prompt prefix, so it skips reprocessing your static system prompt, tool definitions, and reference docs. Providers charge a small premium to write the cache, then a steep discount on every read of that prefix, so the savings add up fast on repeated context. The catch: any change to the cached prefix invalidates everything after it.
Semantic caching goes further. Instead of matching prompts exactly, it stores responses and returns them for queries that mean the same thing, even when worded differently. Your app converts each prompt into a vector embedding and checks it against cached entries; when the match is close enough, the cached response comes back with no LLM call at all. In production workloads, well-tuned semantic caches can serve a large share of requests directly from cache, cutting both cost and latency dramatically.
Make your AI apps faster and cheaper
Cut costs by up to 90% and lower latency with semantic caching powered by Redis.This is where Redis comes in. Redis keeps data in memory, so cache lookups happen in well under a millisecond, fast enough to sit directly in the request path. And because the same platform your app may already use for regular caching and sessions also supports vector search and semantic caching, you don't need a separate system to add an AI cache layer. Redis packages this as a managed context engine for AI apps called Redis Iris: its semantic caching service, Redis LangCache, matches queries by meaning and serves the stored response, so you get the cache layer without running the embedding and vector search yourself. LangCache is currently in public preview.
Managing rate limits, retries & session state
Caching reduces how often you call the API; rate limits govern what happens when you still call it a lot. Providers usually cap you on more than one dimension at once, like requests per minute and tokens per minute, and whichever ceiling you hit first is the one that stops you. Failed requests often still count against your quota too, so blindly retrying a 429 only digs the hole deeper.
Session state is the other thing you own. Most LLM APIs are stateless, meaning you resend the full conversation history on every request. That has two consequences: token costs pile up as conversations grow, and quality can degrade once the context gets long. Some providers now offer server-managed conversation state, but many teams still keep their own store so they can trim, summarize, and share context across services.
That store needs to be fast, since your app usually reads it on every request. Redis fits naturally here—it's in-memory and supports configurable expiration on individual fields. Iris builds on that with Redis Agent Memory, a managed two-tier service: short-term interaction history plus long-term memory for preferences and prior sessions, so your agents keep context across turns and sessions without you wiring up a memory store. In-process session storage, by contrast, disappears on restart and can't be shared across load-balanced or multi-agent deployments.
Building the layer around your LLM API calls
Caching, retries, routing, and rate limiting each solve one problem. At some scale, it makes sense to pull them into a shared layer, often called a GenAI gateway, that sits between your apps and your LLM providers. Guardrails, model routing, identity, and audit can then evolve independently of the services behind them. Uber, for example, built its own GenAI Gateway to serve 60 LLM use cases across the company.
Routing is usually the biggest cost lever inside that layer. A well-tuned router can send only the hardest requests to your most expensive model and hand off the rest to something cheaper, often without a noticeable drop in quality. Observability matters too: tracking token usage per request lets you tie real costs back to specific features, users, or models. Some gateways lean on Redis as their data layer for exactly these reasons, using it for semantic caching and shared state.
That said, a gateway isn't mandatory. If only a few teams or models are involved, building these capabilities directly into your app can be simpler and lower-latency, as long as you keep the pieces loosely coupled.
Now see how this runs in Redis
Power AI apps with real-time context, vector search, and caching.Your LLM API is only half the architecture
Choosing an LLM API means weighing latency, task-specific quality, data handling, and access model, not just the per-token sticker price. Once you've chosen, the real work shifts to the layer you build around the calls: caching to cut cost, backoff and rate-limit handling to stay reliable, and an external session store to give a stateless API something like memory.
Redis Iris covers that layer as managed services on top of fast, in-memory infrastructure: semantic caching, agent memory, and session storage with field-level expiration, with vector search underneath, so you're not stitching together a separate system for each job. If your app team already runs Redis for caching, the AI context layer is an extension of what you have, not a new vendor. Try Redis Iris to see how it fits your workload, or talk to our team about the layer around your LLM API calls.
Get started with Redis today
Speak to a Redis expert and learn more about enterprise-grade Redis today.