Productionizing LLM Gateways: Architecture, Tradeoffs and Hard Lessons — Kanish Manuja, Twilio

Kanish Manuja from Twilio explains how to build LLM gateways — middleware that routes requests between different AI models — and why it's harder than it seems.

A fundamental challenge is that you cannot maximize everything simultaneously: availability, speed, security, and cost often conflict. Once an AI model begins responding and streaming tokens, you cannot switch to another provider if something fails, so you must plan for backups per request instead of global retries. Different models are surprisingly different — a reasoning model taking 60 seconds is completely normal, whereas a chat model would be entirely down — so you must measure thresholds separately for each model and path. Many teams believe they need a central gateway, but often they only need centralized control over which models are used, which can be solved without centralizing all traffic.

More from AI Engineer

Inside Arca's Long-Horizon Agents That Run the Business - Aman Singh, Arca

AI Engineer · 23 Oct

Real AI Agent Attacks and What Would Have Stopped Them — Andrey Rybka, Bloomberg

AI Engineer · 23 Oct

Outside of tech there has been no financial payoff to US companies for using AI, now viewed as a margin-hurting cost center, as VC-backed firms burn more tokens

Techmeme · 4 h ago

Source: Replit recently moved most AI spend from Anthropic to OpenAI, which gave 30%+ off in a deal worth $100M+/year, after Anthropic discontinued its discount

Techmeme · 6 h ago