The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral
Benchmarks won't tell you how a model behaves in your app. Swapping it in and testing will.
NEXT UP: AI ENGINEER CODE SUMMIT Join us in SF for the second annual AIE CODE Summit, Nov 10–12, our last flagship summit of 2026. Applications live now: https://aie.codes Pamela Fox, Principal Cloud Advocate at Microsoft, runs the same scenarios against GPT-5.5, Kimi K2.6, DeepSeek V4 Flash and Mistral Large 3 on Microsoft Foundry so you can see where models differ and what you can do about it. She works up from a letter-counting prompt to RAG groundedness, image input, multilingual pitfalls, tool calling, tool calling in a loop and a LangChain v1 travel agent, then moves to evaluation: string-check evals, LLM-as-a-judge, Microsoft's new ASSERT tool for generating test sets from a spec, and DSPy prompt optimization that finds a prompt the cheapest model can satisfy.
The text is the source's own description of its publication. The content belongs to AI Engineer.
More from AI Engineer
Inside Arca's Long-Horizon Agents That Run the Business - Aman Singh, Arca
AI Engineer 23 Oct
Real AI Agent Attacks and What Would Have Stopped Them — Andrey Rybka, Bloomberg
AI Engineer 23 Oct
Outside of tech there has been no financial payoff to US companies for using AI, now viewed as a margin-hurting cost center, as VC-backed firms burn more tokens
Techmeme 4 h ago
Source: Replit recently moved most AI spend from Anthropic to OpenAI, which gave 30%+ off in a deal worth $100M+/year, after Anthropic discontinued its discount
Techmeme 6 h ago