The Base Model Is Dead — Varun Singh, Arcee AI

Varun Singh from Arcee AI argues that the traditional view of base models—as mirrors of the internet upon which other techniques build—no longer holds true.

Modern models integrate instruction data and synthetic training data much earlier in the process, and a new intermediate stage has emerged. Raw webtext takes a backseat while reinforcement learning (a method to train AI to reason better) has become so important that it changes what the base model needs to do from the start. Singh discusses practical challenges his team encountered when training the Trinity series, such as achieving the right balance across different data sources and achieving model stability early so it is prepared for what comes later. The point is that the base model's role is constantly being redefined as capacity increases.

More from AI Engineer

Inside Arca's Long-Horizon Agents That Run the Business - Aman Singh, Arca

AI Engineer · 23 Oct

Real AI Agent Attacks and What Would Have Stopped Them — Andrey Rybka, Bloomberg

AI Engineer · 23 Oct

Outside of tech there has been no financial payoff to US companies for using AI, now viewed as a margin-hurting cost center, as VC-backed firms burn more tokens

Techmeme · 4 h ago

Source: Replit recently moved most AI spend from Anthropic to OpenAI, which gave 30%+ off in a deal worth $100M+/year, after Anthropic discontinued its discount

Techmeme · 6 h ago