From VLM/VLA's to Embodied Agents — Armen Aghajanyan, Perceptron AI

Feed a model one hour of video and roughly a million visual tokens go in, yet the loss lands on about two percent of them, since the only ground truth is a transcript or a few labeled frames.

Armen Aghajanyan calls that a humongous waste, and predicting every pixel treats a background pixel with the same weight as a gripper tip or a contact point. Perceptron's answer is a perceptive objective that learns which percepts will matter, rather than hardcoding the gripper. The second problem is context bloat from cameras that never switch off. Patch averaging buys ten times compression, but his fix is data sparse mixture of experts, a router that decides per layer which tokens to read and which to skip. Left alone, the model zooms into the graph in a figure and spends more tokens on fruit when asked to segment fruit. Put together, that produced the model his team released a few weeks earlier, trained on a petabyte spanning text, images, video, and trajectories from desktop use to video games, which he says beats a frontier lab's embodied reasoning model at a fraction of the cost.

Mer att läsa

Building advertising for the way people use AI

OpenAI · för 3 tim sedan

Server-Side Code Execution Tools for AI Agents, Compared

OpenRouter · för 13 tim sedan

v0.40.0

Ollama · för 13 tim sedan

Google froze its open source bug bounty program due to a ‘significant rise’ in AI submissions

TechCrunch AI · för 16 tim sedan