Swap two cups and transformers lose the ball

Researchers at Ai2 have released StateBench, a benchmark for long-context state tracking: how well a language model keeps track of things in a long document as they change. The everyday version is a magician's cup game, where a ball sits under one of five cups that are swapped again and again. In code it is variables being assigned and reassigned. The question matters because hybrid architectures that carry a running memory, such as Gated DeltaNet, are spreading, and researchers need to know which kinds of state tracking each design can handle.

The benchmark has four levels, framed as kinds of variable assignment. In the easiest, a value never changes; then values are reassigned and only the latest matters; then each update builds on the current value, like a running score; and finally one variable can take another's value, as in a swap. The levels come from Krohn-Rhodes theory in algebra, and each is built from the same underlying sequence so only the type of update differs.

Tested on open pretrained models with documented training data, including the 7-billion-parameter Olmo 3 transformer and the Olmo Hybrid model with Gated DeltaNet layers, almost all did well on the easiest level, lost accuracy on reassignment as inputs grew, and stayed at chance on the hardest. Format mattered too: small hybrid models near the ceiling on one framing of the easiest problems were near chance on the same problems framed as spreadsheet edits.

The headline finding is about architecture. When trained directly on these tasks, recurrent and hybrid models generalised much farther than transformers, consistent with theory that predicts transformers fail on the hardest class. Frontier models including Gemini 3.8 Flash, Claude Opus 5.5 and GPT-6 Sol and Luna still broke down on harder forms of the problem.