The Attention Problem Nobody Wants to Talk About

The numbers they report are solid: 2.0x to 2.2x inference speedup on various benchmarks, with success rates actually going up, not down. On real-world memory-dependent tasks, they saw a 23.3% absolute improvement in success rate. That's not nothing.

What strikes me about both papers is that they're not proposing entirely new architectures. They're finding ways to make the existing architecture less wasteful. It's optimization, not revolution. And honestly? That's probably what we need right now.

So what

Here's where I'll probably get some angry emails (my address is on the about page if you want to argue). I think these papers are symptoms of a larger problem that the robotics and AI community hasn't fully grappled with.

We've been scaling these models up, adding more parameters, more data, more compute, because that's what worked for language models. But visual transformers for robotics have different constraints. You can't wait three seconds for your robot to decide whether to pick up a cup. You can't ship a warehouse robot that needs a data center to process what it sees. The quadratic scaling of attention isn't just an academic concern, it's a deployment blocker.

The young founders I talk to (and yes, I'm old enough now that most founders seem young to me) often wave this away. "Hardware will catch up," they say. "Inference costs are dropping." And sure, that's true to a point. But it's also what people said about self-driving cars in 2015. We're still waiting.

What these papers suggest is that there's a lot of low-hanging fruit in efficiency that hasn't been picked yet. The DySta team found that VLAs have "a limited context window for input frames and inefficient inference," and their solution improved performance while cutting compute. The Good Token Hunting team found that you can throw away most of your tokens and do better, not worse.

This tells me the current generation of models is, to put it bluntly, doing a lot of dumb work. They're processing tokens that don't matter, recomputing things that haven't changed, attending to everything when they only need to attend to some things. It's wasteful, and the field is only now starting to take that seriously.

What I don't know yet

I should be honest about the limitations here. Both papers are recent, the benchmarks are their own, and real-world deployment is a different beast than research results. The Good Token Hunting paper tested on scenes with 500 images, which is a lot, but I don't know how this translates to continuous video streams in actual robots. The DySta paper did test on real-world tasks, which is encouraging, but the specific tasks and environments matter a lot.

It's also unclear whether these approaches compose well with other efficiency techniques, or whether they're competing for the same gains. If you're already using some form of sparse attention, does token selection help? I couldn't find good answers to that in either paper.

And there's always the question of generalization. These methods work on the benchmarks they were tested on. Whether they work on your specific robot doing your specific task in your specific environment remains to be seen. The field has a habit of overfitting to benchmarks, and I've seen too many promising techniques fail to transfer.

The bottom line

Look, I'm not saying these papers are going to change everything. They're incremental improvements on a known problem. But I am saying that the problem they're addressing, the quadratic attention bottleneck, is real and underappreciated. And the fact that two separate teams, working on different applications, both found significant gains by just being smarter about token selection? That's a signal.

The robotics industry is going to have to get serious about efficiency. Not because it's academically interesting, but because actual robots need to run on actual hardware in actual time. These papers are a step in that direction, even if they're not the final answer.

But what do I know. I've only been covering this stuff since before most of these researchers were born.

Robert "Bob" Macintosh · 2 hours ago · 3 min