Subquadratic Models: What SubQ's 12M-Context Architecture Actually Means

Every long-context announcement of the last two years has been an announcement about attention doing more of the same thing at greater expense. Bigger windows, more tokens, higher prices, the same quadratic architecture pushed a little further each time. SubQ's 12-million-token context is different in kind, not just in size. It doesn't push attention further. It replaces it.

That distinction is the whole story, and it's the part the "12M tokens!" headline erases. A 12-million-token window built on a subquadratic architecture is a genuine breakthrough in the cost of long context. It is not a breakthrough in the model's memory of that context. Those two things get conflated constantly, and conflating them leads to the same disappointment that long-context transformers already taught us — just arriving from a different direction.

This post is about what "subquadratic" actually means, why it unlocks context lengths transformers can't reach, and what it quietly trades away to get there.

The Quadratic Wall

Standard transformer attention has a property that is both its superpower and its ceiling: every token attends to every other token. For a sequence of length n, that's on the order of n² comparisons. The model computes, for each token, how relevant every other token is, and it does this at every layer.

This pairwise comparison is why transformers are so good at recall. If the answer to your question is a specific number sitting 40,000 tokens back, attention can, in principle, form a direct connection between your question and that number. Every token is one hop away from every other token. Nothing is structurally out of reach.

The cost of that superpower is quadratic scaling. Double the sequence length and you roughly quadruple the compute and the memory. Go from a 2,000-token prompt to a 12-million-token one and the pairwise work grows by a factor of roughly 36 million squared over two thousand squared — tens of trillions of times more. No amount of flash-attention kernel optimization or hardware makes that constant-factor problem disappear, because it isn't a constant-factor problem. It's the shape of the curve. This is the wall every pure-transformer long-context model runs into, and it's why the 2-million-token windows we already have are expensive and why nobody was shipping 12 million on the same architecture. (It's also a big part of why a 2M context window is not what you think even when you can afford it.)

To get to 12 million tokens, you don't optimize attention. You change the scaling law. That's what subquadratic means.

What "Subquadratic" Actually Means

A subquadratic model is one whose compute and memory grow slower than n² as the sequence gets longer — linearly, in the best cases. The family includes state-space models (SSMs) like Mamba, linear-attention variants, and RNN-descended architectures like RWKV. They differ in the details, but they share one core move.

Instead of comparing every token to every other token, a subquadratic model summarizes the past into a fixed-size running state and updates that state as it reads each new token. Think of the difference between keeping every email you've ever received and being able to search all of them (the transformer), versus keeping a running set of notes that you update as new emails arrive (the state-space model). The notes are cheap to carry and cheap to update no matter how many emails come in. Searching the full archive is powerful but gets more expensive with every message.

Because the state is a fixed size, the cost of processing each new token is constant — it doesn't depend on how many tokens came before. Process a million tokens and the millionth token costs the same as the thousandth. That's linear total cost, and linear is what makes 12 million tokens tractable. The architecture doesn't hit the quadratic wall because it never builds the pairwise comparison in the first place.

This is not a 2026 invention. State-space models and linear attention have been a serious research thread for years; Mamba made SSMs competitive with transformers on language in 2023-2024. What's new is that the recipes matured enough to build a frontier-scale, production model — SubQ — around a subquadratic backbone and ship a context length that quadratic attention simply can't reach at any reasonable price.

The Trade-off: Compression Versus Recall

Here's the part the announcement doesn't lead with. A fixed-size state is a compression of the past, and compression is lossy.

The transformer's every-token-to-every-token structure means any past token is directly reachable. The subquadratic model's running state means the past is present only as a summary, and everything that happened has to share that bounded summary. The further back a specific detail is, and the more that has happened since, the more that detail has been overwritten, blended, and compressed. The model carries the gist of 12 million tokens beautifully and cheaply. Asked to reproduce an exact token — a specific API key, a particular number, one clause in one contract from eight million tokens ago — it is weaker than a transformer would be, because that exact token was never stored; only its contribution to a running summary was.

Researchers call this the associative-recall gap, and it's the defining weakness of pure subquadratic models. They're excellent at continuity, tracking, summarization, and "keep the whole thing loosely in mind." They're worse at precise, content-addressed retrieval from deep in the context — exactly the thing quadratic attention is best at.

So the 12-million-token window is real, and the tokens really do all go in, and the model really does condition on them. But "conditioned on a compressed summary of 12M tokens" is a different guarantee than "can retrieve any of those 12M tokens on demand." Same lesson as long-context transformers — fits in the window is not the same as attended to equally — arriving here through compression instead of through attention degradation.

Why 12M Tokens Isn't 12M Tokens of Recall

Put the two facts together and you get the honest version of the headline.

A subquadratic 12M-context model gives you enormous, cheap breadth. You can keep an entire large codebase, a book-length document set, or a very long running conversation continuously in context without the cost exploding. For workloads that are about breadth and continuity — "stay aware of this whole system," "maintain this conversation for days," "watch this stream and react" — it's transformative, and the constant per-token cost makes always-on and streaming use cases economical for the first time.

What it does not give you is a perfect 12-million-token lookup table. If your task depends on pulling one exact needle out of a multi-million-token haystack, a pure subquadratic model is the wrong tool for that specific step. The needle has been compressed along with everything else. This is why the naive "just put everything in the giant window and ask" pattern underdelivers on precision-critical tasks, exactly as it does with long-context transformers — for a related but distinct architectural reason.

The number that matters is never the nominal window. It's the effective, task-specific one: how much of that context the model can actually use for the thing you need. For breadth, 12M is close to real. For precise recall, the effective figure is smaller and depends on the architecture's memory capacity, not its window length.

Hybrid Architectures: The Practical Answer

The frontier didn't resolve this as a winner-take-all fight, and that's the most important practical point. It resolved it as a blend.

Most serious 2026 systems that use subquadratic backbones are hybrids: many cheap subquadratic layers for scaling, interleaved with a small number of full-attention layers for precise recall. The subquadratic layers carry the long-range context at linear cost; the occasional attention layer restores the direct-lookup capability where it's needed. You get most of the cost profile of a subquadratic model and most of the recall of a transformer. Jamba-style designs pioneered this mix; SubQ productionizes it at a context length the mix makes affordable.

Layer this on top of the other big architectural shift and the picture completes. Mixture-of-Experts broke the link between parameter count and inference cost; subquadratic attention breaks the link between context length and inference cost. Combine them — MoE feed-forward layers, hybrid subquadratic-plus-attention sequence mixing — and you get a model that's efficient on both of the axes that used to blow up your bill. That combination, not any single trick, is what makes a cheap 12M-token model buildable.

What This Means for What You Build

For teams integrating models through APIs, a few implications are worth internalizing.

Long context and retrieval are complementary, not competitive. The bigger the affordable window gets, the more tempting "just stuff everything in" becomes — and the more you'll be burned on precision-critical tasks. Use the large context for breadth: keep the whole codebase, the full conversation, the entire document set in the room cheaply. Use retrieval when the answer must be grounded in a specific passage, because retrieval puts that passage at high salience instead of trusting a compressed state to surface it. The 12M window widens what you can do; it does not retire RAG.

Match the architecture to the task shape. Streaming, monitoring, long-running agents, and "maintain awareness over time" workloads are where subquadratic models shine, because constant per-token cost is exactly what those need. Needle-in-a-haystack lookup and exact citation are where you either lean on the hybrid's attention layers or add retrieval on top.

Don't benchmark on window size — benchmark on your task at depth. A vendor's context-length number tells you what fits, not what works. Test recall and reasoning at the depths and positions your real workload uses. Effective context is an empirical property of the model on your task, not a spec-sheet figure.

Expect the cost curve for breadth to keep falling. Subquadratic and hybrid architectures make long context structurally cheaper, and that pricing pressure will continue. Workloads you dismissed as too expensive to keep fully in context may be worth revisiting.

The Takeaway

SubQ's 12-million-token context is a real advance, and it comes from a real architectural change: replacing quadratic attention with a subquadratic backbone whose cost grows linearly with length instead of quadratically. That's what makes a context length transformers can't reach economically possible.

But subquadratic scaling buys breadth by compressing the past into a fixed-size state, and compression is lossy. The model carries the gist of 12 million tokens cheaply; it does not store every one of them for exact recall. The honest reading of the headline is "12 million tokens of cheap awareness," not "12 million tokens of perfect memory." Hybrids close most of the recall gap, and retrieval closes the rest where precision matters.

Understand the trade and you'll use these models for what they're genuinely great at — enormous, continuous, affordable context — without expecting them to be a flawless lookup table they were never built to be. The architecture changed. What "long context" means changed with it. Build for the version that's real.

Frequently Asked Questions

What is a subquadratic model?

A sequence model whose compute and memory grow slower than the square of the input length. Standard transformer attention is quadratic: doubling the sequence length roughly quadruples the work, because every token attends to every other token. Subquadratic architectures — state-space models like Mamba, linear-attention variants, and RNN-style designs like RWKV — grow linearly or near-linearly instead, by summarizing past tokens into a fixed-size running state rather than comparing every pair. That is what makes million-token contexts economically possible.

Why can't transformers just scale to millions of tokens?

Because attention is O(n²) in the sequence length. At 12 million tokens, the pairwise attention computation is roughly 36 trillion times the work of a 2,000-token prompt, and the memory to hold the attention state explodes the same way. You can engineer around it partially (flash attention, sparse attention, chunking), but the fundamental scaling wall is why no pure-transformer model offers a 12M window at usable cost. Subquadratic architectures remove the wall by changing the scaling, not by optimizing the constant.

Does a 12-million-token context window mean the model remembers everything in it?

No. This is the key misunderstanding. Subquadratic models compress the past into a fixed-size state, so information competes for a bounded amount of 'memory' no matter how long the input is. The model can carry the gist of 12M tokens cheaply, but precise recall of a specific fact buried deep in the context is weaker than a transformer's — the compression is lossy by design. 'Fits in the window' is not the same as 'perfectly recalled,' the same lesson as long-context transformers, for a different underlying reason.

Are subquadratic models better than transformers?

Not strictly better — different. They win decisively on long-sequence cost and on streaming or always-on workloads, because their per-token cost is constant regardless of how much came before. Transformers still win on exact associative recall — pulling a specific token out of context by its content — which is why most frontier systems in 2026 are hybrids that interleave a few attention layers with many subquadratic layers to get both cheap long context and precise recall.

Should I use a long-context model or retrieval (RAG)?

Usually both. A large subquadratic context is excellent for keeping a whole codebase, a long conversation, or a big document set 'in the room' cheaply. But retrieval still wins when you need the model to ground its answer in a specific passage, because retrieval puts that passage at high salience instead of hoping the model recalls it from a compressed state. Use long context for breadth and continuity; use RAG for precision and citations. The 12M window widens what you can do — it does not retire retrieval.