86% of What Coding Agents Do Is Just Reading — Not Solving | Alexander Whedon of Subquadratic

Transformer attention bottlenecks

0:00In Transformer models, the cost of attention grows with the square of the context length. So doubling context makes the compute roughly four times more expensive. And Alex and his team have solved this issue. When we've done benchmarking the frontier models with Superbench Pro, we found that 86% of the steps were actually read steps, just trying to do that context engineering before execution review, which was only the last 14%. Is this the end of RAG? If you could put everything that's in your RAG knowledge base into the context window, you wouldn't need that more complicated architecture. Is that right?

0:41Yeah, it is. Although I wouldn't say it's the end of RAG. Just because it's too extreme. All of these enterprises that are sitting on massive amounts of data, and a lot of that data has yet to be put to use in an AI product, they're getting pushed to do like a $10 million data transformation project before they can even start to build a product on top. And value prop that we bring to the table is that is no longer necessary.

Introducing SubQuadratic

1:05Hi, this is Eye on AI. And today we're talking to Alex Whedon about his company SubQ, which solves a problem that plagues transformer models. In transformer models, the cost of attention grows with the square of a context length. So doubling context makes the compute roughly four times more expensive. That's what we mean by quadratic compute complexity.

1:43And Alex and Alex and his team have solved this issue. So Alex, can you start by introducing yourself to listeners and we'll we'll go from there. Alex Whedon- Sure. I'm Alex co-founder and CTO here at SubQuadratic. I've been in the space for about 10 years. I started working with language models since before transformers existed when LSTMs were still the hot thing. And our mission here is to always be the first to make the most computationally memory and sample efficient algorithms and build foundation models on top of those.

Explaining sparse attention

2:25Alex Whedon- Yeah. And can you explain, you've tackled this quadratic complexity with sparse attention, am I right? And can you explain what sparse attention is in simple terms and why it matters compared to standard transformer attention? Alex Whedon- Sure. So the quadratic bottleneck of attention is introduced because attention compares two token relationships. So you have 1000 tokens and there are 1000 squared two token relationships or a million squared two token relationships for a million tokens. You have this quadratically growing set of math that you're going to perform with an attention.

3:13Alex Whedon- Sparse attention says we don't actually need all of those token relationships. And so sparse attention attempts, attempts to find which token relationships are important before running attention on only a subset of the token relationships. And one of the main challenges with sparse attention is that there's always been a quality trade off, as you can imagine, like, in theory, if you're not looking at every token relationship, then you can potentially miss some important ones. Alex Whedon- However, you're also looking at a lot that don't actually matter. And so the challenge of sparse attention has been figuring out which ones actually matter. Most sparse attention mechanisms have used fixed patterns. So based off of the distance between the target token and all the other potential tokens, there's some fixed mathematical formula that determines which tokens should be passed through attention.

4:09Alex Whedon- Deep seek sparse attention. Deep seek sparse attention up into this by saying we're actually going to use a separate prediction model to determine which tokens are important. Alex Whedon- And that worked. And they showed that if you dynamically select the token relationships before running attention over a subset, then you can do so without a quality trade off. The only problem is that prediction model that selects the token relationships uses full attention. And so you still have the quadratic problem. In fact, that's the problem. Alex Whedon- That separate selection model uses more compute than the sparse attention layers in the full model at just 52,000 tokens and then continuously grows to an even larger differential.

4:53Alex Whedon- And so what we've done that's unique is we have a sparse attention mechanism that like Deep seek sparse attention dynamically selects the token relationships, but does so without the massive cost of full attention and does so without a separate selection mechanism. So the single model does both the selection generation at the same time.

Enterprise use cases and costs

5:15Alex Whedon- And can you give us a practical use case where this quadratic compute complexity makes it too expensive to run the model on something? I mean, code bases seems like an obvious, but are there other situations where you want a very large context window? Alex Whedon- And that it would get unreasonably expensive to do that with regular attention? I mean, regular transformer algorithms?

5:54Alex Whedon- Yeah, absolutely. I mean, I think a very significant portion of enterprise workloads fall into this category. And by the way, I will just clarify now that we're not just after the absolute context window size, we're after higher intelligence within that context window, lower cost of context, lower latency for context. And so these are actually four different axes of context. We care about size, intelligence, cost and latency. Alex Whedon- And enterprise workloads are dramatically impacted by these. Everybody's making trade-offs. Everybody's trying to reduce the number of tokens that they're using as much as possible. Everybody is doing context engineering that requires a lot of effort and also narrows the scope of what that system can actually handle.

6:42Alex Whedon- But some of the top use cases that we see are, yes, code is obviously one. Alex Whedon- Thinking about code review, planning, and long horizon memory, those are all long context problems. Alex Whedon- Thinking about the fact that when we've done benchmarking of Frontier Models with SweetBench Pro, we found that 86% of the steps were actually read steps, just trying to do that context engineering before the execution review, which was only the last 14%.

7:12Alex Whedon- So, and then financial document analysis is another really big one. Financial documents can be a couple hundred pages. And if you really want to analyze trends, then it's helpful to have many of those documents over a period of time or several different companies. Alex Whedon- In enterprise, you have hundreds of billions of tokens of textual data. And anytime you want to tackle a problem, it's helpful to have a broad range of documents.

7:46Alex Whedon- The key point that we're conveying and working on with enterprises is how do we do coarser grain retrieval, more retrieval, Alex Whedon- steps at once. How do we avoid chucking documents as much as possible or make larger, more comprehensive chunks, etc., to offset the quality loss that we often see with a lot of the context engineering that's done today. Alex Whedon- Yeah. And that quality offset, or you said earlier a moment ago that it's not only the cost, but it's the intelligence within the context.

8:28Alex Whedon- Is that how you phrased it? Because large context windows, many LLMs don't operate optimally within a very large context window. They don't see things or they get lost in the context. Alex Whedon- Can you talk about that? How do you solve that? I mean, even though you can now do very large context efficiently?

Context intelligence and alignment

9:04Alex Whedon- It's a great question. So, yeah, the intelligence within the context window is a very important topic. If we just take a step back and think about what frontier context has looked like. Alex Whedon- Historically, we didn't get frontier models with more than 256,000 token context window until March when the Opus 4.6 general access, 1 million token general access came available. And so until then, most enterprises were building with a less than 256k usable context window. You see that...

9:46Alex Whedon- So a lot of people that we talk to, they're using 100,000 tokens or less, and they haven't yet adapted their workflows or RAG setups to use more than that. Alex Whedon- Sometimes they feel a quality tradeoff above that. With particularly hard problems, actually 100,000 is a fairly common cutoff, surprisingly. Alex Whedon- In the coding space, a lot of people set the compaction limit to 400,000 tokens because that's where they feel there is a qualitative tradeoff within a 1 million token context window.

10:16Alex Whedon- Because of this, in addition to intelligence, I would say there's also an alignment problem within the context window that user preferences with how a model should handle, should leverage 1 million tokens are not super robustly understood today. Alex Whedon- There are just not enough enterprise applications built to actually leverage a full 1 million token context window to really understand what that should look like. Alex Whedon- And so as we push into the multi-millions of tokens, this is very much white space, not even just from a technical perspective, but from our product perspective.

10:53Alex Whedon- What do users want? Alex Whedon- There are all sorts of benchmarks that kind of measure human preferences at short context, but a long context doesn't exist. Alex Whedon- And then finally, I will just add that one of the challenges with large context intelligence and user preference matching is the amount of training that you can do at large context. Alex Whedon- And so that's one of the main reasons that we set out to create SSA in the first place was that it is cost prohibitive to do large scale training on large context.

11:26Alex Whedon- As far as we know, we are the only people to do robust multi-million token pre-training, which we think differentiates our models. Alex Whedon- So like the number of times that a model can see how to handle a certain situation at a reason over some context at large scale. Alex Whedon- As that number goes up, obviously the model gets a lot better and that's really hard to do with quadratically scaling compute costs.

Multi million token pre-training

11:51Alex Whedon- So go a little deeper on that. Alex Whedon- I mean, you've trained and evaluated up to 12 million token context windows. Alex Whedon- So explain how you avoid the problems that large context windows create. Alex Whedon- I mean, you go a little deeper on how the model approaches that.

12:21Alex Whedon- Yeah, so I mean, I feel like I kind of buried the lead on that with one of my earlier answers. Alex Whedon- But I mean, the problem with sparse attention historically has been that they're fixed pattern mechanisms, like sliding window attention. Alex Whedon- Look at the last 1000 tokens that are closest. Alex Whedon- And before and after. Alex Whedon- But these fixed patterns don't robustly model language.

12:54Alex Whedon- So like there's no guarantee that the first and seventh token are always going to be relevant to each other. Alex Whedon- It's highly context or content dependent. Alex Whedon- And so you need a mechanism that accurately models context, accurately determines which relationships are going to be important no matter what the input looks like. Alex Whedon- And so while DeepSeek sparse attention uses a separate model to do that, reintroducing the quadratic problem, we do not.

13:29Alex Whedon- We have a mechanism that is much more lightweight and cheaper and leverages the existing model that does the generation to do the selection as well. Alex Whedon- So it's super cheap, super fast, but also dynamic so every input gets a different attention map. Alex Whedon- And then we train. Alex Whedon- We leverage that the fact that we can do this dynamic selection to achieve high quality results, while also having efficiency differentials.

14:03Alex Whedon- We are around 40 times faster than flash attention before on B200s and B300s. Alex Whedon- And then we're using 64 times less compute. Alex Whedon- Both of these figures at a million tokens. Alex Whedon- So this means that we can do a lot more 1 million token training with the same compute or 2 million token training or whatever. Alex Whedon- And so this enables us to introduce a lot of problems that models have yet to be trained on robustly.

14:40Alex Whedon- The reality is we're training on a lot of data that hasn't been included in training to date and introducing capabilities that have yet to be introduced into the models to date. Alex Whedon- Because we have this mathematical efficiency differential that just makes the math of doing this training so much different. Alex Whedon- Just because my audience isn't all ML engineers. Alex Whedon- So the model, you use sparse attention and you, how does the model decide what tokens to pay attention to?

15:21Alex Whedon- I mean, you were saying that DeepSeek has a separate model that does that prediction, but can you go into how your models with these large context windows then can decide which tokens to focus on? Alex Whedon- Yeah. Alex Whedon- I mean, that's the highly proprietary part that we haven't done properly. Alex Whedon- Yeah. Alex Whedon- But I would say, I guess I could say we creatively use some early information on the inputs in a way that doesn't create all to all comparisons.

16:02Alex Whedon- Which is very high level, but I feel that going deeper than that leaves a lot of breadcrumbs.

Transforming RAG architectures

16:09Alex Whedon- Well, one thing I wanted to ask you, and we've spoken a couple of times, but does this, is this the end of RAG then of having a vector database that the model searches? Alex Whedon- I mean, could you, if you could put everything that's in your RAG knowledge base into the context window, you wouldn't need that more complicated architecture.

16:42Is that right? Alex Whedon- Yeah, it is, although I wouldn't say it's the end of RAG, just because it's too extreme. Alex Whedon- So the way it changes RAG. Alex Whedon- What we're looking to do is simplify the way that people build products with RAG or, you know, agendic workflows. Alex Whedon- Examples are today, a lot of documents are chunks that don't necessarily need to be chunked. Alex Whedon- The chunk sizes are very small.

17:15Alex Whedon- Every 400 tokens is like super common. Alex Whedon- And they're like, what if you could have a multi-page, multi-section chunk instead? Alex Whedon- There would be a lot more contiguous context that would prevent misunderstandings. Alex Whedon- Top K of 10 for the top 10 search results is very common. Alex Whedon- Well, what if you could pull the top 50? Alex Whedon- What if instead of doing one type of search you actually did like 20 parallel searches and dumped all of those search results into the context window?

17:49Alex Whedon- These are ways in which I think we make RAG a lot more, we can enable people to make RAG a lot more robust. Alex Whedon- If you have 400 million tokens like there still needs to be some way for the model to process that data. Alex Whedon- But we're saying like don't compress everything into 100,000 tokens or less. Alex Whedon- You can now afford to because of cost and latency. Alex Whedon- And then also the intelligence and absolute budget of context. Alex Whedon- You can now pull a lot more context and do a lot more work with every single step in your agent workflow.

18:24Alex Whedon- Like I mentioned before, if we're talking about coding problems, Alex Whedon- Opus 4.6 required an average of 66 steps to tackle C-Bunch Pro problems. Alex Whedon- Yeah. Alex Whedon- And if we assume that we only needed like the last 10%, Alex Whedon- We could have gotten away with like six or seven steps. Alex Whedon- If we had been able to do, if we'd been able to just put all the code in the context window, Alex Whedon- Or even if we took like three or four steps to figure out which parts of the code base are more important,

18:54Alex Whedon- Being more aggressive with the context in each step. Alex Whedon- I mean, that is a much faster, much more efficient process that actually uses more token, Alex Whedon- Sorry, fewer tokens in aggregate. Alex Whedon- So it's more token efficient. Alex Whedon- And then it's not going to miss the details. Alex Whedon- We do hear complaints about coding agents missing details. Alex Whedon- And definitely on the longer horizon. Alex Whedon- So not the end of RAG, but the transformation of RAG as we know it.

19:28Alex Whedon- And you talked about doing multi-million token pre-training rather than just post-training. Alex Whedon- Can you walk us through how that works and how that changes the model's behavior and practice? Alex Whedon- Yeah. Alex Whedon- So pre-training is the stage of training where you give the model its core knowledge. Alex Whedon- And to some extent, some core capabilities.

19:59Alex Whedon- But it's very knowledge focused. Alex Whedon- So pre-training simply means you predict the next token over and over again. Alex Whedon- And then the model gets penalized based off of how often that next token matches the actual next token in a corpus. Alex Whedon- So it's crude and rudimentary, but it's also highly scalable. Alex Whedon- And it's largely seen as a requirement to extract additional capabilities in the post-training phase.

20:31Alex Whedon- And so research has largely suggested that your ability to create value in the post-training phase is limited by the pre-training that you've done. Alex Whedon- So if you haven't, for example, done a lot of pre-training on code, then you're going to struggle to post-train a model to be really good at code. Alex Whedon- And so that's why it's important that we've done this multi-million token pre-training because it means that when we get to post-training, there's already a certain amount of that knowledge there.

21:02Alex Whedon- There's a good foundation to build on. Alex Whedon- And in post-training, we can extrapolate even further. Alex Whedon- And in fact, this is something we showed in our technical report for our sub-Q 1.1 small model. Alex Whedon- We actually have a model variant where we pre-trained the model with 1 million token inputs, up to 1 million, but a lot of 1 million token pre-training. Alex Whedon- And we show that that 1 million token pre-training made it so that when we did post-training, largely also only up to 1 million tokens, that the model is able to extrapolate how to do retrieval style problems up to 12 million tokens without ever seeing anything in post-training over a million tokens.

21:42Alex Whedon- So that's the power of pre-training. Alex Whedon- It enables the model to extrapolate much further than if it's seeing stuff for the first time in post-training. Alex Whedon- Yeah.

Cybersecurity and codebases

21:52Alex Whedon- A lot of people right now, particularly with Mythos and Fable out now from Anthropic and GLM 5.2 from ZI, many people worry about large context models making it easier to find vulnerabilities in code bases. Alex Whedon- How do you view that in both offensive and defensive cybersecurity use cases?

22:33Alex Whedon- Yeah. Alex Whedon- I do think that long context windows, long context reasoning capability does help with finding vulnerabilities in code bases, but that's different than finding vulnerabilities in products. Alex Whedon- So finding vulnerabilities in code bases enables you to be more defensive. Alex Whedon- Finding vulnerabilities in products puts your product at risk. Alex Whedon- So they're two very different things. Alex Whedon- In our case, if you already have all the code in the code base, being able to reason over a lot more of it at once enables you to find vulnerabilities much faster.

23:10Alex Whedon- And also find vulnerabilities you would not have been able to find otherwise. Alex Whedon- And so we see that as something that's really valuable for people being able to make their products safer and more robust. Alex Whedon- That is, some of our design partnerships have leaned in that direction. Alex Whedon- I don't think that long context reasoning specifically provides nearly as much an advantage for being able to find the holes in a product from the outside.

23:45Alex Whedon- Because they're more about iterative tests. Alex Whedon- The context requirement is not necessarily significant. Alex Whedon- But the way this works is software is built layer upon layer of abstraction. Alex Whedon- And so, you know, they're probably like 20 layers of different code components underneath the code that you're writing at any given time.

24:16Alex Whedon- And so, if I'm trying to hack your product, I'm trying to guess which open, I mean, there are other things too, but this is a big part of it. Alex Whedon- I'm trying to guess which open source software you've built on top of. Alex Whedon- That and these open source softwares have known vulnerabilities. Alex Whedon- And so, I'm trying to, I'm trying to, I'm testing to see if any of those known vulnerabilities exist in your software. Alex Whedon- So it's not really a long context reasoning problem.

24:47Alex Whedon- To be honest, I think more than anything, it's actually a knowledge problem. Alex Whedon- It's an existing knowledge of all the existing vulnerabilities. Alex Whedon- And that's why you've actually seen that people can get open weight models to have mythos level capabilities for hacking because the model needs to specifically know. Alex Whedon- To be honest, sometimes like the number of vulnerabilities that lead to most of the threat, it's lesser than some people realize because everybody builds on a lot of the same software components.

25:22Alex Whedon- Yeah. Alex Whedon- Yeah. Alex Whedon- Yeah. Alex Whedon- But couldn't this, I mean, to find those, well, you said they're known vulnerabilities in open source code bases. Alex Whedon- But with mythos or, I mean, with your model that can handle massive contexts, couldn't you then just all the various foundations that are managing open source code bases?

25:54Alex Whedon- They could just dump it all into your model and discover vulnerabilities that maybe they weren't aware of, patch them. Alex Whedon- I mean, isn't it advantageous for the open source community to have solved the context, compute complexity problems so they can scan large code bases and fix vulnerabilities?

26:24Alex Whedon- Yeah, I think that's a great point. Alex Whedon- I think partnering with open source communities to find and fix vulnerabilities faster and find and fix vulnerabilities that have just been there for a long time where speed isn't the issue, it's the breadth of perspective that's the issue. Alex Whedon- That'd be a great use case that I'd love to work with folks on. Alex Whedon- Yeah, you've mentioned design partners a couple of times.

Early design partner use cases

26:53Alex Whedon- What are the most compelling early design partner use cases you're seeing for a large context? Alex Whedon- Is it a finance code, internal documents? Alex Whedon- So unstructured document processing and analysis is like the number one right now. Alex Whedon- And the number two is knowledge based products. Alex Whedon- Within those two categories, finance has emerged as a really important one, just because companies working in the finance space are building such data intensive applications.

27:31Alex Whedon- They're processing so much data. Alex Whedon- That's kind of like the core criteria for whether a design partner or early access user is a good fit for us right now is how data intensive is their application. Alex Whedon- So finance is a really big one for us. Alex Whedon- We have had a lot of interest on the coding side. Alex Whedon- We are intentionally moving a little bit more slowly on the coding side because Alex Whedon- we've kind of realized like how much coding is like Alex Whedon- a budget game, like how much data budget do you have?

28:03Alex Whedon- How much label data can you train on? Alex Whedon- And so Alex Whedon- that's an area that we're really interested in and are working on, but with a bit of a longer time horizon. Alex Whedon- And then Alex Whedon- And then Alex Whedon- it's beyond that it's it's like general enterprise document processing for knowledge bases and unstructured inside extraction. Alex Whedon- Those are the main use cases that we're seeing. Alex Whedon- One thing that's great for us is that I think unstructured document processing and knowledge base products are in pretty much every enterprise out there.

28:41Alex Whedon- And we're also seeing that because we can work with data that is less structured and we can process more data at once becomes a lot easier for an enterprise to get their first value from data. Alex Whedon- So all of these enterprises that are sitting on massive amounts of data and a lot of that data has yet to be put to use in an AI product. Alex Whedon- Or maybe it has, but in a very limited fashion and they're getting pushed to do like a $10 million data transformation project before they could even start to build a product on top.

29:16Alex Whedon- And a value prop that we bring to the table is that is no longer necessary, at least not at that level. Alex Whedon- Like we can help you either work with the data with with a lot less structure or create the structure a lot faster because of the our ability to process a large amount of data at once with a lot less curation. Alex Whedon- That's interesting and as we move into agents and particularly coding agents increasingly which is happening.

29:53Alex Whedon- Most coding agents use very surgical context management. Alex Whedon- I mean they they hop from file to file or section to section. Alex Whedon- What do you think breaks first when we move to agents that can read everything at once? Alex Whedon- Yeah, I think that will a couple things will happen. Alex Whedon- One is we will have systems that can generalize better because every time you add a search engine or vector database, which is a search engine or some conditional logic to route between the steps.

30:38Alex Whedon- This human curation really limits the ability of that system to do a lot of different things. Alex Whedon- It makes it so that it works within exactly the scope that you've set it out to be able to achieve. Alex Whedon- And the more complicated you make it, the more narrow that's going to be. Alex Whedon- So I think we'll see agents that can do a lot more. Alex Whedon- We will also see agents that are a lot faster because if you're doing 10 to 60 steps, it's pretty slow.

31:15Alex Whedon- So I always see agents that are cheaper. Alex Whedon- We want a million tokens to feel like 50,000 tokens in terms of intelligence, cost, latency, etc. over time. Alex Whedon- And so I think we'll see people build a lot more too, to be honest. Alex Whedon- Like context engineering step is where the vast majority of human and capital investment is today. Alex Whedon- And so if we can get rid of that, then we'll see people like the barrier to entry goes down.

31:47Alex Whedon- People build a lot more things. Alex Whedon- And when like a single step can do like 10 steps worth of work. Alex Whedon- What are those 10 to 60 step workflows going to be able to achieve now? Alex Whedon- Yeah. Alex Whedon- It'll be a lot more powerful, a lot longer horizon. Alex Whedon- I guess that's another point. Alex Whedon- We really care about long horizon agents and the ability to manage context, not just through a two hour session, but across, you know, weeks or months of work is something that I think will be critical and will also require much longer context reasoning than we have today.

32:38Alex Whedon- Yeah.

Company founding and background

32:41Alex Whedon- We didn't go into your background or the founding of the company. Alex Whedon- And it's interesting. Alex Whedon- Can you give us that story? Alex Whedon- Sure. Alex Whedon- Yeah. Alex Whedon- So I've been in the AI space for the last 10 years. Alex Whedon- Most recently, I was head of generative AI, a mid-market and enterprise AI consulting firm called TribeAI. Alex Whedon- I did over 40 projects with a series of mid-market and enterprise companies with a medium sized group of folks underneath me.

33:14Alex Whedon- And that's where I got a really good horizontal view of what generative AI implementation was looking like in 2023 and 2024. Alex Whedon- And that's kind of what convinced me that like it was overdue to move to another model paradigm that they were bottlenecks in the enterprise implementation layer that were going to be painful for a very long time unless we change the actual mathematics behind the foundation models themselves.

33:45Alex Whedon- Before that, I was at Instagram working on creator monetization products. Alex Whedon- Before that, I was at Stitch Fix where I built the first text based recommendation system. Alex Whedon- Before that, I was at a research lab where I published some research and built our first production products for our sister companies, Glassdoor and Indeed. Alex Whedon- Before that, I was at Academia.edu where I built some of our first AI products for our 200 million user base.

34:15Alex Whedon- Before that, I did research with the World Bank of Blue Cross Blue Shield. Alex Whedon- And so I've done a lot of different things. Alex Whedon- I'm not a domain expert. Alex Whedon- In fact, that's one of the things I pride myself in is I like to look at things from a very horizontal perspective. Alex Whedon- What's the key value point that's missing across everything everybody's doing? Alex Whedon- And that's really what we're trying to tackle as a company is like, how do we take computational memory and sample efficiency and turn that into downstream business value propositions?

34:46Alex Whedon- Context is the first one, but there will be many more. Alex Whedon- Yeah. Alex Whedon- Well, and you mentioned memory. Alex Whedon- Are you working on memory? Alex Whedon- Absolutely. Alex Whedon- So, I mean, we're working on memory within the context of SSA, subquadratic sparse attention, the thing we talked about publicly. Alex Whedon- We're also working on non-attention algorithms, have been for a while, that make it totally necessary to have any form of KVCache. Alex Whedon- KVCache is this massive thing right now where at the multimillion token range can easily use a lot more memory than the model weights themselves.

35:23Alex Whedon- Becomes very hard to host the context without distributing across multiple cloud data center grade GPUs. Alex Whedon- Makes it impossible, in my view, to do long range modeling for robotics on Jets and Orans, which have about as much memory as your phone. Alex Whedon- And so we see this as a critical problem to solve for long context modeling, for general robotics intelligence, and just the ability to create much better frontier models than exists today.

35:58Alex Whedon- This long context, you mentioned that most enterprises don't even fully use 256K context. Alex Whedon- As you introduce them to multimillion token windows, what are you learning about user preferences and behaviors? Alex Whedon- Yeah, that, I mean, that they're not well mapped out and they can vary.

36:34Alex Whedon- So we found, for example, people will tell us that on a news article, they'll ask, what are the top three insights from here? Alex Whedon- And the model will give three top insights that are technically accurate, but they're not the three they cared about. Alex Whedon- Yeah. Alex Whedon- So like this is the level of alignment that becomes necessary with long context reasoning gets very complicated. Alex Whedon- So one of the things we learned is that we are so far from this area of modeling being mature.

37:09Alex Whedon- I think people say, well, the frontier models have a million token context window, but like their ability to deliver what people want at that size is just completely. Alex Whedon- We are far removed from that today. Alex Whedon- And prompt ability is still a bit lacking also for these types of problems. Alex Whedon- And I think to some extent, like this problem can't be solved at the prompt level, because it's just so complex and nuance like we do, we do actually need to solve this through training first, where we see a very wide number of problems that's

37:47Alex Whedon- A very wide number of inputs and outputs or problems through RL that show what user preferences are at scale before this becomes a problem that we can really solve through prompting. Alex Whedon- So that's a super high level insight beyond that. Alex Whedon- I think reasoning is pretty unsolved. Alex Whedon- Oh, here's an interesting one, actually.

38:19Alex Whedon- We did some, we created some evals for code-based question answering and financial document question answering. Alex Whedon- And we ran, we did them at 500,000 tokens, a million tokens per Opus 4.7, which is like 800,000 tokens for us. Alex Whedon- And then as close as possible to 2 million tokens, which obviously only we could do. Alex Whedon- And so we found on code-based question answering,

38:49Alex Whedon- There were like different levels of difficulties, like we could say easy, medium and hard. Alex Whedon- And on the easy and medium problems, like the models, most frontier models and our model could do pretty well, pretty close to 100%. Alex Whedon- And there was really not much of a concern there on performance. Alex Whedon- But it is quite easy to create an eval set that on financial document analysis that is well below 50% at 500,000 or 800,000 tokens.

39:24Alex Whedon- And so it does seem like, obviously there's a bias towards code. Alex Whedon- I think that the model companies have spent like more time in some areas than others. Alex Whedon- And then also I think there are certain types of contexts that are harder than others. Alex Whedon- Like I've been working with financial documents for a long time. Alex Whedon- I've spent a lot of time working with Edgar data. Alex Whedon- And there are a lot of things that make it hard.

39:55Alex Whedon- Some of them are intentional. Alex Whedon- But the structure of the PDFs can be very challenging for LLMs to handle. Alex Whedon- Data can be really distributed up to several hundreds of pages. Alex Whedon- You've got tab viewer and non-tab viewer data. Alex Whedon- You've got tables that are really hard to parse correctly. Alex Whedon- All these challenges that I think make these problems harder than code in some ways.

40:25Alex Whedon- So I guess the TLDR here is that long reasoning capability today is very asymmetric across verticals. Alex Whedon- Yeah. Alex Whedon- You mentioned that you give these models, the standard transformer models, a large context or a large chunk of text in the context window and ask it to surface the most significant things.

41:01Alex Whedon- And it doesn't come up with what you think is most significant. Alex Whedon- How does this, is the reasoning stronger with your architecture or is it that it sees more of the context? Alex Whedon- I mean, why is it better with SubQ's model? Alex Whedon- So the biggest part of it is the training differential that, you know, we can end to end our model training speed is still like somewhere between 5 to 10 X faster at a million tokens.

41:44Alex Whedon- But then as you move to 2 million tokens and beyond, that's higher. Alex Whedon- The differential is even bigger than that. Alex Whedon- And so it's easy for us to do a lot more long context training than other people are doing. Alex Whedon- And we're also very focused on generating a lot more of that data through both synthetic generation and human labeling. Alex Whedon- And so, you know, focus and scale is a big part of that. Alex Whedon- This is a phenomenon that we call the bitter lesson in the AI space. Alex Whedon- Yeah. Alex Whedon- Yeah.

42:14Alex Whedon- And so we, so that's a big part of it. Alex Whedon- I think there's a theoretical benefit that I don't feel like we've fully proven one way or another. Alex Whedon- But that is that we are reducing the noise for the model. Alex Whedon- So instead of looking at a million squared relationships, it'll look at, you know, scaler value times a million token relationships. Alex Whedon- So has less to consider, less to model on top of.

42:44Alex Whedon- And so the same way that like RAG reduces the noise and could sometimes lead to better results. Alex Whedon- I feel like sparse attention could also reduce the noise and lead to better results. Alex Whedon- But again, I don't feel like we've fully proven that one way or another. Alex Whedon- It is not going to be easy to prove because A, how do you separate the architecture from the training? Alex Whedon- And B, what evals are we going to use that are actually helpful today? Alex Whedon- I think we have a massive dearth of high quality long-conducts reasoning evals.

43:17Alex Whedon- Yeah. Alex Whedon- And if someone wants to play around with this, how do they do that? Alex Whedon- So today, I mean, you reach out to us, we'd evaluate you for a design partnership. Alex Whedon- The number of, we're not giving broad access to the model yet just because we have limited compute bandwidth ourselves and are trying to prioritize access to people that we think could help us make a better version of the product. Alex Whedon- And then, and we are continuously trying to increase the amount of compute bandwidth we have.

43:54Alex Whedon- These additional, these early design partnerships are also helpful for that too. Alex Whedon- The market validation enables us to increase our capital budgets and, and thus compute budgets. Alex Whedon- So yeah, I mean, that's kind of the deal of the R is like, reach out, we'll connect if it seems like an interesting fit. Alex Whedon- We usually go through an evaluation process, scope a design partnership, and then get started. Alex Whedon- That'll change over the next couple of months. Alex Whedon- In the next couple of months, we do plan to have complete general access and by then we will have a very different version of the product too.

44:26Alex Whedon- Our models are continuously undergoing training to respond to feedback that we're getting from partners. Alex Whedon- So, so if anybody thinks the model is good now, it's just going to get a lot better. Alex Whedon- Yeah. Alex Whedon- And when you do have general availabilities, is there going to be a free layer or a free trial or, or how, how are you going to get people in the door? Alex Whedon- So yeah, I think we'll have some level of free access for sure.

44:59Alex Whedon- And we'll have tiered pricing and tier rate limits and all of that. Alex Whedon- So it'll be pretty standard. Alex Whedon- There are some non-standard things that I, that, that we are considering doing. Alex Whedon- One of them is like a higher output to input ratio. Alex Whedon- Today, the output token cost is usually three to five X the input token cost. Alex Whedon- We'd like the input token cost to be a lot more cheaper.

45:29Alex Whedon- Oh, sorry, a lot cheaper still. Alex Whedon- To make it, we really want people to feel like the input tokens are free. Alex Whedon- Like just consider the context that you need to for your problem. Alex Whedon- So I think we'll see higher ratios there. Alex Whedon- And then secondly, we've hesitated saying this a bit, but we've considered. Alex Whedon- I think our, our pricing curve as your inputs grow could look different than, than others because our internal price economics look different for it.

46:00Alex Whedon- The external pricing for that could also look different. Alex Whedon- I guess that's all I'll say for now. Alex Whedon- Yeah, and I have to ask, you know, there's a lot of talk about using AI for scientific research. Alex Whedon- Did you guys use any models to design this? Alex Whedon- I mean, did you talk to, you know, Claude 4.8 or now I guess Fable 5 and, and

46:33Alex Whedon- You know, how do we solve this problem and, and go back and forth with a model? Alex Whedon- I'm curious how, how companies like yours are, are doing that architecture design, whether they're tapping the, the knowledge base of the model or, or whether it's all, you know, iterative trial and error among humans. Alex Whedon- Totally.

47:04Alex Whedon- So, I mean, they're, we're definitely using LLMs in the process and always trying to as much as possible. Alex Whedon- I don't think you can get there with just an LLM. Alex Whedon- Sure. Alex Whedon- They're not enough today. Alex Whedon- I don't think they're creative enough either. Alex Whedon- But they're definitely super helpful. Alex Whedon- And I will say sometimes I'll just open-endly ask an LLM to come up with an idea and it comes up with something I hadn't thought of. Alex Whedon- So they're super helpful for that. Alex Whedon- But, but we've been talking from probably like somewhere between a year and a

47:42year and a year and a half about how to get to auto research for algorithms before, you know, way before auto research was termed, way before RSI was termed because of self-improvement. Alex Whedon- We've been trying to build that product. Alex Whedon- And so I want to be the first company that creates this high quality flywheel of a model proposing architectural variations, creating the experiments to validate or invalidate those variations,

48:13running those experiments, looking at the outputs, coming up with what the idea is for the next set of experiments and architectural variations until it comes up with an architecture that looks vastly different and is vastly better on the axes that we care about. Alex Whedon- We will be investing into that very seriously. Alex Whedon- Yeah, that's interesting.

Future roadmap and robotics

48:35Alex Whedon- Looking ahead, what's the most exciting frontier for you? Alex Whedon- I mean, at SubQ, is it bigger contexts and not only at your company, but in across AI research in general, new architectures beyond attention or something else? Alex Whedon- I mean, yeah, a lot of different things. Alex Whedon- Obviously, I think the auto research thing is really exciting. Alex Whedon- Yep. Alex Whedon- I think that, but again, it's not about doing the pre or post training better, which is where most people are focused.

49:12Alex Whedon- It is about the algorithm itself. Alex Whedon- We want to move from like shifting the algorithmic paradigm every nine years to doing it every 12 months. Alex Whedon- So we want to replace transformers or attention entirely in the not too distant future. Alex Whedon- And then whatever we replace that with, I mean, we know what we want to replace it with. Alex Whedon- But when that happens, we want to replace that again within 12 months and just shorten this time as much as possible.

49:44Alex Whedon- So that's interesting. Alex Whedon- I think that from a go to market perspective, we're really excited about working with enterprises on long context reasoning tasks. Alex Whedon- Long horizon agentic tasks as a secondary priority. Alex Whedon- And then in the somewhat longer term, though there is active work being done on it. Alex Whedon- We do care about robotics and see that it's highly intertwined with our current work, actually.

50:16Alex Whedon- Like the reason why the figure AI demo was on the conveyor belt flipping boxes is because it requires a 15 second memory span. Alex Whedon- If you think about asking a robot to do the laundry, the robot has to remember the roadmap of the house to be able to navigate. Alex Whedon- It needs to remember how to open the laundry machine door. Alex Whedon- It needs to understand, it needs to know what cycles are appropriate for which clothing items.

50:47Alex Whedon- It needs to remember what the user preferences are for how to fold each of the items. Alex Whedon- It needs to remember where the clothing goes. Alex Whedon- And so the amount of context that's required there is significant. Alex Whedon- And then additionally, we are in the same place for robotics intelligence as you were for language models in 2019, where we train models for specific tasks with thousands of task specific samples that were hard and costly and timely.

51:20Alex Whedon- Time expensive to curate. Alex Whedon- And so I think we need a 2020 LMR few shot learners GPT three moments for the robotic space. Alex Whedon- But if you think about providing a 10 minute video sample so that the model can learn in context how to achieve a task without being trained on that task specifically, that's like a 4 million token input. Alex Whedon- Yeah.

51:50Alex Whedon- And people are struggling to do that in the cloud. Alex Whedon- How do you do that on a phone worth of VRAM? Alex Whedon- That requires like a 100x plus reduction in memory usage. Alex Whedon- And that's a problem we're thinking about a lot. Alex Whedon- So that's a pretty exciting frontier for us. Alex Whedon- There are a lot of exciting things that we're doing here.

52:20Alex Whedon- And some of the stuff is, you know, stuff we're trying to put out in the next six months. Alex Whedon- Some of it is things that we're trying to put out in the next two years. Alex Whedon- So we've got a longer term roadmap. Alex Whedon- We're not just thinking about incremental improvements in the industry. Alex Whedon- We want to figure out what problems are going to matter two to five years from now. Alex Whedon- How do we like take a shortcut to get there? Alex Whedon- And how big is the team? Alex Whedon- So we have just under 50 people today.

52:51Alex Whedon- We've grown a lot in the last couple months. Alex Whedon- I mean, the fundamental research that we did that we've announced was done by just a few people. Alex Whedon- But we've been sitting on a lot of research for a while. Alex Whedon- That research was done in 2025. Alex Whedon- And then productized this year. Alex Whedon- But it was done with a lot fewer resources and people than we have today. Alex Whedon- And so we've done a lot since then. Alex Whedon- And yeah, I mean, it's a little under 50 people today.

53:22Alex Whedon- Okay. Alex Whedon- Well, this is all fascinating and I'm glad that we've met. Alex Whedon- I'm certainly going to be following you guys. Alex Whedon- If listeners want to follow what you're doing, is it subq.ai? Alex Whedon- Do you have a blog or something there that people can follow? Alex Whedon- Yeah, subq.ai is good. Alex Whedon- We've released a couple blog posts.

53:54Alex Whedon- I wouldn't say that we're prolific bloggers yet. Alex Whedon- We do want to try to push out more, but it does take some time. Alex Whedon- Yeah. Alex Whedon- I am trying to post at least once if not a couple times a week on Twitter. Alex Whedon- And I do want to start going a little deeper on some of our product and technical and research vision here in some of those posts. Alex Whedon- And I'll call out different things every once in a while. Alex Whedon- So that could be an interesting place to follow us as well. Alex Whedon- We have our official Twitter page and you may see some of our other employees

54:26being more public as well about some of their work. Alex Whedon- Social is going to be a pretty good place to stay on top of what we're doing as well. Alex Whedon- Yeah. Alex Whedon- Okay, Alex. Alex Whedon- This is really interesting. Alex Whedon- Alex Whedon- You