The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:56:04 · Thomas Sohmers
[00:56:04] S0: I think with traditional linear or, quadratic attention, there is going to be limits of scale when it came to what the hardware could provide. Now one of the big things for Positron is we're trying to massively increase the memory capacity per device. You know, with our our upcoming generation, we're going to have, you know, eight times more memory capacity than the highest memory SKU from NVIDIA, and NVIDIA is actually decreasing the amount of memory per device, you know, based on the market memory conditions. But I frankly think that context length going from, like, a million tokens that it is today, going to 10 or even great... Much more than that is really, really hard with that quadratic expansion memory cost. The algorithmic advancements that have happened over the past year have been very, very interesting in terms of being able to further reduce the amount of of storage and and compute necessary for that context with linear and sparse attention mechanisms. And those have really, really been innovated by the Chinese model labs. And this is a great example of when you have constraints of, you know, we had export controls on the chips with the highest memory capacity and FLOPS, and so they innovated on not needing that. And so, you know, DeepSeek beginning of of twenty twenty five with DeepSeek v three had, you know, made a lot of waves because they were able to to get massive decrease in KV cache size with multi head latent tension. So you were actually spending more flops to be able to have a smaller KV cache, and that's advanced a lot over the past year and a half. And probably the the most interesting or my my personal favorite right now is Gated Delta Net and and its derivative versions where you can have, like, a 75% decrease in the total time, you know, you're spending on on the attention portion, you know, with with this mechanism. So
00:57:50 · Harry Stebbings
[00:57:50] S1: How important then is new hardware if DeepSeek without it, just on architecture innovation alone can cut cost by 80%?
00:57:59 · Thomas Sohmers
[00:57:59] S0: Yeah. I would say that, like, there's no such thing as free launch. So when they have that MLA compression, it does come at cost of model capabilities in in some form. And, like, there's a reason why the Chinese labs have really heavily embraced, you know, MLA while none of The US labs have. I should say based on rumors, but I also have a a very good information and belief that, you know, none none of the major US companies are... They they definitely are not using MLA and not using some of its, you know, brethren. I think that that that will evolve and change in the future. But, yeah, that bay... Basically, the short version of that is it's not just a a pure savings on that on that side. But the the reality is to... And to go back to your previous question of, like, everyone does want greater context length. Like, if I had 10,000,000 token context length, I think that would be enough for holding multiple of our largest code bases and really have that cross pollination happen between them for an agentic coding model. And I guess one one thing I should have explained, it's not just having the full context. Like, there were some early models that advertised a million token context length. But as soon as you went above, like, 64,000 tokens, you know, that's two years ago, its recallability just went garbage. So just saying that something has this maximum context length is one thing. It's... Can it actually use that context length effectively is entirely different thing. And that's actually going back to the Astra thing. Amazing thing about it. There... There's a couple different benchmarks measuring, like, the long context performance. One of them is called ruler, and there's, like, these find a needle in a haystack. So you just flood the the context window with a bunch of junk, basically, like just passages from books and all this. And you just put somewhere randomly in the middle of all of that, you know, a a hash, you know, some value that looks out of place. And you prompt the model, say, what's the secret value? And a lot of models have done really poorly on this. GPT 5.6, which is only, like, six or seven weeks old, only could do this about 70% of the time. GPT six Astra does it, like, over 95% of the time correctly. So that's... There... There's a lot of room for improvement in these things.
01:00:10 · Harry Stebbings, Thomas Sohmers
[01:00:10] S1: Speaking of room for improvement, can I ask you, when I was doing the research for the show, I saw that the Silicon Data Token Price Index dropped below $1 per million tokens this month, and five years ago, it was $60 per million? $60 to one. What does a million tokens cost in 2028, say, two years from now, do you think? [01:00:33] S0: I think the thing I care a lot more about than just that 60 to one is the fact that a $60 token five years ago, no one would pay a cent for today. Like, that was a complete garbage token, relatively speaking, five years ago. And the level of quality of... For a token that you pay a dollar per million tokens for now is, so much astronomically more valuable. [01:00:56] S1: And so that's because of token efficiency and what can be done?
01:00:56 · Harry Stebbings, Thomas Sohmers
[01:00:56] S1: And so that's because of token efficiency and what can be done? [01:01:00] S0: No. I'm saying just in model capabilities. If you if you say, k. So so it's 2026. So the best model in the world in 2021 was GPT three. It's kinda crazy at the rate models get released today that GPT three was the best in the world, basically, from 2020. I think it was August 2020 when it released all the way up to... They they didn't have a new release until ChatGPT in November 2022. So it was, you know, two, two and a half years between between model releases. And, really, GPT 3.5 was just doing reinforcement learning with human feedback on the same base model. So if you remember how bad a GPT 3.5 was and, like, what was the value of that in terms of economic productivity value of GPT 3.5 versus GPT six today or, you know, pick whatever comparison points you want. The value per token in terms of what it can improve a person's life, you know, a company's, you know, business practices, etcetera, is orders of magnitude. I would say a 100 or a thousand fold. I I think there's actually two points to to your your access of going from $60 to $1. Yes. That's a decrease in cost. But that token today is, let's just say, I think, conservatively, a 100 times more valuable. So I would actually be saying that there needs to be some multiplier there as well where, like, the value per, you know, unit of intelligence is probably closer to a thousand fold, not not just the 60 fold you're you're talking about.
01:02:27 · Harry Stebbings
[01:02:27] S1: What does that mean then if we extrapolate that out to 2028? What does that mean that... Like, does the cost of a token then actually matter? Like, is that the primary unit that we should measure? Everyone talks about cost of token. Is there actually a different metric that we should measure?