The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:53:48 · Harry Stebbings
[00:53:48] S1: We see Jalapeno, terrible name, I think, personally, but, you know, their own chip development, Anthropica developing their own chips, DeepSeaker supposedly developing their own chips. We see the commoditization of the chip player with everyone building their own ships. How should we think about that?
00:54:02 · Thomas Sohmers
[00:54:02] S0: As a consumer of all these things, if I take my Positron hat shirt off, I would say that that's a great thing for the industry having, fundamentally, that's gonna bring costs down and capabilities up and and bring it to to more people. I think it's such an interesting world where when I got started in the semiconductor space, you know, thirteen years ago, you know, silicon was a dirty word in Silicon Valley. And now you have all the biggest companies in the world being, you know, somehow connected to the semiconductor industry and the most interesting, exciting applications and the the companies building them, you know, vertically integrating down to the silicon layer. The interesting thing with all the the ones that you mentioned and and, you know, the broader set is companies have the same macro goals. The implementation details are all unique though, and that's... Just as an engineer and and technologist, it's exciting to me that there are a lot of different ways to skin a cat, and people can have their own architectural view and and go about implementing it and and get different results. And, you know, hot chips, the biggest conference for for this design space and was where OpenAI unveiled Jalapeno last month. I'm happy that the industry is still pretty open and willing to share not as much details as people would have shared, you know, five, six years ago, but still a good amount of in the open discussion of of things. And so I think how that applies to Positron is, you know, we have our particular architectural views and way that we've decided to do things and that will evolve in in the future as will everyone else's. But, you know, there's still plenty of space to make bets and go in different directions. And the great thing about the market is that the market gets to to decide what is is valuable, and those that create value will receive a reward for that.
00:55:50 · Harry Stebbings
[00:55:50] S1: We spoke about context window length earlier and the expansion of it. How much does that expand? Is that infinite expansion capability of context window length? And what does that mean we can do that we can't do today? I'm just fascinated.
00:56:04 · Thomas Sohmers
[00:56:04] S0: I think with traditional linear or, quadratic attention, there is going to be limits of scale when it came to what the hardware could provide. Now one of the big things for Positron is we're trying to massively increase the memory capacity per device. You know, with our our upcoming generation, we're going to have, you know, eight times more memory capacity than the highest memory SKU from NVIDIA, and NVIDIA is actually decreasing the amount of memory per device, you know, based on the market memory conditions. But I frankly think that context length going from, like, a million tokens that it is today, going to 10 or even great... Much more than that is really, really hard with that quadratic expansion memory cost. The algorithmic advancements that have happened over the past year have been very, very interesting in terms of being able to further reduce the amount of of storage and and compute necessary for that context with linear and sparse attention mechanisms. And those have really, really been innovated by the Chinese model labs. And this is a great example of when you have constraints of, you know, we had export controls on the chips with the highest memory capacity and FLOPS, and so they innovated on not needing that. And so, you know, DeepSeek beginning of of twenty twenty five with DeepSeek v three had, you know, made a lot of waves because they were able to to get massive decrease in KV cache size with multi head latent tension. So you were actually spending more flops to be able to have a smaller KV cache, and that's advanced a lot over the past year and a half. And probably the the most interesting or my my personal favorite right now is Gated Delta Net and and its derivative versions where you can have, like, a 75% decrease in the total time, you know, you're spending on on the attention portion, you know, with with this mechanism. So
00:57:50 · Harry Stebbings
[00:57:50] S1: How important then is new hardware if DeepSeek without it, just on architecture innovation alone can cut cost by 80%?
00:57:59 · Thomas Sohmers
[00:57:59] S0: Yeah. I would say that, like, there's no such thing as free launch. So when they have that MLA compression, it does come at cost of model capabilities in in some form. And, like, there's a reason why the Chinese labs have really heavily embraced, you know, MLA while none of The US labs have. I should say based on rumors, but I also have a a very good information and belief that, you know, none none of the major US companies are... They they definitely are not using MLA and not using some of its, you know, brethren. I think that that that will evolve and change in the future. But, yeah, that bay... Basically, the short version of that is it's not just a a pure savings on that on that side. But the the reality is to... And to go back to your previous question of, like, everyone does want greater context length. Like, if I had 10,000,000 token context length, I think that would be enough for holding multiple of our largest code bases and really have that cross pollination happen between them for an agentic coding model. And I guess one one thing I should have explained, it's not just having the full context. Like, there were some early models that advertised a million token context length. But as soon as you went above, like, 64,000 tokens, you know, that's two years ago, its recallability just went garbage. So just saying that something has this maximum context length is one thing. It's... Can it actually use that context length effectively is entirely different thing. And that's actually going back to the Astra thing. Amazing thing about it. There... There's a couple different benchmarks measuring, like, the long context performance. One of them is called ruler, and there's, like, these find a needle in a haystack. So you just flood the the context window with a bunch of junk, basically, like just passages from books and all this. And you just put somewhere randomly in the middle of all of that, you know, a a hash, you know, some value that looks out of place. And you prompt the model, say, what's the secret value? And a lot of models have done really poorly on this. GPT 5.6, which is only, like, six or seven weeks old, only could do this about 70% of the time. GPT six Astra does it, like, over 95% of the time correctly. So that's... There... There's a lot of room for improvement in these things.