The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:45:29 · Thomas Sohmers
[00:45:29] S0: I would say, like, right now, somewhere around 80, 85% of all tokens consumed in Produce are done by just the top four model companies. And I would say the next five or 10% is done by the three or four after them. So I can totally buy... Believe that 5% of all tokens consumed will be done by things on prem, you know, locked into the big guys. But both from Positron's business perspective and just how I see the world evolving, I... I'm going to care more about the the high volume set of things. That being said, I actually think the the small model stuff is actually much more interesting for everything happening on on your phone. And and the amazing thing about that is I think there's this misconception that, oh, if questions could be answered by your phone or or, you know, any prompts can be can be done locally, that actually is meaning that there's less tokens going to be used with the big guys in the cloud. I think it's the opposite. The the reality is that if I have an LLM running on my laptop or phone or in my enterprises, you know, secure on prem cloud, whatever, that is going to be consuming data at such a fast rate of everything coming into it, and it's going to be generating, you know, analysis based on that. And it will decide, okay, what is it that I'm going to actually return to the user, you know, locally, and what is it actually requires more intelligence from a better model that isn't self hosted? I think for, you know, any of the tokens that are being, quote, unquote, saved by running locally, that's actually going to generate more things. Because, I mean, in in some ways, for for a simple naive use case of a personal user of LLMs, they're only going to prompt ChatGBT or Quad every so often. Like, they're they're sort of limited by their thoughts of when to actually ask an LLM something. But if they have a local LLM that is constantly checking their email, their calendar, messages, etcetera, deciding to do these lookups to cloud hosted models frequently, that's now on a per person basis, a massive increase in the no... Number of tokens being consumed and generated by the cloud models even though there was... The naive view is that, oh, there's the shift to this on device, LLM.
00:47:35 · Harry Stebbings, Thomas Sohmers
[00:47:35] S1: So can I just understand? So I completely hear you in terms of maybe 5% of them will be in this smaller model enterprise owned kind of model landscape. Why are you so bullish then on much larger models and the size is increasing? [00:47:49] S0: I'll break it into portions. So there's the increase of the model sizes, which I think of... The thing that would be shared by a lot of people in the AI space is it's kind of a gut feel where based on the fact that we have seen these scaling laws. Like, we we call them scaling laws. The fact that going from a, you know, 100,000,000 to a billion to 10,000,000,000, 100,000,000,000, 1,000,000,000,000 parameter models, we've seen this amazing increase in capabilities with that. We see that, like, still to this day going from a trillion to five to 10,000,000,000,000 at that... The largest end right now. Like, we call it a law because we've observed it, but there's no, like, actual mathematical proof that this will continue. So it's sort of on vibes that, okay, this has continued scale. There's no sign of it slowing down. So is that going to continue to 50,000,000,000,000, 100,000,000,000,000, and and beyond? And I don't see any indication that that's going to stop, so I'll I'll be bullish [00:48:41] S1: about does scaling laws look like at three times what it is now? If AGI has been declared now by Janssen, forgive me, but what is three times this?
00:48:41 · Harry Stebbings, Thomas Sohmers
[00:48:41] S1: about does scaling laws look like at three times what it is now? If AGI has been declared now by Janssen, forgive me, but what is three times this? [00:48:51] S0: That's a good question. I mean, GPT six Astra, my first, you know, twenty four hours with it were basically as magical as my first experience with GPT 3.5 in November '22. I I was at the, ChatGPT launch at NeurIPS in 2022, and it was so funny because, you know, Sam and and Ilya were there, and and they... You know, it was a party in in New Orleans for the NeurIPS conference. And, basically, at the end, they just said, hey. We launched this little fun, you know, experiments called ChatGPT. Go check it out. And, like, zero fanfare, you know, it was it was really just a side mention. And I don't think anyone really gave it a thought at the event. But when I went back to the hotel, I loaded it up, and I got back at, you know, ten, 11PM, or whatever. And I was up for four or five hours straight just giving random props and and this that this was the most magical experience that I've ever had with a computer. I would say I got very close when Sora two came out that I had similar experience, short amount of time, but just mind blown by the quality of the the videos and especially the weekend that Sora two launched when there being no restrictions on what you could generate. But, yeah, GPT six Astra, I do think is is AGI. And to to your question of, like, what does that mean going forward? I think my guess is this is as good as basically anyone's. But
00:48:51 · Thomas Sohmers, Harry Stebbings
[00:48:51] S0: That's a good question. I mean, GPT six Astra, my first, you know, twenty four hours with it were basically as magical as my first experience with GPT 3.5 in November '22. I I was at the, ChatGPT launch at NeurIPS in 2022, and it was so funny because, you know, Sam and and Ilya were there, and and they... You know, it was a party in in New Orleans for the NeurIPS conference. And, basically, at the end, they just said, hey. We launched this little fun, you know, experiments called ChatGPT. Go check it out. And, like, zero fanfare, you know, it was it was really just a side mention. And I don't think anyone really gave it a thought at the event. But when I went back to the hotel, I loaded it up, and I got back at, you know, ten, 11PM, or whatever. And I was up for four or five hours straight just giving random props and and this that this was the most magical experience that I've ever had with a computer. I would say I got very close when Sora two came out that I had similar experience, short amount of time, but just mind blown by the quality of the the videos and especially the weekend that Sora two launched when there being no restrictions on what you could generate. But, yeah, GPT six Astra, I do think is is AGI. And to to your question of, like, what does that mean going forward? I think my guess is this is as good as basically anyone's. But [00:50:12] S1: Why was GPT Astra so good for you? Why was it comparably such a breakthrough? Because I I haven't... It's great, but honestly, kinda the same as before.
00:50:20 · Thomas Sohmers
[00:50:20] S0: Oh, in terms of the things that I've found LLMs to fail the most at in the past. So I'll give a case where it is more linear improvement. So just in terms of general coding capabilities, performance, and analyzing problems, etcetera, it is a step function improvement, but not mind bogglingly. So there are a bunch of things that other models have not been able to fix or kind of went in circles and found inelegant solutions and it's still like a human software architect that was able to to come up with a better solution. With Astra, they're just initially giving in a couple of really hard problems that I've not been able to solve with other LLMs. Was able to do it one shot, having it go through a code base and find both performance improvements, bugs, etcetera. Basically, discovering new spaces that I didn't know existed in our our bunch portions of our code base. So that's one element. Step function, but not mind boggling. The second case that was mind boggling just from a, like, wow, was the the computer usability is with set of generic tools. So, like, being able to do Blender animations. Like, you know, it's become... There... There's a bunch of memes online of of it recreating different videos, etcetera. But just the fidelity of that and where that was basically impossible with GPT 5.6 Soul was massive increase capability. And, like, I had it design... You know, do interior design of my house just based on a couple pictures and just like, wow. Like, I did not think that at... What's fundamentally a text model could could do that. And then finally, like, the the biggest thing for Positron was I've been trying with every single new model release to have these models be able to, like, actually take a relatively simple logic design problem, you know, implementing a a encryption block in this case, and being able to take that through the full RTL to GDS flow. So from basically the specification of do this encryption function, implements the Verilog, so the hardware description language for that. So write that code and then be able to take that code and go through all the way until you have got a chip design that theoretically you could go tape out. LMs could do different portions of that and could, like, write the scripts and, you know, fail a lot of different midpoints on the way.
00:52:31 · Thomas Sohmers
But a big problem with the electronic design automation tools, the EDA tools for doing chip design, is that they were designed in the nineties, early two thousands. They're really unintuitive. None of the documentation exists out, like, in the public web, and so that these models don't have, like, a real good innate view of them. But GPT six, with both combination of computer use and just an ungodly amazing scripting ability has been able to take this this tech hack block and implement it with the TSMC n three PDKs and take that all the way to GDS and do that in, like, a little over, like, 50 and meet timing at, you know, over a gigahertz, etcetera. And that as a task, if I was giving to someone similarly new to a thing, like, getting the flow mostly working, I would say would take on the order of a week And getting it optimized to the point that Astra is at with with that design would maybe be one or two additional weeks depending on the person. So compressing that two to three weeks down to two days and change, it's still mind boggling. It... Like, it shouldn't be this good at this as I would naively think about its training sets. But, obviously, with OpenAI's own chip developments, you know, in house, they've... I'm I'm glad that those capabilities are getting added to the the models they're releasing to the public and not just being kept in... Inside.