The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:00:00 · Thomas Sohmers
[00:00:00] S0: I would say I'm overall opposed to, you know, the pace of the frontier direction it's going in. The scariest thing to me on on the political spectrum and the way all of this being treated is that it's now become a almost unifying issue on left and right about being anti data centers. I think that is almost entirely a Chinese SIOP. A single in and out uses, you know, more more water than, you know, the largest data centers in The United States.
00:00:22 · Harry Stebbings
[00:00:22] S1: This is 20 VC with me, Harry Stebbings. Now we have probably one of the most important times in history for technology. Have the biggest model providers saying we need to pace the frontier. But what does that actually mean in reality? How possible is it? What does it mean for the threat from China? What does it mean for the infrastructure layer moving forwards? We have a true expert of the space on the show today in the form of Thomas Somers. He's the cofounder and chairman of Positron AI. They just raised an $875,000,000 series c at a $5,000,000,000 valuation. They've got some of the best investors in the business, including the one and only Gavin Baker at Atredi's and many more great names. Thomas did not hold back in this episode, and it's this beautiful combination of incredible education on the infrastructure that powers this economy for AI. And then also, I don't know how to say it, but analytical gossip would be a more intellectual way of saying incredible discussion about what we can expect in the next few months from the biggest players in this space. But before we dive into the show today, the best model for your application might not exist yet. The most ambitious AI teams are training open models to beat the frontier in their domain. Now on November 3 in San Francisco, Forge by Fireworks brings those teams together. People like Jensen Huang, CEO of NVIDIA, Michelle Cataster, president and head of AI at Replit, Lin Kwao, CEO of Fireworks, and more to hear how AI leaders are taking control of their differentiation, margins, and road map by owning their intelligence. Learn how they're building it with inference, training, and intelligent routing. Forge is free to attend, but space is limited. Apply today at fireworks.ai/forge. While Fireworks AI powers product intelligence, Asana keeps the work moving. Most companies have tried AI. Most aren't seeing results. Results. Not because AI doesn't work, it's because AI hasn't reached the workflows yet. That's the gap Asana is built to close. Asana is the operating system for human agent teams. Your easy button for AI productivity across every team. Ready to go AI teammates, prebuilt for marketing, ops, and IT. No prompt engineering, no setup. They show up where the work is happening, already onboarded in your workflows, ready to deliver.
00:02:35 · Harry Stebbings
With Asana, your whole company can work on the same plan towards the same goal, whether you're a team of 10 or a team of 10,000. Asana, where humans and agents workflow together. Try it at asana.com. That's asana.com. While Asana organizes the work, Superhuman speeds up the day. Can I be honest with you for a second? Here's what my week actually looks like. Back to back meetings all day, and the real pressure builds up in the gaps between them. I'll come out of six founder meetings with a whole stack of follow ups waiting. And my inbox, it's hundreds of messages. And somewhere in there is the one thing that really matters, which I always tell my mother is her. I've tried AI tools for this, but they always lived in another tab. I'd have to stop, go find it, paste in the context, write the perfect prompt. It was just enough friction that it never really stuck. Superhuman Go from the makers of Grammarly, totally different. It's right there where you need it, when you need it, in the doc, when I'm reading, the email I'm writing. When a thread gets dense, I highlighted and asked Go to pull out what matters without losing my place. And after a run of back to backs, I asked what was actually decided, and Go turns it into clear action items or drafts, the reply I need to send. It's already up to speed, and I'm always the one in control. Try Superhuman Go from the makers of Grammarly and find out more at superhuman.com. You have now arrived at your destination. Thomas, I am so excited for this, dude. I said to you just before this that I think there are some big questions that the world doesn't know or things that they know that I think we're gonna correct today. So thank you so much for joining me.
00:04:13 · Thomas Sohmers, Harry Stebbings
[00:04:13] S0: Yeah. Great to be here. Thanks, [00:04:14] S1: Harry. Can we just have a brief description of what positron is and where does it sit in the stack? [00:04:21] S0: Yeah. So Positron is a fabulous semiconductor startup that's building hardware. So really everything from the chips, the software directly running on top of that, all the way up to the full systems and rack scale deployments to power generative AI inference. So effectively, everything in the hardware and sort of the low level direct talking to hardware software stack that powers all of the applications, you know, everyone in the world is excited about right now. So everything from, the likes of ChatJPT, Claude, etcetera. [00:04:50] S1: How does the infrastructure stat required for inference, what you're working on, change compared to training?
00:04:58 · Thomas Sohmers
[00:04:58] S0: So training, I would say, from the underlying compute level, fundamentally, you know, is a compute bound problem. So it's a workload that's the more flops that you have. And if you look at, you know, be it from the regulatory and some of the, you know, export control frameworks are are heavily focused on the just FLOPS required. So how many floating point operations per second can be can be done. And more or less, you know, the the amazing thing that the scaling laws of the past decade have shown is that the more parameters you you add to a network and the more FLOPS you dedicate to that during training, the better that that model is going to become. The big difference with inference and so the the deployment of those models is the fact that for the actual math and the the steps that you're doing is about half of what you're doing during training in terms of the the steps not... Shouldn't be thought of as, like, the actual compute involved. But what it turns out to be is that that forward pass, that inference portion of it is heavily, heavily memory bound due to the fact that basically for every single token that's generated, every little bit of output, that requires going through the the weights, the parameters you could kind of, you know, from a biological sense, think of the neurons. You have to read the values of that for every single individual token. And so the way that that... That's, I I think, very interesting about this... And and I I don't know if it says anything about the value or or kind of actually saying that the the inference is is somehow... I I don't wanna say more important because, of course, you have to train. But fundamentally, when you're training, you already have the corpus. You already have all of the training data. And so with all that data, you can massively parallelize the token inputs, all of the the sequences of words and sentences, paragraphs, etcetera, that are going into that. So that's something you can just crush through with a bunch of compute. But when you're inferring, because that's actually generative, you don't know what the token is five words down the line. And so you have to generate each and every one auto regressively or in order without, you know, foresight. And so that becomes a hugely memory bound problem that can't just be massively paralyzed like training.
00:07:03 · Harry Stebbings, Thomas Sohmers
[00:07:03] S1: So I totally get that in terms of the shift from compute bound to memory bound. Is that what people mean when they say about the memory wall with regards to what you're doing? [00:07:12] S0: Partially. I mean, the memory wall as a phrase has has been around for a long time before, you know, all of the hype around AI. And, really, what it's come down to is if you look at the past fifty, sixty years of computing, we've been able to, you know, have Moore's Law giving us more transistors per, you know, square millimeter of silicon, you know, consistently. And while that has been able to result in, you know, greater raw computes, flops, etcetera, the improvement of the memory technology is not kept up at the same rate. So roughly speaking, you know, between, you know, 2014, just very early innings of the new AI era till 2024, you had about a 120 x improvement in the FLOPs of GPUs. So a single NVIDIA GPU had about a 120 fold improvement in FLOPs, you know, and that's what enabled a whole lot of the the improvements over that decade. The improvement in memory bandwidth is only 17 x. So I would say, like, the the real embodiment of this is we had massive improvements on a per device basis of the flops and then a whole bunch of elements on the periphery of improving the, connectivity, etcetera, etcetera. But just like the ratio of the compute to memory bandwidth had this divergence. And so you had cases where if a problem was memory bound and you couldn't just scale the compute linearly with that, you were getting more and more memory bound as the decade progressed.