The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:11:32 · Naveen Rao
[00:11:32] S0: I think one thing I wish I'd known is the fact that we don't really see physical efficiency gains anymore. So you can separate out the way we build a computer and let's call it three levels. So the first level is the physical. So if I can send a bit from this point to that point and do it cheaper, you know, let's say I I represent zero as being here and one as being here, My phone has a certain weight. So there's energy required to move between these two states. If I make a lighter phone, it changes the amount of energy required. So that's just a pure physical thing. And Moore's law, as we generally refer to it, or Dennard scaling, you may have heard, where you make devices smaller, is really a physical thing. So you're making the physical layer more efficient and denser. So those innovations have largely stopped. And if we look at the last few generations of GPUs or other chips, you don't see efficiency gains much anymore. You see efficiency gains from the next layer up, which is kind of the computing architecture. So I would say most of the efficiency gains we've seen over the last several years or several generations have come from smaller data types, going from 32 bit to 16 bit to eight bit to four bit. So you get pretty good efficiency gains because you're moving fewer bits, you're computing with fewer bits. Then the next level, which is not really exploited that well, and we started this a little bit mosaic, is about how do I start enforcing locality and reduction of data movement at the model level? And five, six years ago, we didn't think about it like this because the model was not the primary driver of compute or even... Yeah. Maybe six years ago. It was still... Compute was a general purpose thing. I run websites and databases and all that kind of stuff. However, now if I look at the computer solely as a physical device designed to deliver intelligence, it changes the mindset a bit. Now I can co design all the way from the model through the architecture to the physical. And that's what I wish I'd known how much sort of alpha there is left in the top layers, not just the physical. We focus on the physical for a long time.
00:13:26 · Harry Stebbings, Naveen Rao
[00:13:26] S1: How does that design mindset change when moving across the stack with that transition from pre model focus to post model focus? [00:13:34] S0: Yeah. I mean, it it means we don't treat the model as sacrosanct, and we're energy first in our in our mindset, meaning that everything we do has to give us an energy advantage. It sounds like a simple thing, but it's not how anyone has designed these things in the past. It's always been about speed. And this actually has its roots that go all the way back to ENIAC, you know, in the nineteen forties because, you know, computers were designed to be faster than the alternative. That was always the case. And the ENIAC's case... By the way, actually, do you know why ENIAC was built? [00:14:02] S1: No.
00:14:02 · Harry Stebbings, Naveen Rao
[00:14:02] S1: No. [00:14:03] S0: It's an acronym. Let's say, Electrical Numerical Integrator and Calculator, I think is what it stands for. So this was a computer that was built by the US government in 1945. It was one of the first large scale digital computers built, and its purpose, like every other machine that's ever been built has a purpose, its purpose was to do artillery trajectory calculation and do it faster and more accurately than the alternative. The alternative at that time were humans who literally did the maths. So, you have to be faster. That was the key. Next generation comes out, it's gotta be faster than that. You know, GPUs came out, they had to be faster at rendering than a CPU. Each thing successively has tried to be faster than the previous thing. That has led to efficiency for several reasons. I mean, making things faster in the same power envelope is more efficient, but also Moore's law was chugging along and, you know, we went from vacuum tubes to transistors. Transistors kept getting smaller and smaller and smaller. And now we're in a place where that physical side of things is not really changing a whole lot. Transistors haven't gotten much better. However, we have a lot of room left on the model and architectural levels. [00:15:06] S1: Does that mean we're seeing a commoditization of models, or does that mean we're seeing actually a separation of model quality in those that are able to save energy and have efficiency gains versus those that aren't?
00:15:06 · Harry Stebbings, Naveen Rao
[00:15:06] S1: Does that mean we're seeing a commoditization of models, or does that mean we're seeing actually a separation of model quality in those that are able to save energy and have efficiency gains versus those that aren't? [00:15:18] S0: I think that's exactly right. So, you know, one thing I've been pretty good at is kind of picking where people are gonna be focusing in the next couple of years, and I think this is gonna be [00:15:26] S1: a big part of Naveen. And, really, like, credit to you. Did you walk through when you're picking a market and go, like, this is where I think the world's gonna go, and so I'm gonna position myself here? [00:15:36] S0: Yeah. Absolutely. And, you know, it it also comes down just to interest. You know, this is, for me, one of the most interesting problems there is to build a better computer. [00:15:44] S1: So you sat down here pre unconventional in the idea maze, and you were like, problem set that I'm gonna focus on is dot dot dot. What was that?
00:15:44 · Harry Stebbings, Naveen Rao
[00:15:44] S1: So you sat down here pre unconventional in the idea maze, and you were like, problem set that I'm gonna focus on is dot dot dot. What was that? [00:15:53] S0: Well, I kinda look at it like if I were to solve a problem, what is the effect? Like, what what what does it unlock? In fact, it's kind of like being a scientist. Like, being a good scientist is all about asking the right question that if you answer it, it unlocks many more questions. That's almost a hallmark of a good paper. So for me in this in this endeavor, it was clearly like, okay. Well, we're gonna run out of energy. That's clear. The paradigm we have hasn't really been focused on this. I think this is gonna be where the the the focus will be because we need to keep scaling. Right? Like, we need to deliver intelligence in many new form factors and greatly disrupt the cost in order to make it ubiquitous. So no one's thinking about this yet. I should start it. [00:16:32] S1: Can I ask, we were talking about, like, model efficiency and how it can deliver in terms of energy? Well, we see models getting more and more efficient today. How do we think about where we are on that model efficiency curve? Is there a lot of room to run there? Or are we kind of at the asymptotic point where you're like, we're eking out the final increments?
00:16:32 · Harry Stebbings, Naveen Rao
[00:16:32] S1: Can I ask, we were talking about, like, model efficiency and how it can deliver in terms of energy? Well, we see models getting more and more efficient today. How do we think about where we are on that model efficiency curve? Is there a lot of room to run there? Or are we kind of at the asymptotic point where you're like, we're eking out the final increments? [00:16:50] S0: No. No. No. We have a lot to go. And I think the reason is is we've not done this true co design between the model and the physical side of it. So you can come up with a concept and you may be able to get, say, let's call it five percent efficiency gain just from software on a standard GPU. That sounds great. However, you can only exploit whatever concept so much because the physical thing is built in a certain way. If you look at our brain, they coevolve. We coevolve the intelligence layer and the physical layer, and we're able to exploit things in the intelligence layer in the physical layer. So sparsity, for instance. Sparsity means, let's say, just not having connections between computing units. I can get some kind of gain out of using sparsity in a GPU, 100%. But when I have a standard off chip random access memory, the limit of those gains is not... Is capped. I can't go too far because that physical thing is not designed necessarily to exploit this kind of sparsity. So if I can build something that exploits it, I can get massive gains. That's really what the brain did. That's how it coevolved.