Invest Like the Best with Patrick O'Shaughnessy
01:00:54 · Neil Movva
[01:00:54] S2: I think compute scaling is actually very efficient. You give me more FLOPS and I will use more FLOPS. I would say we're actually fairly judicious already with our use of FLOPS. If you look at a modern MOE model, there are very few models that are more than 10% dense, meaning 10% of the possible number of experts you can activate are activated. And I think the frontier models are closer to like 1%. Fairly sparse already. I don't think that we're wasting too much on the MOE side. People have been working with MOEs for quite some time. They're pretty good at squeezing MOEs. Where we are not good is attention and it's use of memory specifically. The KV cache is quite uncompressed right now. I think if you look at the entropy in a KV cache, it's not earning its keep. Like we're storing many kilobytes of data in the KV cache per token. And that's probably off by an order of magnitude or two. I don't know the frontier labs do, but DeepSeq certainly publishes really interesting work to compress that further and further. And they're making good progress. And I think the fact that they're able to make an order of magnitude and progress here every year or so signals that there's a lot more room to go. If you zoom out further, I think that we actually don't marshal our compute effectively at all. We have all this compute in the world. Nvidia is pumping out 5,000,000 black wall chips this year. Where are they all going? Are they all being used at all all the time? I certainly doubt it. I think that at some level, we just need better orchestration of compute across the world. This is very difficult to do because a lot of the compute disappears into private pools of compute that will never see the light of day and those GPUs sit very sadly idle. It pains me physically to see that those GPUs are just silicon and power going into that and it's just sitting idle. And I wanna fix that. How we organize and orchestrate the world's compute as a shared resource and pack it more efficiently. I would estimate that we all make fun of x AI for having, you know, some challenges with total flop utilization on its clusters. The reality for the rest of the world is it's far worse. A ton of GPUs just sit in warehouses or sit in private pools allocated to a specific customer just don't
01:02:44 · Patrick O'Shaughnessy
[01:02:44] S0: get utilized. You're attacking the efficiency of that very directly? That's way more effective, yeah. What about fabs? What do you think is the future of fabs themselves? I think everyone is wondering, the memory companies, will TSMC, will Intel and others, how will they expand capacity basically? Will we do it here in The US? Riff on fabrication of chips themselves. Like, we could just snap our fingers and have a 100 times the chips in the stock today, we'd probably have way cheaper tokens. That seems like an important part of the universe to hear your view on.
01:03:18 · Neil Movva
[01:03:18] S2: Everything grows in balance with each other. If we snap our fingers and nibble all those things, you might fix a TSMC bottleneck, you're just gonna run into another bottleneck. You make 20% more chips, then you have another bottleneck immediately. I will say though, it is interesting what they consider to be a must deliver, like an invariant that their customers, me, are always gonna want versus what I think is like a more fluid relationship. I think that if the fab exposes more of their trade offs to me, I'm able to make more intelligent decisions about what I can do. One of the most interesting examples here is that any fab has a lot of spread in their worst chip that comes out of the production line and the best chip that comes out of the production line. There's a lot of variance in how chips are made. And the question is, if you have a company like TSMC, they work very, very hard to tighten what we call these process corners. We wanna keep the worst chip as close in characterization to the best chip. And they've got a great lens to make that possible. But that means that they are adding a lot of controls in the process that maybe I don't need. Maybe I'm actually willing to find a place for that worst chip. You don't need to tighten the process control as much, which takes more time and cost. Maybe I'm willing to take a lot more rejects. And I think for us, it's like a more holistic optimization around cost of the dies, supply of the dies, then the cost of power and places we can put them. My whole goal is to so dramatically expand the supply of power across The United States that I have a home for a lot of chips that otherwise would not have earned their place in a data center.
01:04:43 · Patrick O'Shaughnessy, Neil Movva
[01:04:43] S0: Can we talk about how you design the system of your own business? What lessons have you learned? You talked about some interesting NVIDIA lessons. Bring me into the culture and how you structure a team and a business where this is the North Star. [01:04:55] S2: There's a lot of in the limit thinking. We don't worry about the immediate nature of when we start working on a model, the efficiency is not gonna be very good. We think about, like, where we could end up in, a month, six months, or a year's time. We don't accept the state of the machines we work on as fixed. Even something like the black wall chip, if we think that there's some bottleneck that is holding us back from achieving this performance, it's very important to me that we understand and characterize that very well and write it down so we can both a, tell NVIDIA about it, who are friends, and also to basically keep this in mind for future chips that we buy. We want to learn things that are variant for us or the company long term and kind of fold that into future decisions that we make. We're very collaborative. I think one of the most important traits that we look for are people who either who are both good students and great teachers. A lot of our people on the team were TAs in college and loved the experience of sharing knowledge in this way. We do whiteboard sessions all the time. The collegial environment where everyone has something to teach and something to learn is extremely important for us.
01:04:55 · Neil Movva, Patrick O'Shaughnessy
[01:04:55] S2: There's a lot of in the limit thinking. We don't worry about the immediate nature of when we start working on a model, the efficiency is not gonna be very good. We think about, like, where we could end up in, a month, six months, or a year's time. We don't accept the state of the machines we work on as fixed. Even something like the black wall chip, if we think that there's some bottleneck that is holding us back from achieving this performance, it's very important to me that we understand and characterize that very well and write it down so we can both a, tell NVIDIA about it, who are friends, and also to basically keep this in mind for future chips that we buy. We want to learn things that are variant for us or the company long term and kind of fold that into future decisions that we make. We're very collaborative. I think one of the most important traits that we look for are people who either who are both good students and great teachers. A lot of our people on the team were TAs in college and loved the experience of sharing knowledge in this way. We do whiteboard sessions all the time. The collegial environment where everyone has something to teach and something to learn is extremely important for us. [01:05:56] S0: What are the attributes of people that you would wanna hire that you think will be resilient to the work environment three years from now when more stuff is handled by machines?
01:05:56 · Patrick O'Shaughnessy, Neil Movva
[01:05:56] S0: What are the attributes of people that you would wanna hire that you think will be resilient to the work environment three years from now when more stuff is handled by machines? [01:06:06] S2: Curiosity. A 100% curiosity. The one thing I cannot teach is love for performance. Love for digging into every microsecond the machine is working and understanding what's happening on the machine at that time. That to me is the most important trait for a performance engineer, is what I look for. I don't look for lots of AI experience. I don't look for CUDA experience at all. That's actually a huge red herring. I mean, CUDA as a concept or GPS as a concept have evolved so much in the last five years. There's no point asking for ten years of experience. I wanna teach that, but I cannot teach the love for performance engineering. That is what I seek. [01:06:39] S0: Can you give your assessment of the major labs one by one, but also then the relationship of closed source as a category to open source? What you think is happening and will happen?