Invest Like the Best with Patrick O'Shaughnessy
00:12:42 · Patrick O'Shaughnessy, Neil Movva
[00:12:42] S0: So if we dream about that future, we then become limited just by the questions that people can ask basically? [00:12:49] S2: Pretty much. The questions we can ask, the models are on the cusp of basically taking even a high level question and chasing it down. Every possible follow-up, you can have the model essentially take that on its own. And the question is, what is your token budget? And we will solve the token budget problem. What about non verifiable tasks? Those are basically the entire category of human taste into that category. We have not solved human taste yet and I don't know that it fundamentally can be. I'm excited to be surprised here but we are focused on very quantitative problems. We leave the quality of writing, we leave the beauty of art to people.
00:13:22 · Patrick O'Shaughnessy, Neil Movva
[00:13:22] S0: Vanta automates compliance so your team can spend less time on security reviews and more time getting customers. Vanta cuts audit prep by 82% and gives you instant up to date proof of trust without all the manual so you can close deals faster and with less friction. Customers report a 526 return on investment, and more than 16,000 companies use Vanta, including Ramp, Harvey, and Snowflake. Get started with $1,000 off vanta.com/invest. Ridgeline is the first end to end system of record with embedded AI for investment management firms running portfolio accounting, reconciliation, reporting, trading, and compliance on one unified platform. Firms are moving off legacy technology and onto Ridgeline because of how far ahead Ridgeline's AI features are compared to anything else in investment management software. I've been hearing from a lot of investment managers about AI, and they fall roughly into two camps, with some unsure where to even start and others convinced they can build their own order management system over a weekend. The reality is that running an investment firm will always require governance, controls, and a single source of truth for your data, and no amount of AI enthusiasm changes that requirement. Ridgeline is built on exactly that foundation, which is why I believe that the firms that come out ahead in the AI era will be the ones running on Ridgeline's unified platform. If you're serious about your firm's AI strategy, Ridgeline [00:14:45] S2: should
00:14:45 · Neil Movva, Patrick O'Shaughnessy
[00:14:45] S2: should [00:14:45] S0: be part of that conversation, and you can request a demo at ridgeline.ai. Alright. Now let's talk about the very clever stack of solutions that you hope to build ultimately to have this giant token factory, supplier of extremely low cost intelligence. I think you think about this in terms of software, hardware, and power. Talk through what your master plan is to approach this challenge that's so different from what others are thinking about doing. [00:15:09] S2: We always have to start with software. Where is the opportunity on today's data centers to improve efficiency? And the first thing we did was we tried to build the entire LLM software stack around peak GPU efficiency, meaning we're using NVIDIA GPUs. We wanted to squeeze out more tokens from the same chip than anyone else in the world. And that starts with the lowest level of programming, kernels. It's actually my background. I spent my whole professional life working on GPUs and kernels. NVIDIA was my first job while I was in college. I got to see how the tensor cores got to earn their right to be on the chip. [00:15:39] S0: What does that mean? Like, what is a tensor core? [00:15:40] S2: Tensor core is a specialized unit on the GPU that accelerates matrix multiplication. Simple as that. There's been a long history of how we evolve that tensor core over time that we'll get into. [00:15:49] S0: And why is matrix multiplication so important?
00:15:51 · Neil Movva
[00:15:51] S2: I cannot say that there is a divine truth of the inverse that explains why matrix multiplies seem to be the atomic unit of computation. But one way I've heard it described to me is, well, it's a really succinct way to mix two blocks of numbers together and have them interact in some interesting way. That's as much as I can say about it. It is really convenient that linear algebra turns out to be a very compact representation of arbitrary relationships and data. So Nvidia, great graphics company, had market share dominance in GPUs and gaming graphics for quite some time. And then starting in like the mid twenty tens, they started to actually start these like Skunkworks projects to make the graphics processor more suitable for machine learning tasks that they were tracking. I remember actually reading some of the lab notebooks of some of my managers when I was at Nvidia. They would visit these small ML conferences like ICML or NeurIPS at the time. They would just take note of these papers like, oh, this deep learning thing seems to be catching on. And what's really interesting is that these grad students are using gaming Nvidia GPUs in order to train their large models. We should double click on this and figure out what's going on here. By 2015, 2016, at least Jensen had the conviction to kind of double down on, hey, this usage of our chips is only gonna grow. Let's start allocating more and more precious silicon die area to this capability that seems to be emerging. Let's put the first version of tensor cores on the chip. So we're talking about taking this gaming chip, which is designed for painting pixel on the screen and adapting it to do mission multiplies. It was early. You would be competing against the graphics teams. Essentially, you ask for more silicon area at any chip company, there's always competition for that. It is something that the designers guard so carefully. You don't ever want to invest in the wrong technology because that's opportunity cost that you could have allocated to some other functionality. We fought tooth and nail and got just a tiny bit of diary, maybe like five, 10%, something like that, for the first generation of these chips to get some amount of acceleration for basic convolutions, which were the fundamental operation for computer vision models of the day.
00:17:43 · Neil Movva, Patrick O'Shaughnessy
And then we had a software team that was trying to squeeze all the performance we could out of the chip. And I think on that software team, which is where I worked, that's what actually taught me the most about the ethos that Nvidia has around this term called speed of light. They always chase the speed of light for any piece of hardware that they make. It is so ingrained in every engineer's mind that if the machine can do it, we're going to push the machine to the frontier until it does what we think... [00:18:08] S0: And the speed of light is the edge of what's possible. [00:18:10] S2: The speed of light is the edge of what's possible. Exactly. We think the chip can run at this frequency and produce this many multipliers per cycle, we're gonna get there. We're gonna break every bottleneck and get to that peak level of performance. To this day, I tell all my engineers, we're chasing a 100% speed of light. Don't I care about relative numbers versus the competition. I only care about absolute numbers. What are we able to do on the chip? [00:18:29] S0: How do we achieve that? Before we leave that chapter of your time at NVIDIA, anything else beyond that cultural touch point that changed the way you think about things or that stood out the most about how the business ran back then or its culture?
00:18:29 · Patrick O'Shaughnessy, Neil Movva
[00:18:29] S0: How do we achieve that? Before we leave that chapter of your time at NVIDIA, anything else beyond that cultural touch point that changed the way you think about things or that stood out the most about how the business ran back then or its culture? [00:18:40] S2: I have a ton of stories about NVIDIA. I could tell you a few of them. One of my favorites is that on the tenure side, a lot of people I worked with NVIDIA in 2015, 2016 are still there today. That company has incredible retention and these are the best engineers on on the silicon side at least I've worked with in my whole career. They're extremely, extremely motivated and passionate. They believed in parallel computing as a concept through its various incarnations and have loved seeing the chip evolve. This is their life's work and they're extremely competent in that direction. They're also a very frugal company. NVIDIA and all of Silicon Valley companies after 2008, they had some cutbacks and like perks. So no free lunch, for example. NVIDIA took it one step further. There was no free milk in the fridge. So if you wanted to drink coffee at NVIDIA and you wanted some milk, you actually had to chip in a dollar every month to the milk club and the milk club would stock Costco milk in the fridge. And I remember that distinctly. We don't do that at sale. It's a frugality that permeates [00:19:30] S0: the company. And so coming out of this time there, you get this experience of what it's like to develop more efficient usage of the underlying hardware through software. So link that to today's environment.