Invest Like the Best with Patrick O'Shaughnessy
00:51:30 · Rob Wachen
[00:51:30] S3: will humans use these tools. We sometimes run experiments internally and we had Codecs actually get GPTOSS running from scratch just based off of our docs completely by itself and it did it I think overnight. We think about game selection a lot, and what we mean by that is making sure we're investing our energy in the right bets. Because regardless of what you choose to work on, it will take tremendous effort. And one of the things that we started with was, you know, the decision explicitly not to build an arbitrary graph compiler, not to support arbitrary PyTorch, not to support arbitrary CUDA, not to support arbitrary ONNX graphs. But instead, we envisioned a world where there was going to be under a 100 models that actually mattered, and they were all going to look very similar from the underlying mathematical perspective, and that we were going to build primitives using physics that were going to accelerate these as much as humanly possible, and we were going to allow the most sophisticated customers to have direct access to the hardware and do whatever they want. And that has saved us a tremendous amount of time not having to build a compiler and that has allowed us to actually get much more performance. And funnily enough, when we started, a lot of people dismissed this idea and the only people that took us seriously were in high frequency trading. They all hate compilers too. They all write their own kernels. And we've had dozens of people from high frequency trading join the team because they saw this philosophy too.
00:52:46 · Patrick O'Shaughnessy
[00:52:46] S0: What are the limits to vertical integration? Like, how do you know where to draw the line? And I'm starting with this question to talk a bit about the broader market. The circumstances of the broader market are really interesting to me, where the vast majority of chips, of AI chips, get bought by a very small set of customers. Many of those customers are themselves trying to design their own AI chips. OpenAI announced TALAPENO. It seems like this very funny circumstance for like the most valuable thing in the world all kind of flows through a couple chip makers, a couple chip buyers. They all seem to be thinking about doing each other's job. And then you've got the circumstance where like, okay, then these things go into data center and you've got neo clouds and inference providers in this other part of the stack. You've got model builders and providers. Like, I could imagine a world where, because you have the best hardware, you design models and you build data centers. You leak outside of your current vertical. So, like, how do you think about where to draw the lines for the business?
00:53:41 · Rob Wachen
[00:53:41] S3: We have a saying that production is the product. Ultimately, matters here is we know inference is going to be the biggest market in the world. Whoever produces the most tokens is going to be the most valuable company in the world. So all the decisions we make is how do we get the most token capacity online as possible? And part of that is building a really good product that has way more throughput, that can run at way better latencies and so forth so we can per chip we make get way more tokens online. Another part of it is like not doing parts of the stack unless we absolutely have to to get to giant scale. So there are parts that we decided to do because it was absolutely required to get the scale, like building the rack instead of just building the chips, like doing a Centimeters model instead of doing a JDM model. But there are parts of it that are kind of noise to us right now. Like, we're not going and building our own data centers today. That doesn't actually help us get more capacity online. In general, our customers are actually making power and moving their clusters around to get our chips online because they're such high throughput. If there was a world where other things are constrained, we would totally go and integrate with them, but the reality is we're just purely focused on getting as many tokens online as possible.
00:54:44 · Gavin Uberti, Rob Wachen, Patrick O'Shaughnessy
[00:54:44] S1: I think this comes down to economies of scale again. A certain part to the stack and there are huge economies of scale and others there aren't. For example, on designing models, huge economies of scale there. For chip fabrication, same story. But for example, if you think about building some small metal part inside of that rack, there's not that same effect. We think the natural boundaries are on the chip side, on the bottom, and the model layer at the top, and we'll fill the whole gap between. [00:55:10] S3: A few weeks ago, there was a guy who was running a next generation AI chip for one of the frontier companies, and he's trying to recruit one of our architects. And, actually kinda did an UNO reverse card and started recruiting the person trying to recruit our guy. Then within a week, we hired him. And I was going on a walk as we were kind of finalizing the offer. I was like, you're leading this super important project. Why are you deciding to join? And his answer is super interesting, which was, it fundamentally is not existential for my company for this product to win. For Google with TPUs, the revenue comes from search [00:55:41] S0: Google won't fail if TPUs fail.
00:55:41 · Patrick O'Shaughnessy, Rob Wachen, Gavin Uberti
[00:55:41] S0: Google won't fail if TPUs fail. [00:55:43] S3: That's right. Meta won't fail if MTIA fails. Microsoft won't fail if Maya fails. And OpenAI won't fail if Jalapeno fails. Ultimately, this is our product. It is completely unsurprising that the best chip in the world is built by a company that only builds that chip. It's NVIDIA. Right? And for us, it is completely existential for us to get as much token capacity online as possible. It recruits a set of talent and recruits a support from suppliers and from customers that view it with the level of intensity that we do. Look at the raw FLOP stats. [00:56:12] S1: You compare any of the chips built by the labs or by the hyperscalers. The FLOP density for say FB8 times FB8 is lower than the Blackwell B300. And that makes sense because they don't have to go take the risk. They just have to go build a similar enough product and not pay the NVIDIA tax.
00:56:30 · Patrick O'Shaughnessy
[00:56:30] S0: Your finance team isn't losing money on big mistakes. It's leaking through a thousand tiny decisions nobody's watching. Ramp puts guardrails on spending before it happens. Real time limits, automatic rules, zero firefighting. Try it at ramp.com/invest. As your business grows, Vanta scales with you, automating compliance and giving you a single source of truth for security and risk. Learn more at vanta.com/invest. Every investment firm is unique and generic AI doesn't understand your process. Rogo does. It's an AI platform built specifically for Wall Street connected to your data, understanding your process, and producing real outputs. Check them out at rogo.ai/invest. The best AI and software companies from OpenAI to cursor to perplexity use WorkOS to become enterprise ready overnight, not in months. Visit workos.com to skip the unglamorous infrastructure work and focus on your product. Ridgeline is redefining asset management technology as a true partner, not just a software vendor. They've helped firms five x in scale, enabling faster growth, smarter operations, and a competitive edge. Visit ridgelineapps.com to see what they can unlock for your firm. As I think about you guys building the solution, the process of doing so is solving a sequence of really hard challenges. What has been the single episode that was the hardest to overcome? We were designing the chip.