Invest Like the Best with Patrick O'Shaughnessy
00:22:38 · Rob Wachen
Like, this is gonna change everything, and we clearly don't have the infrastructure to serve it. And there's very few things you can work on that can actually bring this technology at scale to the world faster. I mean, there's, like, plenty of people that are super smart working on models, and the fabs seem maybe unreachable to work on, but it seemed like the hardware was all designed before ChatGPT. Every GPU, every TPU, every AI chip that was serving these models were just fundamentally built before this and are retrofit to serve these modern models. There's gonna be an entire new wave of architectures that came out. And what a more exciting thing to work on than bringing this to everybody. Very different angle at the same time. I was running a startup incubator called Prod, which has incubated a bunch of different companies, some of the earliest ones being Cursor and AnySphere, which merged, and MerCore, and Etch went through it, a handful of others. At the time, as these models were getting smarter, it was, yeah, 2022, I was realizing all of these companies are spending all the money they raise on compute. And I had this realization as I was working on some of my own stuff that, like, oh my god, all the products I wanna build are gonna cost tens of millions of dollars a year in inference. Hey. This is not going to be tenable. Like, the cost structure of every software company and the COGS is not gonna be, like, zero anymore for an incremental user. It's gonna be, like, quite high and it's gonna be a function of inference. And then the optics of every business is gonna also be inference as people use more and more coding agents. So fundamentally, it seems like inference is gonna be really important, and it feels like we are on a decade march for inference to become, you know, the biggest market in the world. So when you think about that, ten years from now, there's gonna be these giant projects where everything in that data center fundamentally hasn't been designed today. We should go pick something and work on it. That's kind of how it got started.
00:24:14 · Patrick O'Shaughnessy, Gavin Uberti
[00:24:14] S0: Gavin, I'm really excited for you to go back about as far, probably early in high school, maybe even earlier, and tell your favorite hash marks on the timeline that ultimately led to your ambition to drop out of Harvard and start this company? [00:24:25] S1: My first job ever was at a company called Exnor, where I did kernels development. I was 17, and a 17 year old can't sign illegally binding contracts. So rather than go ahead and do a traditional CIA, they only had to sat me down and said, Gavin, don't share this information. And Exner was one of the only companies that saw, hey, maybe this is a good trade, and I had to work building colonels. And a number of other companies since, I talked about bought by Apple for 200,000,000, did the same thing at Octo, they got bought by Nvidia for hundreds of millions of dollars. But when you do this sort of kernels work, what you realize is that the math is relatively easy, but to get high speed decode, the thing that matters is data movement. Almost all the work that you do is optimizing how do you move data around a single chip or across multiple chips. And that's why we went ahead and built this cluster scale memory tech. You bring that interconnect time way, way lower, you can go do way more movement, and as a result, get a much faster time to generate each subsequent token and build these crazy things Rob's talking about for doing a year's worth of work in a month or more than that in the future.
00:24:25 · Gavin Uberti, Patrick O'Shaughnessy
[00:24:25] S1: My first job ever was at a company called Exnor, where I did kernels development. I was 17, and a 17 year old can't sign illegally binding contracts. So rather than go ahead and do a traditional CIA, they only had to sat me down and said, Gavin, don't share this information. And Exner was one of the only companies that saw, hey, maybe this is a good trade, and I had to work building colonels. And a number of other companies since, I talked about bought by Apple for 200,000,000, did the same thing at Octo, they got bought by Nvidia for hundreds of millions of dollars. But when you do this sort of kernels work, what you realize is that the math is relatively easy, but to get high speed decode, the thing that matters is data movement. Almost all the work that you do is optimizing how do you move data around a single chip or across multiple chips. And that's why we went ahead and built this cluster scale memory tech. You bring that interconnect time way, way lower, you can go do way more movement, and as a result, get a much faster time to generate each subsequent token and build these crazy things Rob's talking about for doing a year's worth of work in a month or more than that in the future. [00:25:29] S0: Can you talk about the competitive drive that's evident in some of the high school competitions that you participated in and won?
00:25:29 · Patrick O'Shaughnessy, Gavin Uberti
[00:25:29] S0: Can you talk about the competitive drive that's evident in some of the high school competitions that you participated in and won? [00:25:36] S1: We did a couple. For example, I was very active in the FTC robotics. I'd like you to have a very talented partner, Sanford, for a long time, be a part of a traditional school team, where it was about 20 guys, all working together as often typical of first tech challenge. And the goal is to go out and build a robot that scores the most points, and a bunch of other things too. And first, they put a lot of emphasis around collaborating with other teams, around trying to do really good documentation, around trying to go ahead and get others inspired to go do the same thing. Stanford and I decided, rather than go ahead and do it this way, we're gonna win. And we did nothing else beside the Boulder that scored the most points. [00:26:12] S0: As a two person team. As a two person team. Rather than a 20 person team.
00:26:12 · Patrick O'Shaughnessy, Gavin Uberti
[00:26:12] S0: As a two person team. As a two person team. Rather than a 20 person team. [00:26:15] S1: That we were much, much smaller than almost every other team in the competition. We figured that if we were gonna go specialize, if we were gonna go out and do this, win the damn games really well, we wouldn't need to go ahead and advance based on the quality of our documentation or of our outreach, we were just gonna go win. And so we did, we had branched off, built a two person team that built a robot and decided we were gonna go ahead and redesign it every three months, and we did. We actually had the world record for the highest score during this competition at one point. We were rated by OPR third in the world for software development, and, it was a damn good machine. [00:26:48] S0: What from that episode can I translate as an analogy onto how you built Etch the company? [00:26:53] S1: When thinking about how you want to go ahead and do a full RackScale product like this, there's a couple of key ideas. One of them is like velocity, velocity, velocity, that you win by shipping. You're not gonna go out and win by having the best outreach or the best communications, you win by building the best product. And similarly, we think we can go do it with a lot fewer folks, that if you're willing to go ahead and just focus on product, product, product and paralyze relentlessly, you don't need 20,000 people like the big companies have. You can do the best product in the world with far fewer people.
00:27:27 · Rob Wachen
[00:27:27] S3: Yeah. There's a saying of the best part is no part. I think for us, it's also the best vendor is no vendor. As much as possible, we wanna vertically integrate the entire product, both because we get more performance, but we can move way faster. So everything from the chips to the boards to the cold plates to the interconnects to even the production, We wanna do all of it as in house as possible. I think we're the only startup right now that's building its own rack as well as its own chips, and we did it all at the same time. A couple years ago was the last time we were public. At that point, we just started building our rack team, and we brought over Brian Loyler who built all of NVIDIA's HDX and DGX systems, which is, like, 80% of their revenue. And we said, we're gonna build the rack at the same time. We actually went through multiple iterations of the rack before the chips even came back. Before the chips came back, we made thermal chips that had the exact same hot spots as we expected our chips to have so we could build the cold plates, we could over pressurize them and blow them up. We haven't had a single leak since our chips came back with the cold plates because we already validated them. We have a factory in Taiwan, we have a few dozen people out there. We built a clone of a bunch of the test stations in our office. We have a two megawatt data center on this floor, and we did twenty four seven development cycles. People are doing day shifts and night shifts to actually get the hardware up and running as quickly as possible. But it's that extreme vertical integration and extreme parallelization of the schedule that lets you get products to market way faster.