The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:42:13 · Chase Lochmiller, Harry Stebbings
[00:42:13] S0: I feel... I actually don't feel like it's super unknown. I actually feel like, you know, we've had this breakthrough in overall model development cycles. You know, I think the scaling laws have continued to prove out. I think we've reached... You know, we keep moving the goalposts of what is what is artificial intelligence, what is AGI. You know, you think about the the Turing test. When I took my very first computer science class in high school, the Turing test was heralded as this as this critical notion of true machine intelligence, and it seemed like an impossible task at that point in time. We blew past the Turing test, and almost no one even celebrated. You think about, you know, the model capabilities you have today, and now, you know, we're solving Millennium Prize problems. Right? This was something, you know, when I was an undergrad at MIT, these Millennium Prize problems sort of came about, and it was like this, wow, these are these unsolved problems where there's a real financial bounty out there. Like, you can earn a million dollars if you can solve one of these problems. And they sat there for decades unsolved. Now we have AI solving these these these critical problems. [00:43:19] S1: I agree with you. My question to you is we see these swarms of rogue agents doing things that are nefarious. We see whole industries that are being put to question. My girlfriend's a lawyer. Six months ago, she didn't use the legora. Now she hasn't written a document in five months. There is just such uncertainty across everything. We have RSI that could
00:43:19 · Harry Stebbings, Chase Lochmiller
[00:43:19] S1: I agree with you. My question to you is we see these swarms of rogue agents doing things that are nefarious. We see whole industries that are being put to question. My girlfriend's a lawyer. Six months ago, she didn't use the legora. Now she hasn't written a document in five months. There is just such uncertainty across everything. We have RSI that could [00:43:36] S0: come and and change absolutely everything. So, sure, I get the comment that there's uncertainty around what the future of work will look like in certain ways, but I also feel like it's it's actually not uncertain that AI is gonna be a critical underlying engine for work and productivity that is gonna shift things quite a bit... Quite dramatically. And so it's more about that there is certainty that AI is establishing capabilities that are at human level or greater than human level in terms of overall productive capacity. So sure. Is that gonna change things? Yes. That is gonna change things. So so there's uncertainty in the change, but not... To me, it's certain that there will be change. Maybe that's a different way of putting it. [00:44:18] S1: Can I ask, we we mentioned the managed inference being a great part of your business? [00:44:22] S0: Mhmm. [00:44:22] S1: Gavin Baker, one of your investors, has said that the lowest cost producer of intelligence wins. When you think about that, like, what's the unit? Is it, like, dollars per token, per training run? What is that unit fee?
00:44:35 · Chase Lochmiller
[00:44:35] S0: Dollars per token is a is a good measure in terms of this, but it's it's not necessarily... You know, there is token efficiency, so, you know, just not all tokens are created equal. So it's not like every single token has the same, you know, unit of value, but I think it is a good indicator. I do think that, you know, there are different things that folks optimize for in terms of utilization, whether it's if throughput, so the amount of tokens per second you're able to generate for for for a very specific model use case, as well as time to first token, so latency, time to first token, time to last token, you know, how quickly you're able to actually serve this. And a lot of that comes down to it's a confluence of of different things. Right? So there's, you know, what is the cost that go into that? Obviously, by being vertically integrated and owning the infrastructure, you're able to stack a lot of margin across the process. So it gives you more flexibility in terms of how you actually, you know, bill for that. Additionally, I think utilization of the GPU itself is a critical aspect of being able to deliver tokens ultra efficiently and shine on metrics like latency and throughput. And so how does that actually manifest? Well, you know, the GPU is actually the most valuable thing in the entire data center, right? It is like the most expensive thing. So it's the thing you want to make sure you're keeping busy. If it's sitting idle, that's just money burning, right? If you think about that from that perspective, I think one of the key aspects is actually the management of memory and the management of the, you know, of the KV cache specifically. So the KV cache is the key value cache. It's really like, you know, you can kind of think about, you know, when tokens are fed into a a large neural network, you can compute the output tokens by basically running this feed forward process and running all these matrix multiplications that takes time. You may also already know the answer of that output token before you put the... So if you... If you've actually put put those same tokens in before, you actually know the answers and you can look them up in this look up table. The KBCash can get quite large and so it can expand well beyond the amount of, you know, memory you have on chip in the HBM.
00:46:39 · Chase Lochmiller, Harry Stebbings
And so actually being able to manage that both across multiple different GPUs, HBM layers, as well as system memory, DRAM, as as well as things like NVMe. So any any sort of, you know, solid state drive that you have on the system, and then sort of expanding that into the object storage layer, of your cluster. And being able to really, really efficiently move data around is actually a critical aspect to being able to serve inference and be able to keep those GPUs busy so that you're able to really shine on those metrics of latency and throughput. [00:47:08] S1: What does no one see about this layer of the value segment that you provide that everyone should see? What do we not know? [00:47:15] S0: Well, I think the big aspect is that not all inference providers are created equal. I mean, you have like great open source projects like SGLANG and VLLM. But, you know, there are ways to... You know, those are very generalized or generalizable. [00:47:31] S1: When you look at like an inference provider like Fireworks. [00:47:33] S0: Fireworks is great. They've done an incredible, incredibly great job. [00:47:36] S1: To what extent do they compete with you and cannibalize your managed inference business?
00:47:36 · Harry Stebbings, Chase Lochmiller
[00:47:36] S1: To what extent do they compete with you and cannibalize your managed inference business? [00:47:41] S0: Look, I think there are multiple layers across the stack that Crusoe competes with. What do they have that you don't by being specialized? I think there are certain workloads that, you know, Fireworks, you know, does an incredibly great job serving. I think, you know, the user experience, I think, is something that is unique across all of these different inference platforms that, you know, people may like more on on one than the other. There's a suite of different tools that, you know, people are... You know, there... There's a whole bunch of different features that people basically provide across, you [00:48:12] S1: know, the producers. You're, I wish I was them because then we could do this.
00:48:12 · Harry Stebbings, Chase Lochmiller
[00:48:12] S1: know, the producers. You're, I wish I was them because then we could do this. [00:48:17] S0: I sort of come back to this notion of, you know, the oil and gas analogy analogy where where the... If you look at the value chain, there are different places where you can compete and there are different places where you can collaborate. When Exxon is pumping wells at the wellhead, they may compete with another upstream oil and gas company to to, you know, most efficiently drill and operate wells. But they may sell to those same customers if they're building a midstream pipeline to transport oil from the wellhead to the refinery. They may provide capacity in that pipeline to other folks that they're competing with. And so, again, I I think it comes down to this perspective that, like, I just view there are so many more ways for us to work with folks across the AI infrastructure stack than compete with them. [00:49:01] S1: Can I just ask you, when you look at, like, open versus closed, given the purview that you have from the managed inference perspective, what do you see about distribution of dollars of tokens across open versus closed given that unique perspective you have?