00:34:02 · S2, Rajkumar Sen, Siddhartha Ahluwalia
[00:34:02] S2: Sure. [00:34:03] S1: Uh, what is reinforcement learning? Like, you know, um uh how each front-end model is different. I think when I quit, I think GPT OSS and Llama, a couple of models were. When did you quit database? Uh February. One thing I took up as a challenge is to actually study the source code of Llama because some of the Llama code is in C my favorite language. So um I took a challenge upon myself to because you know data space is awesome, but then I want to see what's what is in AI. Like I always get technologically interested, not as a user. Find a lot of people are using MCPC, et cetera. My I'm like a hardcore systems guy. So my thing was like, okay, let's I have to go to the code and see you know what they're doing. At least I need to know fundamentally how the state transitions happen and all that. And I realized at the end, after doing my thing, that you know, it is not as hard as I thought it is, though now I'm so I had to dig a little bit on uh, you know, how do they do reasoning on what is the state of the art? And obviously, a lot of new models are coming, you know. Yeah, uh, very, very, very interesting. The AI space is changing. Um, it's I I feel like there are a lot of parallels, you know. Like uh when new databases were created, like Mongo was there, like it one new paradigm starts in, then you know, you have to even people in the database space had to learn how MongoDB does things and all. It's similar now, like the data people are having. [00:35:33] S0: You have to learn how databricks does things.
00:35:33 · Siddhartha Ahluwalia, Rajkumar Sen, S2
[00:35:33] S0: You have to learn how databricks does things. [00:35:34] S1: You have to, because that's the future, right? If you are like uh if you're not AI trained now, yeah, you're not gonna have a job. So, and being AI trained, I always tell people not only using like the harness stuff or like them, like if you can write a new application that does reasoning, you can spin up your own text-to-sQL thing, but also try to learn what the models are internally doing. Yeah, how does inferencing, like you know, I'm doing a lot of research on like my ex-boss Samir, he's joined Baseten as head of engineering. So I started learning like you know, why inferencing companies are existing, why Baseten exists, why Together AI exists, why fire you know fire exist, why not anthropic? Uh so you know how about what is your insight? [00:36:19] S0: Why they exist? [00:36:21] S1: See, I think one part is obviously the OSS movement is happening, and then I think um fundamentally not every company can solve all the problems. [00:36:31] S2: Yeah.
00:36:31 · S2, Rajkumar Sen
[00:36:31] S2: Yeah. [00:36:31] S1: So if you have a very high throughput, this is the classic, like I came back to the Arcion example, right? People are doing batch data movement, but then they want that continuous CDC, and I feel like training and inference is kind of the parallel thing, right? The inferencing, because the continuous process and live operational has to happen at super high throughput, super high uh scale, but also a lot of custom things have to be done per application. Like if you're running an application that does 700, 800 calls per second, that has some bespoke features that can only be done with a combination of base 10 or together with a chemie or one of the OSS deep sick models. Sure, entropy can solve that problem, but it's all about like you know, can you solve all the problems on Earth? [00:37:18] S2: Yeah. [00:37:19] S1: So there is that section of I'm sure people when we use Cloud or OpenAI, or obviously you're doing inferencing there. But if you want to run that kind of scale application where you need that at high throughput, you have to go to another vendor. Maybe one of these vendors will be bought by somebody at some point. But uh same thing when things get big, they will need scale, they will need uh you know, throughput, you will need one customization, which bigger companies may not want to do, but then that's where the startups thrive. And uh, I'm I'm super uh excited on the influencing companies because they feel like with the combination of open source and uh you know, then they can bring that value, right?
00:37:19 · Rajkumar Sen, Siddhartha Ahluwalia
[00:37:19] S1: So there is that section of I'm sure people when we use Cloud or OpenAI, or obviously you're doing inferencing there. But if you want to run that kind of scale application where you need that at high throughput, you have to go to another vendor. Maybe one of these vendors will be bought by somebody at some point. But uh same thing when things get big, they will need scale, they will need uh you know, throughput, you will need one customization, which bigger companies may not want to do, but then that's where the startups thrive. And uh, I'm I'm super uh excited on the influencing companies because they feel like with the combination of open source and uh you know, then they can bring that value, right? [00:38:06] S0: We have about two companies which are in inferencing. One is a company started by the founder of Qure.ai. Qure is one of the leading radiology AI companies, 40 million R valued at 400. So the the hospitals that were using Qure, uh they never relied on models, right? They said we want influencing in-house. The founder came out of it and just built inferencing in-house.
00:38:06 · Siddhartha Ahluwalia, Rajkumar Sen
[00:38:06] S0: We have about two companies which are in inferencing. One is a company started by the founder of Qure.ai. Qure is one of the leading radiology AI companies, 40 million R valued at 400. So the the hospitals that were using Qure, uh they never relied on models, right? They said we want influencing in-house. The founder came out of it and just built inferencing in-house. [00:38:30] S1: Yeah, that's the whole point, right, with the open source thing, right? Like if you're building IP, depending on your relationship with the uh with these big companies, you may want to trust them or not trust them, right? It's it's subjective, or maybe for certain applications you trust them. So if you want to build in-house, you've got to, you know, and then I think the inferencing space now is going to be like, you know, all day there's only one craigslist. Now we have Airbnb, DoorDash, everything, right? Now instead of horizontal inferencing, now you have inferencing for video, inferencing for you know image, like some something. [00:39:01] S0: Okay, but another company that was running inference for voice. So because everybody wants to specialize voice. [00:39:06] S1: Yeah, yeah. So I think uh as a category, it's very fascinating to me. Yeah. Uh, because as a systems guy, I feel like that's one space where a lot of what I learned to scale databases, all the techniques that were there to scale data management will apply. [00:39:24] S0: But can inference get bigger than cloud?
00:39:24 · Siddhartha Ahluwalia, Rajkumar Sen
[00:39:24] S0: But can inference get bigger than cloud? [00:39:27] S1: Can inference get I truly believe that can be true. Because if you think after five years, if every app is AI, it's only inferencing, right? That was the grand vision of Arcion. At some point, data will become so big that you cannot do batch data copy, you have to do differential data copy. It doesn't happen, yeah, it's happening. Uh now there are so many GDC companies including PATB, but the vision was that. Now think of like the same thing, like you can't you will have to train periodically training, but inferencing is similar, like the continuous thing. And if every app is shifting to that, it is going to be bigger. Uh it will be at very big. I don't know how you know, but um, and I feel like most of the innovation at least will be in inferencing side. [00:40:30] S0: Yeah, because cloud today combined AWS are at between 180 and 190 billion, uh Azure is at 140 billion, they run rates, uh, 140 billion run rate, and GCP is at 90 to 100 billion. So combined if you see 180 plus 140, the other 500 billion.