01:39:53 路 John Coogan, Matthew Prince
[01:39:53] S1: I wanna talk about the inference stack, I guess. Like, we're seeing two things play out, sort of a bifurcation, like a WWDC, Apple's launching on device inference that's it's not going to be frontier, but it's going to be usable for sure on your phone, on a computer. They can obviously go to the private cloud as well. And then you have like the new NVL 72 models. There's stuff in between, a Cerebras chip, fast but expensive. Then there's everything from, oh, it runs on commodity hardware, but it's still pretty big. You need a real desktop for it. And I'm wondering about how you see Cloudflare fitting into that. It would be a logical extension to Cloudflare workers to have inference on the edge, inference in different places. Like, where do you see yourself offering inference, if at all? [01:40:47] S13: Yeah. Well, we we we it's it's actually kind of funny. Yeah. Back in 2022, we we issued a a press release that there was a graphics chips company that we were gonna partner with in order to put GPUs at the edge edge of our network. [01:41:04] S1: Yeah. [01:41:05] S13: And that graphics chip company [01:41:06] S1: turned Little into bit of Nvidia. Nvidia. Yeah. [01:41:08] S13: And and and what was what was amazing was at the time, it was just crickets, like nobody cared. [01:41:14] S1: Sure.
01:41:14 路 John Coogan, Matthew Prince
[01:41:14] S1: Sure. [01:41:14] S13: And but but we we had we had and and so we we sort of did a little of it, but we hadn't really Yeah. Rolled it out broadly. And and what was funny was and then you fast forward two years later, and all of a sudden everyone cared. And so we we basically just reissued the exact same press release. [01:41:29] S1: Mhmm.
01:41:29 路 Matthew Prince
[01:41:29] S13: And and and now, you know, that's become that's become a pretty pretty important part of our our our business. And so today, you can run inference at the edge of Cloudflare. Yep. And because of the fact that we're, you know, in over 300 cities worldwide, you know, we're within within milliseconds of the vast majority of the world's population, we become a very natural place for inference to happen. I my my working assumption has always been that about 50% of the inference that happens will be on device, whether that's your your phone or your or your laptop or or whatever. But then there needs to be some standard protocol where your phone or your laptop or whatever that local thing is can hand those either long running tasks or larger tasks off next to the network, and so that would be to us. And then if if for some reason you need something that's more than that, then it could handle it back to, you know, some centralized data center with, you know, with with more more capacity than than we may have. And I think that that that's what that's what we're increasingly seeing. I think my my assumption was a little bit probably off. I think actually, I think less is gonna happen on device today because I think more and more of the tasks are going to be these long running Yeah. Task where it's not gonna be just, you know, what what's, you know, what's the, you know, what's the temperature in in New York today. Yeah. Instead, it's gonna be something like, help help me plan a a vacation, go you know, take into account all of these different things, shop for, you know, different hotels in different places, plan all the plan all the, the the travel between between the different locations. Here are the criteria I have. And that might be something that takes, you know, maybe, you know, certainly not not gonna be seconds. It may might be, you know, minutes or hours or days in some cases to run-in that case.
01:43:11 路 Jordi Hays, John Coogan, Matthew Prince, Trent Simonian
[01:43:11] S0: I feel like you're gonna have long running agents just for Park City snow forecasting and reporting, tracking like individual I already do. Individual runs, you know. Maybe a network of maybe a network of drones that are identifying what's tracked out [01:43:26] S1: Satellite photos, planet lapsed. [01:43:28] S0: Optimizing your routes on the mountain. [01:43:30] S1: A lot of stuff. [01:43:30] S13: Exactly that's that's that's exactly right. Hopefully hopefully it snows this year unlike last year's. [01:43:35] S0: Yeah. So [01:43:36] S1: what are the decisions when you're doing inference on the edge? Because it's sort of a hard pitch to say, I'm going to shave off three hundred milliseconds, six hundred milliseconds, when you're talking about a twenty minute workflow or even a ten second workflow, you're getting me a 3% boost. Is that gonna make the difference? But are you optimistic that there will be sort of like a in between state where there will be inference that happens and needs to happen within the request loop under a second and then the the latency matters? [01:44:12] S13: Yeah. Well, I think I think there are there are sort of three different [01:44:15] S12: Yeah.
01:44:15 路 Trent Simonian, Matthew Prince, John Coogan, Farza Majeed
[01:44:15] S12: Yeah. [01:44:16] S13: Buckets why why people are choosing to run inference tasks on on Cloudflare. I think I think the first bucket is exactly what you said. It's just it's latency. And so especially for those things that have human and computer interaction where where, you know, any delay feels like it's it's it it hurts you that people are are trying to make sure that they can get the best best performance possible. [01:44:36] S1: For voice models, I mean, it feels like that has to yeah. Voice models have gotta be local. Yeah. I love that. [01:44:42] S6: Yeah. And so and
01:44:42 路 Farza Majeed, Matthew Prince
[01:44:42] S6: Yeah. And so and [01:44:42] S13: so that's and so, you know, again, there there's a case for that. I think the second case is that a lot of times for either privacy or regulatory reasons, people wanna keep things as close to where they physically are Sure. As possible. And so, you you know, I think that a lot of especially in Europe and otherwise, of places think that, you know, they made a mistake with the Internet kind of one point o and two point o of having everything go back to Ashburn, Virginia, everything go back to The United States. And so I think that there's there's a real kind of sovereign desire to keep things local, and that's important. And I think the third thing is we we actually can be oftentimes much more cost effective because of the fact that, you know, we we are this we are a network and and because of how we're interconnected with networks around the world, bandwidth for us is effectively free. And so it's very easy for us to get things onto our network. And because of where we deploy, systems, we often are in places where we don't have to pay for the space or the power, which allows us to then pass those savings on to customers. So it can be significantly more cost effective to use inference with us than it can be in some other other places. That's somewhat counterintuitive, but but but it's sort of the nature of what we've what we've built and kind of the secret to what's allowed Cloudflare to to to to deliver as much as we have over time.