411: Jeremy Burton, GM of Observability at Snowflake: The Data Layer Was Never a Commodity. It’s the Whole Game

00:31:30 · Jeremy BurtonPlay from here

The agent could suggest structures or entities extracted from that data, and then as we further ramp queries, it could optimize the query performance. So I think AI agents will essentially end up creating that graph. That graph then becomes the basis for other agents to drive the investigation flow. So, you know, you can start to see, yes, there's more telemetry flowing in the bottom. The good news is we don't have humans creating entities and structure, and we have agents doing that. Oh, and by the way, that structure they created is going to help other agents advance the troubleshooting flow more quickly. So, yes, there's going to be humans involved. There's there's unknown problems you've never seen before that, you know, will need some, you know, sort of human insight. But I think the goal here is that 90% plus of the legwork here is is really done by a collection of AI agents.

00:32:31 · Dan TurchinPlay from here

[00:32:31] S1: Makes sense. In a pre AI world, as much as it sounds like an insult, a lot of observability could be performed using heuristics. Back in the day, we didn't think of that as an insult. That was just, you know, it was a nice reliable way to, observe applications. Now as we introduce AI, kind of these probabilistic or nondeterministic outcomes, there's a different level of responsibility. It's it's like... Observe such Snowflake as a vendor. It's less about just verifying the integrity of the data or, you know, its state and more about taking responsibility for some of these actions, the decisions that are made. How do you think about the shifting role of what you do with respect to these, you know, probabilistic outcomes?

00:33:20 · Jeremy BurtonPlay from here

[00:33:20] S0: No. It's a it's a great question. I think for the last decade, the sort of unit of currency, the thing that we've been trying to troubleshoot really is a microservice. And a microservice is a lump of code that typically talks to another microservice that talks to a database. An agent I know we profess that this is something wildly different, but what's an agent? Well, it's kind of a lump of code that will talk to maybe another agent, it'll talk to a database, but it'll also talk to an LLM. And so that changes the picture quite a lot, because I always describe an LLM as like a new animal in the zoo. We know databases. We've had them for years. We know how to monitor them, we know how to investigate them. It's a well known problem domain. LLMs, we don't know. There isn't the level of observability, if you like, into the LLM that we would get from, say, like a database. Also, I think you touched on that a second ago a database gives you sort of empirical facts back. It's a deterministic system. You will get the same answer. Given the same question, you'll get the same answer every day. That is not true of an LLM. And so I think what becomes more and more critical as we try and troubleshoot these agentic systems, is a way to evaluate the LLM. There's gonna be certain bounds within which an answer is acceptable, and we have to constantly run these evals against those LLMs to see if the answer is still within the bounds of acceptability. And if it's not, that... Like, why is that? Like, you know, something obviously has changed. It could be something, you know, with the data source. It could be something with the way it's been prompted. You know, there's many possible reasons, but I think what what we would call LLM observability, in particular, evals, are just gonna be a critical part of, I guess, now we'd call it the the harness for the LLM. You know, what what what... What's one of the essential things that you need in order to run an LLM? It's gonna be in order to determine how the responses are changing over time. And by the way, that that could be down to a, like, a new a new model been released. It's kind of topical given Anthropix new release this week. That release changed dramatically, probably, the function of a lot of existing applications. And I think this is one of the big challenges with AI right now.

00:35:47 · Jeremy Burton, Dan TurchinPlay from here

Not only are we trying to go at a million miles an hour, we're trying to do it at a million miles an hour while the sand is shifting under our under our feet because the the, you know, the models themselves are still advancing quite dramatically. [00:36:03] S1: So there could be time when observability owns evals kind of [00:36:08] S0: Yes. I I I think it's a it's a it's a critical component for sure. [00:36:14] S1: Each week on this podcast, we talk about the future of work. And one of the things I'd like to get your perspective on as a leader is this tension that we all face right now. There's obviously a limitless amount of work to be done. Work... You know, the the amount of work to be done isn't going away. It's not shrinking. It's only growing. And yet there's this trope about, you know, reduce reduce the size of our organizations, you know, prevent bloat dot dot dot without editorializing. I would just love to get your perspective on, you know, as a leader, growing an organization, super ambitious. Does the team... How does the team look different in one, two, you know, maybe three years down the line?

00:36:58 · Jeremy BurtonPlay from here

[00:36:58] S0: Yeah. I think you... You've you've always got... With new technology, you've you've always gotta reexamine, like, what is work? What work for for for, like, an employee? If there are clearly things that can now be automated and and it's... And and that was a large part of someone's job, then I I guess, you know, you you you know, the role has changed. And there's no... To me, that's been the entire history of technology. I mean, a hundred years ago, you know, like, eighty percent of The US were were... Was agriculture and farmers, and then the tractor came along, and we're like, oh my god. We're we're doomed. And it turns out we were okay. You know? And I feel like, yeah, the the LLM is is sort of the the the modern day tractor. And right now, I think what's very hard for for for us humans to to to see past is what new work will be created. I... I'm generally an optimist on most things. I mean, look, I I I loved all the coverage recently around, the the almost trip to the moon or the trip around the moon. But it's like, okay. We did this, guys. Like, how many years ago? Two years after I was born, and I'm really old. And it's like, okay. What what problems can we now conceive because we're not dealing with minutiae? If you think about... I mean, supersonic flight travel. Right? We did that in the early seventies. Right? We don't have a supersonic jetliner today. So there's a part of me thinks like, okay. We we we can only conceive that a job that did x, y, and z today, well, x and y is taking care about AI, therefore, you know, we we don't need that job anymore. But if you look at the... If you take a step back and say, what does this technology enable? If we're operating upper level of abstraction, what problems can we now reason about that we could never reason about What things can we now achieve that we could never achieve before? What work can we now do that we could never do before? It's... I think it's impossible for us to to conceive of that until it sort of hits us between the eyes. You know? It was impossible to conceive of, like, the Hoover Dam beef... You know, when we just had shovels. You know? That that that... You know? So, you know, I I I feel like right now what's healthy as a leader is you have to examine the work that your team is doing. You have to examine whether that can be done by AI.

00:39:35 · Jeremy BurtonPlay from here

The reason is if you don't examine it, your competitor is examining it, and they're gonna be able to move faster and do it for less and blah blah blah than than than you. And then it's also the job of a leader to think about, okay. Given that we're moving five x as fast, what can we now add to the road map that we couldn't do before? You know? I I am still quite bullish on the case for software engineering and software engineers. I I just think we'll be able to take on wild projects that we couldn't ever conceive of. I do think education will have to change. Right? I I I feel like all of these great things that were proven to be not that useful in software engineering back when I did computer science are actually more relevant than ever. Things like architecture design. You know, it... Like, good practices around architecture and design will allow you to define a much better system for AI to go off and implement. But I think the last forty years, we've tended to dive straight into implementation, and generally hack away or rewrite, and probably not employ great software engineering practices. Right? The nice thing now is implementation is promising to be free. That doesn't mean to say software engineering is free. You know? You you can still mess it up with cogeneration.