00:27:38 · Thariq Shihipar
[00:27:38] S0: Yeah. I mean, so on the first part, I think harness engineering overall is an art. You know I mean? Like, I think evals are a little bit more of a science because, like, you can, you know, hill climb against them. I I do think, like, designing evals is very, very hard. Like, I don't know if you guys saw the, like, Japanese guy who's, like, annotated, like, millions of images with bounding boxes or something, you know. It's gone kind of viral on Twitter, but it's like this guy who spent three years just like going through millions of images and drawing very specific bounding boxes around like the eyes of people and mouths and like the skeletons and stuff. And his like setup now like outperforms every other like, you know, skeleton recognition and face recognition thing. But he's just like, it's mind numbing work. You know what I mean? And it's even hard to like outsource because, you know, like it's very high skill too. You know? And so I I think evals are one of these things where, yeah, it's actually almost hard to buy them even. Right? Because like you need like very high skill people to do very honestly boring work, know? And so, yeah, you can't have an eval for everything. And and for example, like let's say that someone... People are always like, okay, Claude Code feels dumber. You know what I mean? Like, in order for us to like... If we run Claude Code against SWE-bench or SWE-bench Pro or whatever, that will measure like a software engineering performance. But like maybe what people are saying is that like sometimes Claude stops early. You know what I mean? When I think it should continue. Then we need a stop early eval. Right? But how would we know we need that eval until we see the like the the problem, you know? And so I I think there's like this... Like you can't have an eval for any possible problem that Claude could run into. You know, it's it's hard, I I think. And evals are very, very valuable. I think I think, like, this is one of those problems where I sort of point people at sometimes when they're like, hey, like, I want to get into AI. I want to do something. And I'm like, okay, great. Choose something low status and not like sexy or interesting. And like, evals are like easily it, you know? And then on your second point about stuff that Anthropic does, obviously, like Evan said, like, the talent is very high.
00:29:48 · Thariq Shihipar
I think a particular thing I've appreciated about like, Thariq's approach is that he... I think sometimes you can say reasonable things about trade offs. You know, you can be like, okay, like, here, pick two, like good, fast, cheap, you know, or something. Right? And it's like, that makes sense. Right? Like, it's very reasonable that you would have to pick two out of these three things. But then it's like, well, no, what if we did all of them actually? You know what I mean? And like, I think sometimes you can sort of have this like reasonable product brain where you're like, hey, look, let's just prioritize these things and let's just like do the highly prioritized things. But like, maybe you're wrong. Like, you know, maybe you can do all these things. Right? And I I think that like forcing reality to show you, like, oh, no. These are the real trade offs and how things trade off against each other is sometimes unintuitive. And it's better to be like, maybe like somewhat unreasonable and just be like, hey. Like, yeah, we can do... Like, we can build a safe AI that also makes a lot of revenue and, you know, like brings, you know, a lot of, like, value to the world, and we can race to the top. So, yeah, I think that's something I've really appreciated. Let's spend a little more
00:30:59 · Aditya Agarwal
[00:30:59] S1: time here because I think this is one of the topics we wanna talk about was like how AI is changing org design. And I think examining how it's changed the Claude Code org is interesting as maybe as a suggestion for how many of people's companies will eventually change. Like, if you compare how you guys build a year ago to now with the new capabilities, are there specific things that come to mind in terms of, like, the ways you build feel very different to a year ago or maybe the, like, past lives building stuff that you would highlight as, like, this is probably the new way given that we have these new capabilities?
00:31:37 · Thariq Shihipar
[00:31:37] S0: Yeah. I I mean, I I do wanna say that I I don't think there's one pattern for org management, and it's not even clear to me that the Claude Code team should be the pattern because we are very, like, dog food oriented. Right? Like, we get... Like, we're literally building the product to build the product. Right? And so that has like a different feedback loop than maybe other organizations might. So, I'm hesitant to say that the Claude Code team is a model for, like, you know, how other organizations should work. But I think that like you can sort of think through some like first practices of like, how do you turn productivity into more, honestly, revenue, you know, is like ultimately the goal. Right? And I definitely talk to customers who I'm like, if you doubled your PRs, you probably wouldn't even change your revenue at all. You know, like you just are not able to turn that into increased revenue because there's so much of org design is, for example, like buy in, you know, and deciding what to do or like do people even want this as the market saturated or not or something. Right? And like, you know, most things fail. And so like, you don't actually make more money if you fail faster necessarily, you know, as long as... Unless you find the thing that that really works. So I I think that the prompt I... You use is like, okay, let's say that you're running a car dealership in, like, 1995 and email comes out. One thing you can do is like you can give your receptionist and your team email and they can communicate a little bit faster and they're gonna be more productive. But I don't know if you'll sell more cars, or you could like let people order cars via email, you know, and like... And via the web. Right? And hopefully that will like drive more revenue. And so I think trying to find that analogy for your specific problem, I think is one way to think about it.
00:33:19 · Aditya Agarwal, Thariq Shihipar
[00:33:19] S1: Are there specific things that are super extreme or heretical that maybe have become normalized to you? But if, like, somebody were to drop into the Claude Code team as, like, member technical staff, they'd be like, this is totally different. [00:33:33] S0: Probably some stuff I can't talk about. Yeah. Yeah. Yeah. I think I think one [00:33:37] S1: thing I've observed is because it's so cheap to create. I've been trying to wrap my head around what is... How do people prioritize now? And how do you prioritize work? Because you can kinda just spin up everything in parallel. And I think there's just weird tension with I'm not really sure... Kind of old ways in which you might like t shirt size something. Like, why even bother with that? And I just curious like how that's changed internally. [00:34:01] S0: Yeah. I mean, I I think that it still takes a lot of tenacity to make something, you know, and I I don't think that amount has decreased almost at all, you know. And I I think that when I see things that work at Anthropic, like Claude design, I was... As soon as I met Nate, I'm like, dude, this guy has it. You know what I mean? And I'm like, he will like, he really wants this to happen, and he will make it happen. You know? And I think that like, yeah, like it's cheaper to make things, but now you have to get buy in, you have to believe in it. Right? And so I think that like, it should increase your expiration phase, but you're the one who has to decide, hey, I think this is it. You know, like, is the direction. So... Yeah. One additional question on Oregon,
00:34:01 · Thariq Shihipar, Aditya Agarwal
[00:34:01] S0: Yeah. I mean, I I think that it still takes a lot of tenacity to make something, you know, and I I don't think that amount has decreased almost at all, you know. And I I think that when I see things that work at Anthropic, like Claude design, I was... As soon as I met Nate, I'm like, dude, this guy has it. You know what I mean? And I'm like, he will like, he really wants this to happen, and he will make it happen. You know? And I think that like, yeah, like it's cheaper to make things, but now you have to get buy in, you have to believe in it. Right? And so I think that like, it should increase your expiration phase, but you're the one who has to decide, hey, I think this is it. You know, like, is the direction. So... Yeah. One additional question on Oregon, [00:34:41] S1: and I don't know who's next with a mic. You can pipe in. Hiring is probably pretty different now in sort of what you look for. Are there specific questions or exercises that, like, are net new to Anthropic that probably other people should adopt when we're looking to hire really talented engineers?