The Pragmatic Engineer
01:01:26 · Titus Winters, Gergely Orosz
[01:01:26] S1: Yeah. I mean, we've taken it as axiomatic that it is hard, if not impossible, to measure productivity because what are the units even? I had, a while back, a sort of clever idea, and I use that as a loaded term. I had this idea that if we don't know what the units are, like economics has prior art for this, right, you have an index metric where you measure all of the things that might influence this, and you come up with a way to effectively basket those together. And then you have a unitless, but directionally useful and useful over time, number that you can sort of pin to. And my theory had been, so measure all of the things that we think are positive or negative indicators of something is going well. Right? Releases, test coverage, number of PRs, right, like everything. Right? Just cast a wide net. My initial formulation on this was, you know, the wisdom of the crowds idea. Right? Like, no individual can estimate how many jelly beans are in the jar, but you get a 100 people and you'll get real close. So we do a, like, am I hot or not? Of like, which would you prefer? Like, one release per week more or x tests. Right? And you run a bunch of iterations of that game with your, you know, relatively experienced engineers Yeah. To work out the linear trade offs between all of those factors. [01:03:01] S0: To figure out what matters more. Right.
01:03:01 · Gergely Orosz, Titus Winters
[01:03:01] S0: To figure out what matters more. Right. [01:03:03] S1: To figure out what that is. You probably wanna dedupe some things, because they're just gonna be the same stat in a different package. Right? But find the ones that have some nonlinearity, and then mush that all together. Right? Now you have something that sort of gives you an aggregate. Like, this is kind of our productivity number. And Adobe is doing something along these lines. Slightly smarter than my initial formulation. But like, then you can watch that track up or down and, you know, over time for a given population. What I've realized is that won't capture everything in this post AI boom. And the reason that I came to that was a paper that I put out, the develop deploy operate paper Mhmm. Recently, where one of the points in there is there's four forms of impact in tech. There's product success, the stuff everyone talks about and everyone else is better at than I am. Like, that's that's not my job. [01:04:06] S0: Will people buy a product or or use a product, whatever. Yeah. [01:04:08] S1: Net promoter score revenue, like all of the stuff. Right? There's compute efficiency. Right? Like, how much are you blowing on AWS? [01:04:16] S0: Oh, yeah. Right? Or if you I'll ship send shares [01:04:18] S1: and all that. [01:04:18] S0: All too well. Yeah. [01:04:19] S1: Right. You know? Or, like, even like the Android team has a trade off for if you can reduce the size of an application download by [01:04:28] S0: a It makes a difference for
01:04:28 · Gergely Orosz, Titus Winters
[01:04:28] S0: a It makes a difference for [01:04:31] S1: many metrics. Yes. Right. You know? And so like all of those things, like that is the form of impact. Then there's engineering productivity, or we phrased it as engineering capacity. Because if we save you, you know, 5% productivity gains, capacity gains, you don't get to spend that somewhere else. You just have more capacity on doing more engineering. [01:04:52] S0: Or having slack time for innovation or picking up when something happens, that kind of stuff.
01:04:52 · Gergely Orosz, Titus Winters
[01:04:52] S0: Or having slack time for innovation or picking up when something happens, that kind of stuff. [01:04:58] S1: Right. But they're not like dollar fungible. Exactly. You don't get money in a different account. And then the fourth form of impact, and this ties back to the AI thing. Fourth form of impact is changing the game. Right? Strategic innovation. Like, research makes impossible things possible. Oh, yeah. Security makes incredibly bad things impossible. Right? But if you change the nature of what we are doing, the cost structure, etcetera, then that's a different game. And the problem that I think that we see with AI, even with a clever you know, index metric approach to trying to capture the gestalt of productivity, if you're not looking for the value of a given change is substantially different, then there's no way you'll ever be able to capture that in the signal. Right? And we don't have anything that says the value of a change of this size has changed. So for instance, it is entirely possible at this point, and I hang out in security spaces and do a lot of low level c plus plus library stuff, you can write a JPEG parser in a couple weeks. Yep. Right? With whatever feat... Additional features you need. That would be insane to say that word... That sentence out loud three years ago.
01:06:18 · Gergely Orosz
[01:06:18] S0: Okay. So so what what what you're saying is what none of the measurements really capture is the possibility, the impact, the the effort it is to do a massive innovation, which right now with AI is happening and Right. And I'll I'll I'll give you also one example that I... And tell me if if this is what you mean with it is. There's a friend who works at a CTO at an engineering company about like a midsize engineering company. They're they're a niche industry. And, you know, like the the CTO, they they have been looking at like release frequency of the usual metrics, talk with their engineering team, are they happy, are they not? And of course, now they're adopting AI as well. They're seeing that, well, people are more efficient and just privately... See, this person was telling me, like, look, like, I kind of know that for a fact that we could probably do the same amount of work with, like, half the amount of people and then we could be cash flow positive. But, of course, there's a there's a morale thing. So he's, like, also, like, he likes the people. So it's like, I I don't... I'm not sure I want the emotional package on on me, but we can see some people are picking it up faster and some slower, and we kinda left it at that. A few weeks later, I'm talking with this person and says like, you wouldn't believe, but we're probably gonna be cash flow positive. I'm like, how? Like, did you go ahead with the cut? He's like, no. What happened is one of our engineers came to me and said like, we license this component, think of a library, like it's a binary Yeah. But that was written by a vendor in the space that has a high performance thing that they use in their in their software. It's it's a core part of it. And they have a revenue share where they they pay like, I don't know, like five, 10, 15% of their revenue for this component. It's so important. It's obviously licensed, closed source, but it's it's really important and it would take them years of a five person or 10 person team to write it and it might or might not work. So it's not worth it. Then Gergely came and said, like, what if what if we just gave the specs? Because they have all massive amounts of tests around this thing, a test harness. They gave it to AI. AI wrote a component to them which is faster and better what they have. No prior domain knowledge.
01:08:21 · Gergely Orosz
And at first they thought, well, maybe we'll open source it, but then they told the supplier and the supplier said, like, it's a huge business for them. They're paying them to To not. To not open source it and to give them the implementation, which they can relicense. And suddenly Yeah. The game was completely changed. Like like, if if this person would have gone ahead with, let's say, the layoffs, this engineer might have just got... And no one might have known that the game could be changed Yeah. Because of this AI makes things crazy things sometimes possible. Right. Exactly.