Venture with Grace
00:14:31 · Grace Gong, Vitaly Gordon
[00:14:31] S0: You guys have the newest report called The Speed Trap. Can you explain to, let's say, the non technical founder, what is a trap and who is caught in it? [00:14:43] S1: Yeah. So the speed trap, it just goes to say that as you kind of increase, you know, velocity and throughput for the system, you like, you kind of push the can down the road, right? It's not like like this whole thing does not come free because and again, to use, you know, one of the analogy we use about from the Industrial Revolution is like, you know, putting a steam engine in an, you know, arson, like, know, woods kind of crafting shop is just like not gonna make things, you know, run faster. It's just gonna, you know, cause a lot of chips to fly. And so the thing is, like, you have to also work on the rest on the rest of your pipeline. You have to think about the code reviews. You have to think about your CICD. You have to, like, again, Anthropic just, you know, recently released that they were, you know, pushing code, you know, so fast that, like, they broke GitHub pretty much. Right? And so they they have to kinda go and build another system. Like, you have to really think about the end to end process. And, like, it's great that we, you know, sold one bottleneck, but, you know, as ball... Bottlenecks don't disappear, they just move. Right? And so what we talk about in the report is where kind of the bottleneck shifted to and what can you do about it?
00:14:43 · Vitaly Gordon, Grace Gong
[00:14:43] S1: Yeah. So the speed trap, it just goes to say that as you kind of increase, you know, velocity and throughput for the system, you like, you kind of push the can down the road, right? It's not like like this whole thing does not come free because and again, to use, you know, one of the analogy we use about from the Industrial Revolution is like, you know, putting a steam engine in an, you know, arson, like, know, woods kind of crafting shop is just like not gonna make things, you know, run faster. It's just gonna, you know, cause a lot of chips to fly. And so the thing is, like, you have to also work on the rest on the rest of your pipeline. You have to think about the code reviews. You have to think about your CICD. You have to, like, again, Anthropic just, you know, recently released that they were, you know, pushing code, you know, so fast that, like, they broke GitHub pretty much. Right? And so they they have to kinda go and build another system. Like, you have to really think about the end to end process. And, like, it's great that we, you know, sold one bottleneck, but, you know, as ball... Bottlenecks don't disappear, they just move. Right? And so what we talk about in the report is where kind of the bottleneck shifted to and what can you do about it? [00:15:58] S0: The time in QA is up to 300%. And it is like, quote unquote, the worst number in your whole dataset. So what is the bottleneck that AI can't write code or human can verify it fast enough?
00:16:15 · Vitaly Gordon
[00:16:15] S1: Well, that's exactly the problem is because we sometimes we talk to these organizations and they're like pattering themselves on the back. Oh my god, 80% of my code is now written by AI. Right? And what we go is say, okay, you know, what do you do with the humans that are now not writing code? And they said, well, we kind of moved them to now review the code. And we're like, okay. And so, you know, if you kind of subscribe to the meme, but it's twice as hard to refode than to write code, then actually, you slow down your entire organization. So congrats. You know, like, you know, spending a lot on tokens and you're moving slower. Right? That's really what's happening in many organizations. And so, like, you have to rethink how you do QA. Right? You have to rethink on how you do basic quality assurance. And I think, like, there is... Again, you know, there are multiple ways to go about it. But, you know, over time, I think if you talk to a developer five years from now, they would probably kinda see it very, very odd that, you know, peak quality was to have another human to go, like, you know, line by line on the diff and, like, you try to, like, replay the code in their brains and saying, like, oh, that... This seems right. This seems like it would work. This seems like it would not work or something like that. It's... You know, our kind of quality assurance systems are probably like... Or, you know, kind of code reviews are so archaic. And so there's like, you can use AI in completely automated, you know, the code review as well. And then you can automate other tests and checks. Like, you can do so many things. But, like... But the but the thing is, you know, the fact that you could do them doesn't mean you do do them. And that's what we, again, we talk about in the report. Like, it's just... Like, there's no... There are no free meals. Like, you still need to go and do the work of, like, investing in your SDLC and, you know, to really gain the the right benefit from AI.
00:18:12 · Grace Gong, Vitaly Gordon
[00:18:12] S0: So you found that 76% more code changes merging with zero review. So no human or like no AI. So like nobody's kind of like reading the code anymore. So like what happened to a company if that like happens? [00:18:29] S1: Well, it kind of depends because, you know, unfortunately, the thing that was a little bit harder in some cases to do is to understand whether this, you know, zero human review means zero review or it means that all the review was automated. Right? And so in some cases, actually, the reviews were automated, but it's still... That is still unfortunately, the minority of the cases. In some cases, people just got tired of reviewing code because quite honestly, like, you know... And you see it in open source now that there is, like, so many open source projects that are just, like, refusing to accept contributions. Right? Because they just don't wanna review AI slob. Right? And so... And that's actually what happened. But, you know, unlike open source in many, especially large companies, you cannot really, like, not accept code. Like, you have to go through it. So people are just, like, not reviewing it. And then, you know, as we also show in the report, there is, like, downstream impact of, like, yeah. You know, just guess what? You will have more production issues.
00:18:29 · Vitaly Gordon, Grace Gong
[00:18:29] S1: Well, it kind of depends because, you know, unfortunately, the thing that was a little bit harder in some cases to do is to understand whether this, you know, zero human review means zero review or it means that all the review was automated. Right? And so in some cases, actually, the reviews were automated, but it's still... That is still unfortunately, the minority of the cases. In some cases, people just got tired of reviewing code because quite honestly, like, you know... And you see it in open source now that there is, like, so many open source projects that are just, like, refusing to accept contributions. Right? Because they just don't wanna review AI slob. Right? And so... And that's actually what happened. But, you know, unlike open source in many, especially large companies, you cannot really, like, not accept code. Like, you have to go through it. So people are just, like, not reviewing it. And then, you know, as we also show in the report, there is, like, downstream impact of, like, yeah. You know, just guess what? You will have more production issues. [00:19:27] S0: So speaking of, I guess, the open source, people doesn't take AI generated codes anymore. And so is all the code on the open source? I'm not saying every open source team is banning ad generated code entirely, there are some. Are human actually generating these code for the open source projects? Or can you maybe walk us through a real life example of maybe a customer or something when they hit like speech trap? Like how how do you kind of like fix it basically?
00:20:01 · Vitaly Gordon
[00:20:01] S1: Yeah. So so the fix of the ease, I would say, and again, this is kind of like tooting our own horn, but it's like... It starts with like with measuring what is important. Right? And, like, again, what we argue that should be measured is kind of like one is the end to end outcomes, which is, again, features, you know, in production for your customers. Now, you know, there are some people say, oh my god, like, you know, is that's not the wrong account? But it's like... Look. Assuming that you cannot tie directly the work of engineering to, like, you know, movements in revenue because, there's, like, a lag and there's, like, a lot of things. So I'm saying, like, you know, most cases when you hire an engineer is because you need them to build features. Right? That's kind of their job. And, you know, there's someone else's job to make sure that those features are, you know, kind of, like, translating into revenue. So so first of all, let's start by measuring, you know, what matters is like, okay, are my customers getting me more value? Right? That's kind of... That's the first thing. Now you can add to it. It's like, okay, but are we spending way more? Because it's easier, you know, it's easier to add more features by, like, doubling your engineering team. Like, probably that's not what you wanna do. You wanna, you know, double or triple your output without kind of massively your org. So you kind of wanna divide by something. Divide by, you know, kind of like how many engineers you have and how much they spend on AI. Right? That's kinda like you start. So that's probably kind of, you know, the starting point. Now, from that, just, you know, really start kind of like looking at the, you know, entire process and seeing where things get stuck. Right? Is like, is it the review? Is it, you know, the deployment? Is it the quality assurance? Is it like... And kind of things like that. And those are kind of the the things that we do. And it's kind of like... Again, that thing that, like, did not change materially since when we started the company. What changed materially is just like how bad those are now compared to like, what what used to be because before, like, you know, the word bottlenecks, but like, you can, you know, kinda easily fix them. Now you have to, like, rethink a lot of kind of prior assumptions on how to address it.