Vitaly Gordon, Faros AI Leaders on AI Engineering & GTM

00:16:15 · Vitaly GordonPlay from here

[00:16:15] S1: Well, that's exactly the problem is because we sometimes we talk to these organizations and they're like pattering themselves on the back. Oh my god, 80% of my code is now written by AI. Right? And what we go is say, okay, you know, what do you do with the humans that are now not writing code? And they said, well, we kind of moved them to now review the code. And we're like, okay. And so, you know, if you kind of subscribe to the meme, but it's twice as hard to refode than to write code, then actually, you slow down your entire organization. So congrats. You know, like, you know, spending a lot on tokens and you're moving slower. Right? That's really what's happening in many organizations. And so, like, you have to rethink how you do QA. Right? You have to rethink on how you do basic quality assurance. And I think, like, there is... Again, you know, there are multiple ways to go about it. But, you know, over time, I think if you talk to a developer five years from now, they would probably kinda see it very, very odd that, you know, peak quality was to have another human to go, like, you know, line by line on the diff and, like, you try to, like, replay the code in their brains and saying, like, oh, that... This seems right. This seems like it would work. This seems like it would not work or something like that. It's... You know, our kind of quality assurance systems are probably like... Or, you know, kind of code reviews are so archaic. And so there's like, you can use AI in completely automated, you know, the code review as well. And then you can automate other tests and checks. Like, you can do so many things. But, like... But the but the thing is, you know, the fact that you could do them doesn't mean you do do them. And that's what we, again, we talk about in the report. Like, it's just... Like, there's no... There are no free meals. Like, you still need to go and do the work of, like, investing in your SDLC and, you know, to really gain the the right benefit from AI.

00:18:12 · Grace Gong, Vitaly GordonPlay from here

[00:18:12] S0: So you found that 76% more code changes merging with zero review. So no human or like no AI. So like nobody's kind of like reading the code anymore. So like what happened to a company if that like happens? [00:18:29] S1: Well, it kind of depends because, you know, unfortunately, the thing that was a little bit harder in some cases to do is to understand whether this, you know, zero human review means zero review or it means that all the review was automated. Right? And so in some cases, actually, the reviews were automated, but it's still... That is still unfortunately, the minority of the cases. In some cases, people just got tired of reviewing code because quite honestly, like, you know... And you see it in open source now that there is, like, so many open source projects that are just, like, refusing to accept contributions. Right? Because they just don't wanna review AI slob. Right? And so... And that's actually what happened. But, you know, unlike open source in many, especially large companies, you cannot really, like, not accept code. Like, you have to go through it. So people are just, like, not reviewing it. And then, you know, as we also show in the report, there is, like, downstream impact of, like, yeah. You know, just guess what? You will have more production issues.

00:18:29 · Vitaly Gordon, Grace GongPlay from here

[00:18:29] S1: Well, it kind of depends because, you know, unfortunately, the thing that was a little bit harder in some cases to do is to understand whether this, you know, zero human review means zero review or it means that all the review was automated. Right? And so in some cases, actually, the reviews were automated, but it's still... That is still unfortunately, the minority of the cases. In some cases, people just got tired of reviewing code because quite honestly, like, you know... And you see it in open source now that there is, like, so many open source projects that are just, like, refusing to accept contributions. Right? Because they just don't wanna review AI slob. Right? And so... And that's actually what happened. But, you know, unlike open source in many, especially large companies, you cannot really, like, not accept code. Like, you have to go through it. So people are just, like, not reviewing it. And then, you know, as we also show in the report, there is, like, downstream impact of, like, yeah. You know, just guess what? You will have more production issues. [00:19:27] S0: So speaking of, I guess, the open source, people doesn't take AI generated codes anymore. And so is all the code on the open source? I'm not saying every open source team is banning ad generated code entirely, there are some. Are human actually generating these code for the open source projects? Or can you maybe walk us through a real life example of maybe a customer or something when they hit like speech trap? Like how how do you kind of like fix it basically?

00:20:01 · Vitaly GordonPlay from here

[00:20:01] S1: Yeah. So so the fix of the ease, I would say, and again, this is kind of like tooting our own horn, but it's like... It starts with like with measuring what is important. Right? And, like, again, what we argue that should be measured is kind of like one is the end to end outcomes, which is, again, features, you know, in production for your customers. Now, you know, there are some people say, oh my god, like, you know, is that's not the wrong account? But it's like... Look. Assuming that you cannot tie directly the work of engineering to, like, you know, movements in revenue because, there's, like, a lag and there's, like, a lot of things. So I'm saying, like, you know, most cases when you hire an engineer is because you need them to build features. Right? That's kind of their job. And, you know, there's someone else's job to make sure that those features are, you know, kind of, like, translating into revenue. So so first of all, let's start by measuring, you know, what matters is like, okay, are my customers getting me more value? Right? That's kind of... That's the first thing. Now you can add to it. It's like, okay, but are we spending way more? Because it's easier, you know, it's easier to add more features by, like, doubling your engineering team. Like, probably that's not what you wanna do. You wanna, you know, double or triple your output without kind of massively your org. So you kind of wanna divide by something. Divide by, you know, kind of like how many engineers you have and how much they spend on AI. Right? That's kinda like you start. So that's probably kind of, you know, the starting point. Now, from that, just, you know, really start kind of like looking at the, you know, entire process and seeing where things get stuck. Right? Is like, is it the review? Is it, you know, the deployment? Is it the quality assurance? Is it like... And kind of things like that. And those are kind of the the things that we do. And it's kind of like... Again, that thing that, like, did not change materially since when we started the company. What changed materially is just like how bad those are now compared to like, what what used to be because before, like, you know, the word bottlenecks, but like, you can, you know, kinda easily fix them. Now you have to, like, rethink a lot of kind of prior assumptions on how to address it.

00:22:18 · Grace GongPlay from here

[00:22:18] S0: Can we talk about like, I guess, like when you're working with the current companies, maybe you want to talk about like, would the enterprise be the target customer compared to like, more of like fast growing AI natives? Or like, how does like the customer aspect kind of like shape up? And when you guys are like working with the enterprise clients, like, what is like a common mistake that most people are have kind of like being just making the mistakes? And the other part is like when you're working with these companies, is there any kind of like checking framework? So for example, you just track like the core three things typically that's like the most amount that's expended by the company or how does that kind of work? So basically when you're actually tracking the token, where do you start?

00:23:10 · Vitaly GordonPlay from here

[00:23:10] S1: Yeah. So we are... Historically, we've been, you know, working with a lot of, I would say, yes, like, traditional enterprises. I would say now it's of the... There's more and more kind of AI native companies because, you know, they can have far fewer engineers, but still, like, their engineering output and kind of the bollocks might be actually, you know, larger than kind of some enterprises. So that's the... That's gonna other type of customers. So now what is the kind of, you know, the typical mistakes? And the typical mistakes is just like... I would probably say is like kind of looking for shortcuts. And the shortcuts are it's kind of like, let's look at, like, you know, before, in the... You know, when AWS just came out. Like, if you would go to, like, a large enterprise and just say, you know, give every developer a credit card, say, hey, you can now can spend, like, $20 on... A month or whatever, like, on AWS. That does not make you a cloud native company. Right? Just like the fact that... But somehow, we are doing similar things where we just, like, saying to every single engineer, oh, you know, here, you know, have get a pop out or cloud code or cursor or whatever, you know, tool you want, and now you're an AI native company. Like, again, unfortunately, that's just not how it works. Like, you cannot just basically buy your way into becoming AI native. Right? And so... And of the the way... And I shared at least kind of, you know, one metric that we go go through. So we start with, like, okay, measuring the outcomes, measuring the price per per outcome, but then start seeing, you know, distribution of word spend. So, like, I'll give you an example from, you know, one of our customers is what we found is their most kind of efficient engineers... And by the way, there could be, like, literally 10 times more efficient on a price per outcome. Could be also, like, their most expensive engineers, but could be also the... Their kinda cheapest engineers in terms of their salary and their AI spend because they're just so productive that, they actually, you know, aren't using AI in great manner. So one of the things that we also look at is on what can we learn from the best engineers who use AI and, like, what are they doing? We now...