Invest Like the Best with Patrick O'Shaughnessy
· Jesse Zhang
[00:00:00] S1: Yeah, they're very different. Maybe one framework to think about this is that at the end of the day, what AI agents are there for is to essentially replace human labor. That's why it's exciting, that's why everyone's so focused on it. One thing you can do then is you can just map out the spectrum of how much that human labor currently costs. So with customer service, it's generally outsourced already, especially for the tier one, tier two type inquiries that AI is now handling. It's generally not folks that are super highly paid, and then on the other end, it's engineers, which is the most highly paid people. The one way to think about it is that AI use cases will start eating the spectrum from both ends. And the reason why is that because engineers are the highest paid, they have the sophistication to really leverage it well. And AI just gives them so much leverage. I mean, there's other factors as well. It just happens that coding is tokenizable and the models are really good at it. That's one way to think about it. I don't know any company that's saying, hey, I would like to let go of a bunch of my engineers because now I have coding agents. There's infinite engineering work to do. So you're just augmenting them. On the other end, it is more of the replacement sense. You have a large BPO and that's costing you a ton of money. And it's also a really high operational thing to maintain because you have to hire people all the time. There's a ton of churn. You need to train them, you need to QA them, you need to make sure that nothing goes wrong. So AI is really valuable there as well, because the work is easier for the AI to do. It can fully replace. And I think that goes into the conversation of, I think it's a little bit overblown of AI is replacing jobs and so on. Even the BPOs we talk to, they're not really that concerned because what typically happens anyways is that there's already very high turnover in these BPOs. People are just hopping around, doing all sorts of different things, kind of just naturally let it decrease, and then they just go on to do sort of the next level of task that the AI can't do yet. Right. And so maybe that's data labeling or something. That's generally what we're seeing from the BPOs. But yeah, anyways, I think this spectrum is pretty real. And so then the question is what is the next thing that happens?
· Patrick O'Shaughnessy
[00:00:00] S0: So it's a really interesting conclusion, which is try to augment the very highest talent or replace the most replaceable end of the spectrum. That's a really interesting conclusion. I want to talk about what you've begun to learn about how to do that second thing. Well, so if others out there wanted to start a company or invest in a company that was doing sort of that end of the spectrum eating its way in as you described. What have you learned? Are the key things to the setup process with a given company to increase the likelihood that you can replace a lot of the low hanging fruit for types of customer service calls or types of what used to be human to human interaction and can now be handled by AI on one end of the spectrum.
· Jesse Zhang
[00:00:00] S1: Yeah, I would say the biggest learning we had is that oftentimes a long pole in the tent. And again, if by definition, if you can solve this well, it just makes things go a lot easier is aligning on what does good look like. And you would think that in our space it's pretty easy because it's, okay, maybe you just have a bunch of questions and answers and that's what good looks like. But unfortunately it's a lot more nuanced than that. What good looks like could be in the sense of tone and brand guidelines or how conversational you are. And even for the actual answers, we work with a lot of enterprises where obviously their scope is broad. And so one of the things you need to set up beforehand is what this could look like. So we have in our product essentially a testing or simulation suite of, hey, we're going to build out 10,000 tests and each one is going to be constantly running five times. And then you can get a sense of how well things are performing. And that's actually pretty difficult. And I do think that is broadly true for anyone that's trying to build in this style of company. If you're going to be replacing human labor, you need to know what good human labor is. So what we found is, okay, well can someone just tell us what are the answers to all these questions? And most people don't actually know because these are large, complex organizations. No one is the person where it's, hey, I know how to answer all these questions. And so you have to design a process where it's very easy to extract these answers from all the people that do know. So maybe it's all the CX leaders or people lead different areas of the product. And so you have to get them all together and get them to align on, okay, here's what the eval is. Essentially, if you can do that well, then it makes everything a lot easier because now you're just building, building, building. You have this quantifiable score that's, hey, here's how well the AI is performing and then once you're done building and the score is high, then you can go live.
· Patrick O'Shaughnessy, Jesse Zhang
[00:00:00] S0: Is it fair to think of this as a captive eval/continuous-improvement loop within an organization? Is that the simplified version? [00:00:00] S1: Yeah, yeah, yeah. That's an interesting way to think about it. And it is not RL model training, more agent improvement via evals/guardrails. It can just be reinforcement learning and making the agent improve. And that could be compiling more evals, that could be compiling just more guardrails guidelines around what it can and can't do. [00:00:00] S0: How fast does the spread happen? If I'm a customer and I've got the 500 person customer service call center or whatever, I'm actually curious. I don't know what the volumes are, how much call volume or interaction volume a center like that handles for a given company. But if I give you 5% of my workload and I'm satisfied with AI's performance, it performs well and there's not lots of problems, how fast are people willing to go from 5 to 10 to 15 to 20%?
· Patrick O'Shaughnessy, Jesse Zhang
[00:00:00] S0: How fast does the spread happen? If I'm a customer and I've got the 500 person customer service call center or whatever, I'm actually curious. I don't know what the volumes are, how much call volume or interaction volume a center like that handles for a given company. But if I give you 5% of my workload and I'm satisfied with AI's performance, it performs well and there's not lots of problems, how fast are people willing to go from 5 to 10 to 15 to 20%? [00:00:00] S1: Very fast. I would say even for large enterprises, within weeks, everyone wants to just go live to everything. But the reason why you stage it out is so you can make sure nothing's going wrong. And you can tell if something's going wrong almost immediately because you have these metrics. So even within a week you have 500 person organization. I would probably estimate mid to high six figures of conversations a year, maybe slightly higher. What you're doing there is making sure that things are going well. So within a week you can see, okay, what is the resolution rate? Is that what we expect? Okay, great. What is the customer satisfaction? People have those scores as well. And then they'll probably have some sort of accuracy metric based on human review. If those all check out, there's really no reason why you shouldn't roll it out. And again, the business case is so obvious there, right? So hey, we're both generating a ton of operational efficiency and our customers are happier. So yeah, let's just send it out to everything.
· Jesse Zhang, Patrick O'Shaughnessy
[00:00:00] S1: Very fast. I would say even for large enterprises, within weeks, everyone wants to just go live to everything. But the reason why you stage it out is so you can make sure nothing's going wrong. And you can tell if something's going wrong almost immediately because you have these metrics. So even within a week you have 500 person organization. I would probably estimate mid to high six figures of conversations a year, maybe slightly higher. What you're doing there is making sure that things are going well. So within a week you can see, okay, what is the resolution rate? Is that what we expect? Okay, great. What is the customer satisfaction? People have those scores as well. And then they'll probably have some sort of accuracy metric based on human review. If those all check out, there's really no reason why you shouldn't roll it out. And again, the business case is so obvious there, right? So hey, we're both generating a ton of operational efficiency and our customers are happier. So yeah, let's just send it out to everything. [00:00:00] S0: What goes most wrong when something bad happens? I'm sure this is happening less and less as the product's gotten better. But even in the early days, what sort of thing would go wrong in one of the customer to AI interactions?