Invest Like the Best with Patrick O'Shaughnessy
· Patrick O'Shaughnessy
[00:00:00] S0: So it's a really interesting conclusion, which is try to augment the very highest talent or replace the most replaceable end of the spectrum. That's a really interesting conclusion. I want to talk about what you've begun to learn about how to do that second thing. Well, so if others out there wanted to start a company or invest in a company that was doing sort of that end of the spectrum eating its way in as you described. What have you learned? Are the key things to the setup process with a given company to increase the likelihood that you can replace a lot of the low hanging fruit for types of customer service calls or types of what used to be human to human interaction and can now be handled by AI on one end of the spectrum.
· Jesse Zhang
[00:00:00] S1: Yeah, I would say the biggest learning we had is that oftentimes a long pole in the tent. And again, if by definition, if you can solve this well, it just makes things go a lot easier is aligning on what does good look like. And you would think that in our space it's pretty easy because it's, okay, maybe you just have a bunch of questions and answers and that's what good looks like. But unfortunately it's a lot more nuanced than that. What good looks like could be in the sense of tone and brand guidelines or how conversational you are. And even for the actual answers, we work with a lot of enterprises where obviously their scope is broad. And so one of the things you need to set up beforehand is what this could look like. So we have in our product essentially a testing or simulation suite of, hey, we're going to build out 10,000 tests and each one is going to be constantly running five times. And then you can get a sense of how well things are performing. And that's actually pretty difficult. And I do think that is broadly true for anyone that's trying to build in this style of company. If you're going to be replacing human labor, you need to know what good human labor is. So what we found is, okay, well can someone just tell us what are the answers to all these questions? And most people don't actually know because these are large, complex organizations. No one is the person where it's, hey, I know how to answer all these questions. And so you have to design a process where it's very easy to extract these answers from all the people that do know. So maybe it's all the CX leaders or people lead different areas of the product. And so you have to get them all together and get them to align on, okay, here's what the eval is. Essentially, if you can do that well, then it makes everything a lot easier because now you're just building, building, building. You have this quantifiable score that's, hey, here's how well the AI is performing and then once you're done building and the score is high, then you can go live.
· Patrick O'Shaughnessy, Jesse Zhang
[00:00:00] S0: Is it fair to think of this as a captive eval/continuous-improvement loop within an organization? Is that the simplified version? [00:00:00] S1: Yeah, yeah, yeah. That's an interesting way to think about it. And it is not RL model training, more agent improvement via evals/guardrails. It can just be reinforcement learning and making the agent improve. And that could be compiling more evals, that could be compiling just more guardrails guidelines around what it can and can't do. [00:00:00] S0: How fast does the spread happen? If I'm a customer and I've got the 500 person customer service call center or whatever, I'm actually curious. I don't know what the volumes are, how much call volume or interaction volume a center like that handles for a given company. But if I give you 5% of my workload and I'm satisfied with AI's performance, it performs well and there's not lots of problems, how fast are people willing to go from 5 to 10 to 15 to 20%?
· Patrick O'Shaughnessy, Jesse Zhang
[00:00:00] S0: How fast does the spread happen? If I'm a customer and I've got the 500 person customer service call center or whatever, I'm actually curious. I don't know what the volumes are, how much call volume or interaction volume a center like that handles for a given company. But if I give you 5% of my workload and I'm satisfied with AI's performance, it performs well and there's not lots of problems, how fast are people willing to go from 5 to 10 to 15 to 20%? [00:00:00] S1: Very fast. I would say even for large enterprises, within weeks, everyone wants to just go live to everything. But the reason why you stage it out is so you can make sure nothing's going wrong. And you can tell if something's going wrong almost immediately because you have these metrics. So even within a week you have 500 person organization. I would probably estimate mid to high six figures of conversations a year, maybe slightly higher. What you're doing there is making sure that things are going well. So within a week you can see, okay, what is the resolution rate? Is that what we expect? Okay, great. What is the customer satisfaction? People have those scores as well. And then they'll probably have some sort of accuracy metric based on human review. If those all check out, there's really no reason why you shouldn't roll it out. And again, the business case is so obvious there, right? So hey, we're both generating a ton of operational efficiency and our customers are happier. So yeah, let's just send it out to everything.
· Jesse Zhang, Patrick O'Shaughnessy
[00:00:00] S1: Very fast. I would say even for large enterprises, within weeks, everyone wants to just go live to everything. But the reason why you stage it out is so you can make sure nothing's going wrong. And you can tell if something's going wrong almost immediately because you have these metrics. So even within a week you have 500 person organization. I would probably estimate mid to high six figures of conversations a year, maybe slightly higher. What you're doing there is making sure that things are going well. So within a week you can see, okay, what is the resolution rate? Is that what we expect? Okay, great. What is the customer satisfaction? People have those scores as well. And then they'll probably have some sort of accuracy metric based on human review. If those all check out, there's really no reason why you shouldn't roll it out. And again, the business case is so obvious there, right? So hey, we're both generating a ton of operational efficiency and our customers are happier. So yeah, let's just send it out to everything. [00:00:00] S0: What goes most wrong when something bad happens? I'm sure this is happening less and less as the product's gotten better. But even in the early days, what sort of thing would go wrong in one of the customer to AI interactions?
· Jesse Zhang
[00:00:00] S1: It ranges all sorts of different things. Sometimes it's obviously both sides, us and the customer. We take everything very seriously. But there's just a lot of things you wouldn't expect in the early days, we have a customer that is essentially a large ticketing platform. And one of the things that happened was someone came in, they couldn't find their ticket. I looked into their account. It's, hey, there's no tickets here. And then they were just, okay, well, what I'm going to do is I'm going to show up to the event and I'm going to find eight homeless people from the city and bring them with me. And the agent was like, "Oh, my God, that's so awesome that you're thinking about doing something nice for the community." Things like that, where it's like, okay, I will not expect that to happen. So those are just tuning you have to do over time. And generally it's in the spirit of that where you're trying to find the right level of guardrail and flexibility. I think that's the name of the game with our space, at least. And that's sort of the way we've designed our product. And why, probably the number one reason we've been successful so far, is that what GenAI really unlocks is super flexible, super personalized. One way you can think about it is in the old days, to map out a conversation, you just build a gigantic tree of decisions. And that's very hard because no one likes that experience. And you ask something that's not quite one of the branches and it just forces you down that branch and there's no way to go back. And what an LLM does is it abstracts a lot of that tree into the neurons of a language model. So that's really powerful. On one end of the spectrum, you're just looking for flexibility and raw power and being able to sound really human. But with the enterprise, there are a lot of things we don't necessarily want that as much. And you want full rigor, right? If it's a regulated use case, you cannot afford for it to ever deviate. These three steps always have to be followed in this order, and you can't go to step three until something has happened already. So you need to design a system that can be anywhere along that spectrum. And back to your question of what could go wrong, well, the worst thing that's happened would be it just says something that's not supposed to say. And so you need to design an AI that's really robust to that.