Invest Like the Best with Patrick O'Shaughnessy
· Jesse Zhang
[00:00:00] S1: It ranges all sorts of different things. Sometimes it's obviously both sides, us and the customer. We take everything very seriously. But there's just a lot of things you wouldn't expect in the early days, we have a customer that is essentially a large ticketing platform. And one of the things that happened was someone came in, they couldn't find their ticket. I looked into their account. It's, hey, there's no tickets here. And then they were just, okay, well, what I'm going to do is I'm going to show up to the event and I'm going to find eight homeless people from the city and bring them with me. And the agent was like, "Oh, my God, that's so awesome that you're thinking about doing something nice for the community." Things like that, where it's like, okay, I will not expect that to happen. So those are just tuning you have to do over time. And generally it's in the spirit of that where you're trying to find the right level of guardrail and flexibility. I think that's the name of the game with our space, at least. And that's sort of the way we've designed our product. And why, probably the number one reason we've been successful so far, is that what GenAI really unlocks is super flexible, super personalized. One way you can think about it is in the old days, to map out a conversation, you just build a gigantic tree of decisions. And that's very hard because no one likes that experience. And you ask something that's not quite one of the branches and it just forces you down that branch and there's no way to go back. And what an LLM does is it abstracts a lot of that tree into the neurons of a language model. So that's really powerful. On one end of the spectrum, you're just looking for flexibility and raw power and being able to sound really human. But with the enterprise, there are a lot of things we don't necessarily want that as much. And you want full rigor, right? If it's a regulated use case, you cannot afford for it to ever deviate. These three steps always have to be followed in this order, and you can't go to step three until something has happened already. So you need to design a system that can be anywhere along that spectrum. And back to your question of what could go wrong, well, the worst thing that's happened would be it just says something that's not supposed to say. And so you need to design an AI that's really robust to that.
· Jesse Zhang, Patrick O'Shaughnessy
And you can choose, okay, for this use case, we really need it to be over here. That's a lot more robust. But for other use cases where if you're just asking basic questions on their account, we don't want it to be that. We want it to be super free form and that's how we get the customer satisfaction up. [00:00:00] S0: I think most people are still focused on things that could go wrong because it's non-deterministic. What about the total other end of the spectrum? What have been the things that have gone way more right than you expected? Where has the potential of agents outperformed your expectation in terms of what they can handle or what they can do?
· Patrick O'Shaughnessy, Jesse Zhang
[00:00:00] S0: I think most people are still focused on things that could go wrong because it's non-deterministic. What about the total other end of the spectrum? What have been the things that have gone way more right than you expected? Where has the potential of agents outperformed your expectation in terms of what they can handle or what they can do? [00:00:00] S1: It's really just elevating the experience. One sort of metric, which is usually a secondary metric that folks think about later but is quite important to us, is just how often do people come in and just say, "Agent, agent, agent, give me to a representative, I don't want to talk to you." I've done that. You probably have as well where you're just calling into some sort of customer service and you're just pressing zero the whole time. And that's because people are used to bad experiences so they've already lost the trust of these systems. I think what surprised us was that if you just make it really clear off the bat that this is a different experience, people are willing to give it a chance and then the outcomes are just way different. One of our customers, Oura Ring, the wearable ring, we did a case study with them where before having any sort of GenAI system, one in three customers that came in would just not bother saying anything and just keep jamming agent until they got to one. Now it's one in 20 because we just spent a lot of time making the beginning of the process just feel very different and folks are willing to give it a chance. So I think that's been exciting.
· Jesse Zhang, Patrick O'Shaughnessy
[00:00:00] S1: It's really just elevating the experience. One sort of metric, which is usually a secondary metric that folks think about later but is quite important to us, is just how often do people come in and just say, "Agent, agent, agent, give me to a representative, I don't want to talk to you." I've done that. You probably have as well where you're just calling into some sort of customer service and you're just pressing zero the whole time. And that's because people are used to bad experiences so they've already lost the trust of these systems. I think what surprised us was that if you just make it really clear off the bat that this is a different experience, people are willing to give it a chance and then the outcomes are just way different. One of our customers, Oura Ring, the wearable ring, we did a case study with them where before having any sort of GenAI system, one in three customers that came in would just not bother saying anything and just keep jamming agent until they got to one. Now it's one in 20 because we just spent a lot of time making the beginning of the process just feel very different and folks are willing to give it a chance. So I think that's been exciting. [00:00:00] S0: Where do you think that can go? How good can the experience get in ways that it's not yet that good with subsequent evolution of your product but also of the underlying capabilities of the model?
· Jesse Zhang
[00:00:00] S1: The biggest frontier right now is voice. Voice models and so now there's a lot of interesting startups as well working on voice. It's exciting because it's still, I would say definitely not solved. There's a lot to be done there. The bar is very high. So if you just think about how humans communicate for literally the entirety of humanity, I don't know, 150,000 years or something, the UI for every human is language, spoken language. Listen, you speak. And that's how our brains have evolved. That's the most natural way for us to communicate. Only in the last, what, 60 years of that entire time, did we have keyboards and phones and communicating through typing? I would say fundamentally, in any sort of agent that communicates with humans, voice has to be a critical factor because that's just how we communicate, because our brains are so evolved for this. It's very easy to tell when something feels not quite right. And so the bar is very high. Uncanny Valley is quite large. That's why there's a lot of effort going into making the voice experiences good. I would say ChatGPT voice, for example, or the Seamless or these voice-to-voice models, they're starting to feel very impressive. But if you talk to it long enough, you can actually—you can definitely tell it’s not a human, so there's that element of it. But then, even for enterprise use cases us, there's still a ton of hurdles to cross because those models, even though they're good, the hallucination rate's really high, so you can't really use them necessarily as is in the current systems. And so a lot of people, what they do now is they go from voice into text and then back to voice, and then you can run a lot more checks there to make sure that things are accurate. And so a lot of cool ideas there to explore on how do you make it both human-like, but accurate, and how do you tie everything together. So that's where most of the work is going to these days.
· Patrick O'Shaughnessy, Jesse Zhang
[00:00:00] S0: At the risk of getting too technical, why is voice-to-voice interesting and worth pursuing versus just always going back to text and being able to manage that way? [00:00:00] S1: So the fundamental difference is if you're just going to text, then no matter what, the final audio is just a narration of the text. The voice-to-voice is powerful because it takes into the entire audio of what you said, so it knows cadence and maybe how upset you are and the tone and everything. Latency is a lot less as well because you're going straight from voice to voice and latency matters so much when we're talking. When we're talking right now, our brains are constantly going, okay, when's he done talking? When should I start talking? If someone interrupts someone else, it's in a polite way, people adjust very naturally. So that is the biggest proponent of voice-to-voice. And ultimately, I think the prevailing view is that for whatever the final experience is, if you really want to make it indistinguishable from a human, you have to do voice-to-voice, or you have to at least take into account the voice. The issue with voice-to-voice though, is that also fundamentally because voice has a lot more dimensions to it, the amount of tokens you generate per sentence is just a lot higher than when you generate text. The more tokens you have, the easier it is for something to go wrong. And so the hallucination rate has so far just been a lot higher. [00:00:00] S0: How much higher? Give us a sense of how far we are from these being really good.