Invest Like the Best with Patrick O'Shaughnessy
· Patrick O'Shaughnessy, Jesse Zhang
[00:00:00] S0: I think most people are still focused on things that could go wrong because it's non-deterministic. What about the total other end of the spectrum? What have been the things that have gone way more right than you expected? Where has the potential of agents outperformed your expectation in terms of what they can handle or what they can do? [00:00:00] S1: It's really just elevating the experience. One sort of metric, which is usually a secondary metric that folks think about later but is quite important to us, is just how often do people come in and just say, "Agent, agent, agent, give me to a representative, I don't want to talk to you." I've done that. You probably have as well where you're just calling into some sort of customer service and you're just pressing zero the whole time. And that's because people are used to bad experiences so they've already lost the trust of these systems. I think what surprised us was that if you just make it really clear off the bat that this is a different experience, people are willing to give it a chance and then the outcomes are just way different. One of our customers, Oura Ring, the wearable ring, we did a case study with them where before having any sort of GenAI system, one in three customers that came in would just not bother saying anything and just keep jamming agent until they got to one. Now it's one in 20 because we just spent a lot of time making the beginning of the process just feel very different and folks are willing to give it a chance. So I think that's been exciting.
· Jesse Zhang, Patrick O'Shaughnessy
[00:00:00] S1: It's really just elevating the experience. One sort of metric, which is usually a secondary metric that folks think about later but is quite important to us, is just how often do people come in and just say, "Agent, agent, agent, give me to a representative, I don't want to talk to you." I've done that. You probably have as well where you're just calling into some sort of customer service and you're just pressing zero the whole time. And that's because people are used to bad experiences so they've already lost the trust of these systems. I think what surprised us was that if you just make it really clear off the bat that this is a different experience, people are willing to give it a chance and then the outcomes are just way different. One of our customers, Oura Ring, the wearable ring, we did a case study with them where before having any sort of GenAI system, one in three customers that came in would just not bother saying anything and just keep jamming agent until they got to one. Now it's one in 20 because we just spent a lot of time making the beginning of the process just feel very different and folks are willing to give it a chance. So I think that's been exciting. [00:00:00] S0: Where do you think that can go? How good can the experience get in ways that it's not yet that good with subsequent evolution of your product but also of the underlying capabilities of the model?
· Jesse Zhang
[00:00:00] S1: The biggest frontier right now is voice. Voice models and so now there's a lot of interesting startups as well working on voice. It's exciting because it's still, I would say definitely not solved. There's a lot to be done there. The bar is very high. So if you just think about how humans communicate for literally the entirety of humanity, I don't know, 150,000 years or something, the UI for every human is language, spoken language. Listen, you speak. And that's how our brains have evolved. That's the most natural way for us to communicate. Only in the last, what, 60 years of that entire time, did we have keyboards and phones and communicating through typing? I would say fundamentally, in any sort of agent that communicates with humans, voice has to be a critical factor because that's just how we communicate, because our brains are so evolved for this. It's very easy to tell when something feels not quite right. And so the bar is very high. Uncanny Valley is quite large. That's why there's a lot of effort going into making the voice experiences good. I would say ChatGPT voice, for example, or the Seamless or these voice-to-voice models, they're starting to feel very impressive. But if you talk to it long enough, you can actually—you can definitely tell it’s not a human, so there's that element of it. But then, even for enterprise use cases us, there's still a ton of hurdles to cross because those models, even though they're good, the hallucination rate's really high, so you can't really use them necessarily as is in the current systems. And so a lot of people, what they do now is they go from voice into text and then back to voice, and then you can run a lot more checks there to make sure that things are accurate. And so a lot of cool ideas there to explore on how do you make it both human-like, but accurate, and how do you tie everything together. So that's where most of the work is going to these days.
· Patrick O'Shaughnessy, Jesse Zhang
[00:00:00] S0: At the risk of getting too technical, why is voice-to-voice interesting and worth pursuing versus just always going back to text and being able to manage that way? [00:00:00] S1: So the fundamental difference is if you're just going to text, then no matter what, the final audio is just a narration of the text. The voice-to-voice is powerful because it takes into the entire audio of what you said, so it knows cadence and maybe how upset you are and the tone and everything. Latency is a lot less as well because you're going straight from voice to voice and latency matters so much when we're talking. When we're talking right now, our brains are constantly going, okay, when's he done talking? When should I start talking? If someone interrupts someone else, it's in a polite way, people adjust very naturally. So that is the biggest proponent of voice-to-voice. And ultimately, I think the prevailing view is that for whatever the final experience is, if you really want to make it indistinguishable from a human, you have to do voice-to-voice, or you have to at least take into account the voice. The issue with voice-to-voice though, is that also fundamentally because voice has a lot more dimensions to it, the amount of tokens you generate per sentence is just a lot higher than when you generate text. The more tokens you have, the easier it is for something to go wrong. And so the hallucination rate has so far just been a lot higher. [00:00:00] S0: How much higher? Give us a sense of how far we are from these being really good.
· Patrick O'Shaughnessy, Jesse Zhang
[00:00:00] S0: How much higher? Give us a sense of how far we are from these being really good. [00:00:00] S1: I think probably 8x higher or something that. [00:00:00] S0: Wow. [00:00:00] S1: It is quite a bit higher. Of course you want to leverage that technology. So now maybe there's creative ways to make a hybrid of the two. Maybe you can have a text model generate the content, but you take into account the audio from before as well, and that makes something very realistic. But at the same time, latency is still the hard problem because at the enterprise what's happening is you are doing a lot. Before you can start responding, you have to figure out what are they asking about? What materials do I need to collect, do I need to hit any APIs and get that data back? And so you have to do it in a way that feels very natural. And sometimes if you think about how a human does it, you might have to say something such as, "Give me a sec to look that up," because it actually genuinely takes 10 seconds for the API to come back. These are all interesting problems to think through. [00:00:00] S0: So give us a sense today: if you add up all the interactions, some idea of how long they are, what type they are, voice versus text versus some other modality. What does the entire corpus of interactions between a Decagon agent and a customer look like today?
· Jesse Zhang
[00:00:00] S1: I would say pretty balanced at this point between chat and voice on a raw customer basis. There are more people in chat, at least for us. But if you just think about the large enterprises—the Fortune 100—they've just been around for so long and everyone just calls them. So voice is just disproportionately higher there. A lot of them are 90, 95% voice and 5% chat. So that's what we're seeing. And then in terms of the types of conversations, it's generally ones that are fairly... you start with the easier one, of course. And so these are things where they're question-answer based. So that's the tier one: hey, you answer their question based on what you statically know. So that could be questions about how your loyalty system works, or questions about if I bought something, would I still be able to refund it? Then the next level is it's still question-and-answer based, but you're leveraging a lot of real-time data. So that could be: I got 2x points on this transaction, but I should have gotten 5—why is that? And then it would actually go and look into your account and reason through things: okay, I see the account's listed this type, let me go find all the documentation on this type. And okay, actually it's because you booked through a travel agency. If you had booked directly with the airline, you would have gotten 5x, but this thing doesn't apply to a travel agency or whatever. And then the third tier is you're actually taking action. So I lost my credit card, I need a new one. And it's actually walking through a pretty large flow. And that's where AI agents have been really excellent because you actually wouldn't expect it to be able to do that. And so it's able to go in—it can be a pretty complicated system where it's: okay, well first I need to figure out what your address is and confirm if the address is correct. And then I need to look and see, hey, do you want me to lock the old card? Okay, great, I'll do that. I might need to check for fraud to make sure this person is not just constantly asking for new cards. And it's just all these things stitched together—that's really what makes it agentic. And that's why there's been such a step-function improvement with LLMs.