00:17:38 · Mati Staniszewski, John Collison
[00:17:38] S1: I mean, there is... It seems like they might be going in that direction. Right? And I recently... They announced that we'll open up the LLM ecosystem. Hopefully, they will do the same with voice ecosystem, which is kind of similar. [00:17:48] S0: But again, it's I think rational to do when it's moving so quickly. Yeah. The voice assistant paradigm is one of the oldest paradigm... You know, UI paradigms in computing, like the open the pod bay doors, Hal, from Yeah. '19 Yeah. '69. Yeah. I will claim it's not working yet. So Siri doesn't have the intelligence. And then on Gemini and ChatGPT and those apps, I mean, I want to use the voice mode, but I don't know about you. It just doesn't work. And so, like, sometimes I'll be using my phone, and I'll use the iOS keyboard transcription Yeah. Yeah. Yeah. To type in the field and then, like, say a bunch of stuff and then send it off. But this suggests to me that consumers really want voice mode that works, and yet it's just not working yet for the major LLM apps or for anyone. Why isn't it working yet?
00:18:41 · Mati Staniszewski, John Collison
[00:18:41] S1: It is pretty hard to do because you want you want two things. You want you want to be able to say things that you want, but you want sometimes for it to execute it, sometimes to wait for you to, like, finish and add something in the sentence. Sometimes you want it to be interactive so it asks you questions back to clarify and get some of the additional detail. And all of that is actually pretty hard. That's where kind of the magical... Ideal version of a voice agent for us comes through, where you need the speech to text element, you need the transcription side, unique... You need then the kind of the turn taking mechanism. So, like, when do you finish sentence? When when is it likely based on silence, based ways... Likely on the context? And then sometimes you want it to speak back and clarify or at least give you the text back to clarify and then maybe execute set of instructions. So that problem is still very hard research. So I agree with the claim that this orchestration side has not passed a true conversational agent Turing test where it, like, behaves as you would expect from another person, where you [00:19:37] S0: can say, I that's the simpler way of saying what I'm saying is that we have passed the Turing test with text LLMs a long time ago, and we're actually nowhere near that on voice LLMs. So it's kind of interesting how that's a final frontier. [00:19:47] S1: Yeah. I feel like it's it's it's going to work in, like, specific domains. Like, in customer support, call [00:19:53] S0: Yes.
00:19:53 · John Collison, Mati Staniszewski
[00:19:53] S0: Yes. [00:19:54] S1: Passes the voice tune test. Works well. And, like, let's take another, like, spectrum of that, an interactive gaming experience, like, a truly interactive as you would have with another human in that game, it's, like, you know, so hard and and and and further out there, we haven't passed it yet there. [00:20:10] S0: Yes. Yes. [00:20:10] S1: Sounds good. Yeah. But I think that's a combination of, like, know, even like a simpler version of within that. Sometimes you might give a response immediately back. Sometimes you need a tool call to get additional information from the database, how you orchestrate that. So, like, that's probably the most common thing we see as we work with some of the companies out there is you want those systems to orchestrate extremely well where if it's a conversational use case, pretty simple, you can route the agent to speak with, but if you need to authenticate, if you need to pull additional information from the database, what do you do? How do you handle that graciously? [00:20:41] S0: That's really tricky. [00:20:42] S1: And, yeah, and and to that extent, I would grade that. That's just getting getting getting there. Yes. And we'll hopefully see that we... Our goal is to, like, pass the voice chewing test in all those cases or the chewing test for all conversational agents outside of voice two, and I hope we will all be there in the in the next year or so.
00:21:02 · John Collison
[00:21:02] S0: For subscription businesses, a lot of revenue is lost in that last few seconds before the checkout. Someone has to get up and find their wallet or they mistype their card number or they hit an error. They just get up and you lose the sale. For a company like ElevenLabs, adding hundreds of thousands of subscribers, even a tiny bit of friction like that, it would really add up. But that's why ElevenLabs uses Link from Stripe. Customers save their details once, and then they can check out in seconds across more than a million businesses with saved credentials. So if you want a faster checkout for your customers, you should turn on Link from Stripe. Are you guys working on personalized voice transcription? Where it feels like part of the way we're making it hard for ourselves is when I speak to Siri, I have a bit of an accent, and so it sometimes has a hard time understanding me, but my accent doesn't change. And so it could just get good at listening to John. But my understanding is it's not. It's just like running the global voice recognition model. And I'm guessing it's the same for ElevenLabs where you're running the global voice recognition model. But, again, you have an accent. And so if someone's understand... Like, if you walked up to someone in a coffee shop and said two words, they might have a hard time understanding it because they'd like... They're not putting it through their Mati Polish accent filter. And so where is this going with, like, actually interpreting the person that you know to exist on the other side?
00:22:24 · Mati Staniszewski, John Collison
[00:22:24] S1: Yeah. I have a I have a very tricky one to detect. So my voice is frequently used in the test Ah, [00:22:29] S0: you're from the test suite. [00:22:30] S1: Yeah. Text to speech, for speech to text, for, like, everything. Yeah. It's it's pretty... [00:22:34] S0: But, again, trying to parse your voice in a global model is just making life hard. It's like have a Mati specific model. [00:22:40] S1: Yeah. So so on the on the speech to text, like transcription, exactly. Like, the big part now that we are bringing in is you have two parts. One, effectively like a person or a voice specific detection, which is true for the accent side, but it's also true for a crowded room. That's where we incredible research team that's able to continually do both the accuracy high, but also Yes. Also also, like, add things like speaker detection, of course, noise reduction. But then the second part is also keyword detection. So there are specific words that you would want to say in those settings that you want to effectively monitor for. So we spoke about, you know, like, let's say I'm going to the coffee shop and order things. The set of actions the coffee shop would expect me to do [00:23:26] S0: is It's like there's a... You know, they can just listen out for the the coffee words.
00:23:26 · John Collison, Mati Staniszewski
[00:23:26] S0: is It's like there's a... You know, they can just listen out for the the coffee words. [00:23:30] S1: Exactly. And then try to, like, match it to the closest proximity. So both things will help. A setup where you have my voice, perfect, you can decode it and code it on that. If you don't have my voice or even if you want to double amplify it, we already support effectively a keyword detection, which is useful for, like, real time setting and async settings. So Yeah. Back to, Cheeky Pint transcription [00:23:54] S0: Yeah. [00:23:54] S1: You could effectively pre pre generate that from the previous podcast and, like, look for a set of words that you would use Yes. Traditionally in that. [00:24:02] S0: And so how hard... Okay. So you do the keyword detection already, but how hard are the... I want to get superhuman transcription performance by feeding it an hour of Mati audio before it listens to Mati, and then it should be able to do a much better job transcribing. Is that just a really hard research problem? [00:24:18] S1: No. Solvable. It's solvable. We think we can we can we can roll it out in one of the next versions, which is, like, hopefully in the next in the next month. [00:24:26] S0: Oh, so you think this year, you're doing Yeah. For sure. Person specific transcription. [00:24:30] S1: Person specific transcription. Like, we can already diarize speakers extremely well. Like, if we are speaking, we can, of course, dissimilate who is speaking when. [00:24:38] S0: Yes.