00:23:26 · John Collison, Mati Staniszewski
[00:23:26] S0: is It's like there's a... You know, they can just listen out for the the coffee words. [00:23:30] S1: Exactly. And then try to, like, match it to the closest proximity. So both things will help. A setup where you have my voice, perfect, you can decode it and code it on that. If you don't have my voice or even if you want to double amplify it, we already support effectively a keyword detection, which is useful for, like, real time setting and async settings. So Yeah. Back to, Cheeky Pint transcription [00:23:54] S0: Yeah. [00:23:54] S1: You could effectively pre pre generate that from the previous podcast and, like, look for a set of words that you would use Yes. Traditionally in that. [00:24:02] S0: And so how hard... Okay. So you do the keyword detection already, but how hard are the... I want to get superhuman transcription performance by feeding it an hour of Mati audio before it listens to Mati, and then it should be able to do a much better job transcribing. Is that just a really hard research problem? [00:24:18] S1: No. Solvable. It's solvable. We think we can we can we can roll it out in one of the next versions, which is, like, hopefully in the next in the next month. [00:24:26] S0: Oh, so you think this year, you're doing Yeah. For sure. Person specific transcription. [00:24:30] S1: Person specific transcription. Like, we can already diarize speakers extremely well. Like, if we are speaking, we can, of course, dissimilate who is speaking when. [00:24:38] S0: Yes.
00:24:38 · John Collison, Mati Staniszewski
[00:24:38] S0: Yes. [00:24:39] S1: Which is like, in transcription side, apart from accuracy, diarization is one of the harder problems. And We do that extremely well. Now it's going to be effectively what you're saying, fine tuning based on the speaker that I want to listen to, which we know will be important. In healthcare setup, such an important part. You're in an operating room, you're a doctor, you want to say a command, then you want to really be able to listen to that one person specific piece. [00:25:03] S0: Yes. [00:25:03] S1: You have a hardware device at home, let's say it's a pilot, that helps you control the the the TV. Here too, you will want that to listen to you versus, let's say, the family roaming around, or maybe you want it to everyone. So, like, you you could decide that, but in in many cases, you you want to be able to specify that. [00:25:20] S0: Okay. That's really exciting. [00:25:22] S1: It's great because there's, like, still so many unsolved research problems that yeah. We They're, there's just [00:25:28] S0: breakthrough after breakthrough coming in the in the domain of voice models. How about on the flip side when it comes to speech generation? You know, the Zoom touch up my appearance feature? Yeah. Yeah. I've always thought about that in the context of voice where should you offer a deaccenting filter for voices? Or, like, even this one podcast that I like to listen to put the voice a little mumbly, and I always thought they should put it through a demumbling filter just to, like, make Or slow [00:25:54] S1: it down, slow it down.
00:25:54 · Mati Staniszewski, John Collison
[00:25:54] S1: it down, slow it down. [00:25:55] S0: Make the enunciation a little better. Yeah. But all these things, again, like photoshopping an image Yeah. There's no reason that the... Like, have you thought about voice to voice, basically, rather than voice to text or text to voice?
00:26:07 · Mati Staniszewski
[00:26:07] S1: Yeah. Well, that that... So there are of two big parts. One, on the speech generation side, similar. So many so many innovations still there. There's, like, a wider piece, and that's, like, the... We released a v free model that kind of we're solving that for the first time. It's like, can you control speech? So you can have the text to speech. You generate something that sounds emotionally great. Previously, until end of last year, effectively, would rely on model to decide what's the best performance. You could regenerate it, but but, ultimately model decides the best performance. So that's where the controllability came in, where we can finally give it cues of, say it in a slower way or change how you deliver the dramatic pause or kind of any, cues that you give, and to be able to do that, you need architectural changes and the data that we kind of created over time where you annotated what was said and how it was said, you can actually train the model to do that. So today, finally, you can have both speed generation or entire voice agent experience with what we call expressive mode, where the agent knows the emotions on the other side, so if the person is stressed, it can react and be reassuring, and that's generating a LM response on the reassuring side and response in that set of emotions too, and that breakthrough was super hard to do, and that of course stretches to a lot of what you said. It could be some version of speech enhancement, either real time or in a pulsed setup to change how that's delivered, and that's relatively recent innovation, and we know it can still be so much better. Like, the edge cases of how you want to describe it is pretty large. So that's one. And then the second part of the question, which is a huge question, the speech to speech models. So as you said, our approach, as you think about voice agent, conversational agent, is effectively a cascaded approach. You use transcription of speech to text, LLM, text to speech, and orchestrates all of that together. And then you have a speech to speech, which kind of goes directly from speech, and there's a speech response on the other side.
00:28:04 · John Collison, Mati Staniszewski
[00:28:04] S0: When I say speech to speech, is that the idea that it doesn't go through text as an encoding in the intermediate? Oh, exactly. Okay. For performance reasons, for accuracy reasons... [00:28:13] S1: You usually do it for latency. Okay. For latency. Like... [00:28:15] S0: Faster to run a model that does not have to transcribe and then generate. [00:28:19] S1: Exactly. It's it's quicker, but on the flip side, you lose reliability. [00:28:23] S0: Yes. [00:28:23] S1: You lose, like, all visibility into the parts of the pipeline. And emotionality, we think you can deliver both on both sides extremely well, and maybe you can make it more controllable too. So today, we are optimizing heavily on a cascaded approach. [00:28:36] S0: I'm sorry, a cascaded approach is... [00:28:37] S1: Is the speech to text... Going for the text layer. [00:28:39] S0: Okay.
00:28:39 · John Collison, Mati Staniszewski
[00:28:39] S0: Okay. [00:28:40] S1: Going for the text layer. And and as we work with, like, a lot of the businesses and enterprises, they they will need that visibility into what happens. They will want to execute certain tasks. On top of that, they want a good visibility into each of the steps and and great accuracy of all the models. But beyond that, they can abstract away what's the LM layer, what's the intelligence layer. The integrations are easier in that system. So that's, like, where we are betting a lot of the the research work of how you can make that great, and we think we can make that great. And speech to speech, as you think about maybe more of, a companion version of the applications, of that's where that will flourish because maybe the hallucinations aren't as important, but the latency is a little bit more, and maybe hallucinations are even a feature. And maybe in the future future, just to finish that part, you will have, like, some version of combination of the models that for, like, low complexity, easy models, you will have speech to speech. And for, like, higher layer complexity, you will have the cascaded. [00:29:32] S0: Okay. So I was gonna ask about this. Like, you know the way there is research on how the invention of writing changed humans' brains and just, like, changed the the neural pathways in ways beyond kind of the the actual written language. Do you observe that speech to speech models think differently than cascaded models? Like, sounds like they're dumber.