00:07:56 · Mati Staniszewski, Joubin Mirzadegan
If you can create synthetic speech that's high quality, the the voice will be such a powerful medium for for interaction and start ElevenLabs in 2022. We also... You know, we we we... Through the years prior... So here's how Google, I was a Palantir. We would meet and do hack weekend projects together across variety of different domains. I'm curious how you star... Stumbled on your on your idea as well. But in our case, we we were exploring new technologies and seeing what what they can do and how we can implement some of the... Mostly ideas around them, less so commercially focused ideas, and one of those was was was in audio. But that gave us a good kind of glimpse into how we can work together. We're so excited about going going full time and and dropping dropping our irrelevant companies and and starting ElevenLabs. [00:08:43] S1: Crazy. What an insane story. And then when the when the... When, I guess, what [00:08:49] S0: was it? GPT three came out? Was a year later? So the ChatGPT came December 2020 till January 2023. [00:08:58] S1: Not long into the journey. [00:08:59] S0: Not long into the journey. It's like Did you basically scrap everything? No. It was it was it was... So in a way, a lot of our ideas were based on the foundational stack anyways. [00:09:11] S1: Yep.
00:09:11 · Mati Staniszewski
[00:09:11] S0: So they were, like, just validated how relevant they are. They they... So, like, it was almost a parallel invention path, just a different technology. We, of course, have... A lot of the transform and diffusion ideas are are coming into our work into our work too. So when we started 2022, we would use some of the most relevant work recreating what should be do... What should be done in text to speech. And and then when chat GPT moment came and and a lot of the technology around us, this was, a, validating that we are doing the right thing because the ideas were giving good results. And, it kind of gave you this amazing overlap of what's possible where you could have ChatGPT now and you could have Voice on top of it. Suddenly, you can do content in such a much easier way Mhmm. Where ElevenLabs has even more advantage. I I guess, you know, like, the... At the time... So in 2022, most of the synthetic speech just was so poor. It was still very robotic. It's still... I don't want to name some of the big other companies, but but you can imagine the the hardware devices around how, like, the... That's like... The... It's still not good. And back in 2020, it was even worse. So it was truly robotic speech, and nobody could figure out how to make the speech sound human. So, like, actually represent emotions. And and that's where the 2022 for us was all focused. Like, how can we get the text, understand the context in a much better way, understand that the emotions around the text. Is it happy text? Is it dialogue? And bring that in the same way that the voice actor would or human would. When you read, you understand that context, and then you bring those emotions across. So that was the big innovation we brought in 2022 where text to speech sounded human. And the second big was being able to recreate voices. So how can you effectively encode a voice and the characteristics of the voice? And then when you produce synthetic speech, it still carries the style of the voice, the accent, the dialect, the... Whether it's narrated voice or conversational voice. How do you bring that all across? And we figured this out... This one out. But a lot of those ideas were based on similar ideas that the LLMs would carry as well, and then, of course, a lot of the the nuance that were required for the audio space.
00:11:32 · Joubin Mirzadegan, Mati Staniszewski
[00:11:32] S1: Mhmm. One of the things that I've been thinking about that I wanted your opinion on, and maybe this is just, like, a very stupid a stupid thought, but, like, we have this concept here in Silicon Valley about, like, crossing what's called the uncanny valley. And the uncanny valley is this concept where basically machines become indistinguishable from humans. Right. You cannot tell if it's an LLM talking to you or writing to you versus a human. Right. And so far, certainly in text, I can, within ten seconds, tell if an LLM has written it or not. It's very, very obvious. Based on [00:12:05] S0: the em dash? [00:12:06] S1: Or Based on the em dash. Based on just the way that it writes. It's not even just the em dash. Based on [00:12:11] S0: No. No. I agree.
00:12:11 · Mati Staniszewski, Joubin Mirzadegan
[00:12:11] S0: No. No. I agree. [00:12:12] S1: It's Like, not even random... Like, forget about the bolding and the em dashes. Right? Because nobody actually writes like that. Just the way that it actually writes does not sound human to me. And it's very obvious the minute that somebody sends me something, immediately, I'm I'm like, you didn't write it. Just like something inside my brain is like, that's not a person that wrote this. Okay? And so maybe audio, if it's coming off of text, is a similar paradigm today. And everybody continues to say, oh, no. No. It's happening. Like, it's going to happen for sure. And one of the questions that I've been asking myself, like, you know, these models have gotten a lot better, and they're still getting better. And everyone's like, well, as long as it keeps getting better, it's definitely gonna not sound... Like, it's gonna sound is... Indistinguishable from us. And, you know, I've listened to the, like, podcast LM from Google. Like, I've listened to all of this stuff. I've done the use ElevenLabs to clone my voice. And I'm not even saying... It's not a knock. Like, I'm not even saying it should or shouldn't sound like me. I'm like, it... Maybe it shouldn't. Like, maybe if you're from India, your dialect should be Indian. Like, maybe if you're from Iran, your dialect should be something else. Maybe if you're from machines, your dialect should be machine. You know? Like, I don't I don't know. And and and on that on
00:13:28 · Mati Staniszewski, Joubin Mirzadegan
[00:13:28] S0: that Lam quick back... Question back, is it across all use cases you can tell, or you can... Like, know, like, it would try... Writes a copy for your email or writes a copy for your post? Or is that... What what... Across all settings or just specific settings? [00:13:41] S1: I think unless they... Somebody has to really I think the only way that I generally can't tell is if somebody writes down what they first thought, then puts it into the LLM to to tweak what they thought, and then goes back and then, like, unbundles the things that sound like an LLM. [00:14:00] S0: Right. [00:14:01] S1: But I think that's across every use case. I I have seen I have seen that. Yeah. Do you agree, by the way? [00:14:08] S0: No. I don't think... I I I think I... Yes. In many settings, it's obvious where somebody is an LM and some of the things you mentioned make it obvious, but, of of course, like, there's something about a style. Like, if somebody goes the lazy effort of, hey. Write this for me. Immediate tell of, like, it's easy. But but I see more common the kind of the last thing of what you mentioned. So somebody will write the thoughts, get the LLM, rewrite the thoughts so they are, like, more human. And and I don't know if we are classifying that in the same category, but in those cases, I think it's very not... Or it can be very good. Like, I'm sorry. [00:14:42] S1: I classify it in that same category. I'm just saying if you tell an LLM to write something for you, just like if you tell an LLM to speak something for you [00:14:51] S0: Got it.
00:14:51 · Mati Staniszewski, Joubin Mirzadegan
[00:14:51] S0: Got it. [00:14:51] S1: Is it always gonna sound like an LLM? [00:14:53] S0: Yeah. The... So I think if you just use, like, a default, likely, yes. I do think now you can kind of, like, preprompt it where it becomes becomes a little bit less of a tell. And in some settings, it's it's it's it's it's as good or better than the the human would. And let... Let's let's take a customer support setting where we work a lot. If you're working in customer support, the baseline you are comparing it to is usually a IVR flow or, like, a decision tree of if this, then that, or a potentially human that has maybe not had as much of the training and and will try to help you but might not be able to help you. In that setting, I find ElevenLabs can provide as good, if not bad... Just on the text level, if better than human human experience where you get the support and, you know, you are comparing it to to something that's not there. [00:15:47] S1: Totally agree [00:15:47] S0: with you. [00:15:47] S1: If you're using Sierra and you're using ElevenLabs on the back end, and Sierra is partnered with United, and you're comparing the experience that United would give you pre and post, it's night and day.