No Priors: Artificial Intelligence | Technology | Startups
00:07:36 · Mati Staniszewski
[00:07:36] S1: And I think that's right. It's almost exactly like when we started originally, what we did was try to actually use existing models that were in the market and optimize them for our first use case was actually starting with a combination of narration and dubbing, and then on the creative side, and we realized pretty quickly that the models that existed just produced such a robotic and not good speech that people didn't want to listen to it, and that's where my co founder's Genius came in, he was able to assemble the team and do a lot of the research himself to actually create a new version of creating that work. But, like, to your question, I think that the way we are kind of organized internally and how we think about sequencing a lot of that was looking at the first problem and then creating effectively a lab around that problem, which is like a combination of mighty researchers, engineers, operators to go after that problem. And the first problem was the problem of voice. So how can we recreate the voice? And like you say, it needs to have that research expertise to be able to do that well. So we started with effectively a voice lab, which was that mission of can we narrate the work in a better way. There was a combination of roughly five people that were doing that work, and then sequenced the research first, and then built a simple layer on top of that work to allow people to use that work, and then kind of expand it from there with a holistic suite for creating a full audiobook, and then creating a full movie narration, or movie dab. And then we move to the next problem, which is the realization that okay, we have solved the voice, great for making content sound human, the first problem. For that to be useful for us to interact with the technology, you need to solve how you bring the knowledge on demand into that. So, we effectively started then the second team, which was a second lab, an agent lab effectively, which was a team that would combine researchers, engineers, and operators once more, which would try to fix, okay, we have text to speech. How do they now combine this with LLabs and speech to text and orchestrate all those components together while integrating that with other systems to make it easier.
00:09:41 · Mati Staniszewski, Sarah Guo
And then similarly, you know, you kind of expand from looking just at the voice layer into how those systems work together, and here too, you need the research expertise to do that in a low latency way, efficient way, accurate way, but at the same time there's that product layer that starts forming that it's not only the orchestration that matters, it's also the integrations of how you link up to the legacy systems, how you build functions around it, or how you deploy that in production and test, monitor, evaluate over time. [00:10:11] S0: Do you feel like you were creating new use cases when you built the tools? Do people know that they wanted to do this already? Because one argument like that I remember hearing was like, ah, like, you know, enterprises don't know what to do with voice. How many people really want to do it? And then you're serving essentially like perhaps the creator publisher side of Right. Your Yeah.
00:10:31 · Mati Staniszewski
[00:10:31] S1: It's definitely a combination of like initiatives that we believe will happen in the world, and they're like response to a lot of that. Like as I think back, we, you know, of course the internal VoiceLab or AgentsLab then kind of that kick started so many of the other labs in response to the problems. We started a MusicLab because people wanted to create music with ElevenLabs, so was a fully licensed model where people wanted to use and create speech, but they wanted to add music in a simple way. We wanted to deliver that. And then, of course, that kind of came together through how do we combine music, audio, sounds. We are now integrating partner models from image and video into that suite, how could you combine all of that in one, and all of that was in response to the market saying us like, hey, we would love this. And then you will have completely different use cases, even in that space, let's say dabbing. Dabbing is a use case that we didn't feel there was a big push for that, but we knew that in the ideal world in the future, you will be able to have that content delivered naturally around the languages, still carrying that. I still think actually this market will be immense because it's not going to be only the static delivery in movies, but if you travel around the world want to in real time, like the full Babelfish idea from Hitchhiker's Guide to the Galaxy, this will happen. It will be like the biggest like the whole breaking down language barriers, the barriers to communication, to creation, all of that will break, and that will be the foundational real time dabbing concept. So, excited about that part. And similarly, on the agent side, are some obvious things that of course customers that we work with or partners will want to integrate, which is we want integrations with XYZ systems. But then there are other parts that might not be as easy to predict of as you interact with technology, of course want to understand what's happening, but you also want to understand how the things are being said and bring that into default, which would be something we try to prioritize on our side, so then the people, when they actually interact with technology, they realize, oh, expressive thing is actually so much more enjoyable and beneficial and helpful.
00:12:36 · Sarah Guo, Mati Staniszewski
[00:12:36] S0: So, I wanna ask you a question about this, which relates to quality. I work with a series of companies where we're selling a product to the buyers. There's generally not machine learning scientists. [00:12:49] S1: Right? Right. [00:12:49] S0: And even the scientific community does not have the full suite of evals and benchmarks to understand every domain well. It's a well known problem. But I imagine for a lot of your customers, it's not like they know how to choose good voice. So how do you deal with that problem? Is it like, hey, I make a clone, and that sounds like me, and I believe in it. I'm gonna try all of these different options, or actually, are you teaching people to do eval? [00:13:15] S1: It's a great question because I think there are two big problems. One is how do you benchmark the general space in audio where, like you say, it's so dependent on the specific voice, let alone if you are training it to interactive, then it's even more tricky. And then the second piece, which is as you are working on a specific use case, how you select a voice. So, I'll take the second front first, which is we have a voice sommelier effectively with us. We work with enterprises. We deploy that person to work with them and help them navigate. That person is like a voice coach, has an incredible voice themselves, and now we have like a team under that person that like will partner to help you find what's the right branding of Vanity Boy. [00:13:57] S0: And now you have like the celebrity marketplace.
00:13:57 · Sarah Guo, Mati Staniszewski
[00:13:57] S0: And now you have like the celebrity marketplace. [00:13:59] S1: And now you have a celebrity marketplace to like help you even get get the iconic talent in there, like Sir Michael Caine. That piece was important because, of course, their voice will depend on the use case that you are trying to build, the language. All of that will have an impact of what's the right voice for your customer base. So, we have effectively a voice person helping those companies, and some companies will be very opinionated on what they want. So, they will sometimes select themselves, sometimes give us a brief of, hey, we want a voice that sounds professional, neutral, it's coming. We recently had a company, one of the biggest European companies that gave us a brief, which is very original, that they wanted as robotic voice as possible. [00:14:42] S0: Okay. [00:14:43] S1: It was counterintuitive. Yeah. First of [00:14:46] S0: So you're like, we can't do that anymore. [00:14:48] S1: Almost, but we are like trying to go backwards of like, how do we do that? But I think we got a good result. But recently we had a company in Japan and Korea, where they wanted to serve different voices depending on the customer that's calling in. They have an older population and a very younger population. The younger one, they wanted one of the famous voices in the market that's very excitable and happy, and for the older one, wanted a calm, slow speaking one. We help a lot with that. So, that's on the voice piece, and I do think it's going to be a big and important one.
00:14:48 · Mati Staniszewski, Sarah Guo
[00:14:48] S1: Almost, but we are like trying to go backwards of like, how do we do that? But I think we got a good result. But recently we had a company in Japan and Korea, where they wanted to serve different voices depending on the customer that's calling in. They have an older population and a very younger population. The younger one, they wanted one of the famous voices in the market that's very excitable and happy, and for the older one, wanted a calm, slow speaking one. We help a lot with that. So, that's on the voice piece, and I do think it's going to be a big and important one. [00:15:20] S0: It's like a personalized choice, and then it can even be dynamic in a customer.