"Turpentine VC" | Venture Capital and Investing
00:06:01 · Brendan Foody
[00:06:01] S2: Yeah, so I'll start with Toptal. And then we can get into Turing. Toptal started really as an arbitrage on Eastern European developers, where they would hire, they would hire people there, they would manually interview them manually match people with jobs. And it was a much better experience than Upwork, which they were competing against at the time because they could assure a certain level of quality. The flip side of that quality is that they would charge an insane fee structure, oftentimes averaging 40% and not having the same wide net that you can get with automated processes. And so our thought process was, what if you can take this, you know, high service quality of Topdol and finding really high caliber candidates, but instead of having huge margins and huge fees, because all of those processes are manual, doing that with software, assessing not just like 1000s of people, but hundreds of 1000s of people finding better candidates at a lower cost structure, and delivering them to candidates much faster. So that's the comparison to Topdal and automating those processes. Turing started with the premise of doing a lot of automation in in the late 2000s and 10s. But the challenge at the time was that LMs were not capable enough to achieve what they wanted to or models in general. And it wasn't until, you know, chat GBT and GPT four, and all of those models being possible to fine tune that it actually became possible to assess talent at the cost of software. And so Turing built up their entire business as a manual business that was manually doing all of these processes with a lot of AI product marketing, and really, really doesn't have complete automation. So multiple executives that have left Turing are now advising us are excited about the approach of of completely automating these processes, and really building the best models to predict human ability is the most impactful thing on just ensuring that clients are having good experiences.
00:08:11 · Usman Hanif, Brendan Foody
[00:08:11] S1: Yeah. And so I guess touching on the why now a bit, would you point that to technology, which is sort of the advancements in AI recently allow you to do some of these operations a lot more efficiently and cost effectively? [00:08:26] S2: Hugely. Yeah. I mean, language models are getting extremely intelligent, especially when fine tuned on the right data sets. And that was just previously impossible. Like the idea of an agent conducting an interview of a human two years ago was completely impossible. Right? And now agents are able to not only conduct human level interviews, but conduct interviews that are way better. It's as if a human had prepared for twenty four hours before their interview, right, reviewing all of their GitHub repositories, thinking about all of the details of projects that they have on their resume and asking this in-depth, thoughtful interview about someone's background, giving each person the chance to be considered. And that was, you know, wasn't even close to possible before we had super powerful foundations.
00:08:26 · Brendan Foody, Usman Hanif
[00:08:26] S2: Hugely. Yeah. I mean, language models are getting extremely intelligent, especially when fine tuned on the right data sets. And that was just previously impossible. Like the idea of an agent conducting an interview of a human two years ago was completely impossible. Right? And now agents are able to not only conduct human level interviews, but conduct interviews that are way better. It's as if a human had prepared for twenty four hours before their interview, right, reviewing all of their GitHub repositories, thinking about all of the details of projects that they have on their resume and asking this in-depth, thoughtful interview about someone's background, giving each person the chance to be considered. And that was, you know, wasn't even close to possible before we had super powerful foundations. [00:09:16] S1: Yeah. It sounds like you've sort of offloaded a lot of the vetting process from humans to technology, specifically AI. We have a pretty technical audience. I think it'd be really fun to dive into some of that. On the side of the candidate, you know, they're often, as you mentioned, talking or always talking to the AI interviewer when they first sign up. There's also a resume analyzer tool I believe you guys use, plus a system that you mentioned earlier that can even predict candidate performance. I'd love to just start diving into all of that. You know? How did... You know, what is the technology behind that? How did you start building this? How do you even tackle this problem of teaching AI how to vet a candidate?
00:09:58 · Brendan Foody
[00:09:58] S2: Yeah. Yeah. I mean, it's it's a really challenging problem because humans struggle with it significantly. Right? So when you're creating evaluations for model performance, the the baseline in question that that you're trying to optimize for is very challenging. I would say, like, the first thing that's really interesting about the product is the AI interviewer, where being able to, you know, scalably do this volume of traffic requires a lot of pre processing to, you know, achieve the results that we want to as well as agentic behavior during the interview to be thoughtful about what the next questions are going to be. And then the other thing that's really fascinating are all of the models around predicting which candidates will do well, when we're surfacing candidates. The rough framework that we use for this is that there's two dimensions to how good a fit a candidate is for a role, there's the relevancy of that individual, and then there's the excellence of the individual, how competent they are. But these dimensions are like deeply intertwined. And so we do a lot of work with not only leveraging language models, fine tuning language models on huge amounts of proprietary data, but also on training, like learn to rank models from the ground up, and doing all sorts of fine tuned embeddings to surface the best candidates for customers.
00:11:19 · Usman Hanif
[00:11:19] S1: Yeah. In terms of, like, quantifying or even trying to qualify excellence, how do you sort of put that? I mean, you're mentioning how you're fine tuning models. What data are you looking at? How are you training these models? Anything that you can speak on there would be great.
00:11:34 · Brendan Foody
[00:11:34] S2: Yeah, there's two kinds of evaluations or KPIs that we go on. The first are KPIs around human preference, where we have internal members of the team that will create evaluations for like which candidates we think are best for roles for what reasons. And then we'll map to allowing models to predict that in the same way that humans can, considering, you know, all sorts leveraging language models and various other models to predict based on all of the attributes of an individual ranging from features of their GitHub contributions to their facial expression during an interview, who is likely passionate and likely perform well in that given role. The other kind of, like, evaluation KPI that we go on, and the more important one, in particular, as we scale, is around customer satisfaction, where we collect the reviews of all of the candidates and the experiences that customers are having on them, and then train models that can predict for a given candidate, what is the likely experience that they're going to have. So imagine like a transformer model that could intake all of the performance reviews of candidates for given roles and then can predict what is the most likely next performance review that this candidate is going to have, based on all of the data about their background, the role that they're being considered for and all the data that we find tuned up.