Y Combinator Startup Podcast
00:15:03 · Max Hodak
Ideally, we'd be using entirely AI resistant homeworks now, so our favorite types of homeworks are things that don't saturate, have a very high ceiling, and are naturally scorable to two or three numbers that we can put on a plot so that when we get responses to homework, we can plot them all, and then it's very obvious when someone has really beaten the Pareto frontier, and we otherwise don't care whatever AI models they use. Like, that can make you better. In cases where that's not possible for homework right now, we're doing increasing number of technical phone calls or on-site practical tests. But ideally, we would have an AI resistant take home for each of these. Anthropic had a really interesting take on the AI resistant homework where they've had a couple tasks where there's like... It's like kernel optimization, like what is the minimum number of cycles you can get it down to, and this is naturally adjusting. Like, the hurdle for a while was... I think it was Sonnet's performance. If you could beat that, then you could get an interview. I think that there's a bunch of ways to construct AI resistant homeworks. And then by the time you get to the interview, it is important that you have, from there, reasonably high, like at least 25% conversion to an offer because otherwise it is just you're gonna waste too much of your time doing onsites for employees that don't convert. You can't get that down. And so this is... There's four steps to this, initial voting, the phone screen, homework, and a full interview. And this is, as far as... Like, from what... My experience, this is the minimum set of information that we need to make a, like, a full decision, and I don't think that there's a a more efficient way to elicit this. Like, I don't think there's a smaller number of steps that we could use. So this has become our process. So you're hiring people. They're coming into work. They're starting. They're incurring payroll. But how do you know that you're good at this? Like, eventually, you'll get feedback from the market on how good you are at hiring because the company will work or it won't. Like, your team will be capable of accomplishing the stuff that you've set out, and they'll help you course correct through that. But this is a very, very long feedback, and it's very poorly behaved loss function.
00:17:14 · Max Hodak
And so it's kind of your job as management to design synthetic gradients that allow you to find out earlier and along the way how recruiting is going and if need a course correction. A conventional answer to this is the three sixty review process. So once a year, you send out a lot of forms, you gather up a bunch of feedback around each employee, you set up a bunch of meetings with HR and with the various managers, and you can do the conventional performance review cycle, which, based on my experience, is this is a very disruptive process that doesn't tend to surface issues that you don't already know about but haven't acted on because you knew that thing was there, but firing people is hard, and so people drag their feet on it, and this is kind of reinforcing things you already knew. And it only happens once a year, maybe twice a year if you split up the company into cohorts. But I think, really, what would be nice to have is a signal that gives you this kind of natural feedback from across the company about who's good and who isn't and what's working and what's not in a way that is largely unbiased and is more continuous. Imagine if you could get feedback kind of every few weeks on where there are issues and where things are going well. And so the process that I had developed, which I've now used for the last six or seven years, is every few... Like, every couple weeks, every four to six weeks, it's not that often, people around the company get pinged with a question through the software, through Helix. And there's a form, but there's only one question that really matters, which is knowing how this person turned out, would you vote again today for their hire? It's the same questions we use on the initial voting. And so you'll get a prompt to say, like, this person you work with, like, how would you vote for their hire today? And then what we can do is we construct a graph over the company of all of the feedback. And so the basic intuition is that, like, your vote should be weighted more highly if everybody else has rated you highly. And the astute may notice that this looks a lot like the original Google algorithm PageRank, which is an idea called eigenvector centrality where you can create a weight over the graph by looking at how the graph points together. This is a little bit different than actual...
00:19:26 · Max Hodak
Literally, eigenvector centrality, but it's very similar, and so we call this technique eigenreviews. And I've become convinced that this is more or less the right way to do performance reviews. There's some other tricks that you have to apply to get this to work really well. For example, we apply dropout where we'll run a thousand iterations where we'll randomly remove some percentage of the edges each iteration. And then when you look at the distribution of scores that you get out of that, if you see additional peaks, example, this is included. There could be voting clicks that need further investigation. But as a whole, this is... It distributes the judgment across the company, updates more or less continuously with about a month lag, and gives you just way better insight into what's going on really around the company. So this is... And it also totally gets rid of that kind of traumatic, super heavy, once a year HR driven performance review process. So the point of this talk is not that you should use this in particular, although you should consider it. And if you actually roll this out at your company, you can email me, I'll send you a doc with more specific tricks on how to actually get this to work well. But the real theme of the talk is that rate of iteration separates success from failure. And if you can get a fast iteration loop, that really overcomes many other things you're gonna run into. And this effect is so severe. I mean, if you can learn one thing every week and there's a competitor that's learning a thing every month, they will never matter. Overwhelmingly, if there's... You're looking at different way... Like, two different approaches to solve a problem, if there's one that allows you to compound, like, in half... In a much shorter amount of time than the other, even if the other approach has significant redeeming characteristics, you should really consider going with the shorter iteration cycle because the compounding effect is just so dramatic. And so speed determines success and failure, and speed is determined by infrastructure. This is driven by really boring sounding things like how well do your purchasing and recruiting and spending processes work. This is as important as how well do you understand the object level technical content of the thing that you're building.
00:21:37 · Max Hodak
I see companies founded by just, like, stellar pedigree scientists and engineers all the time that die on the vine because this execution is tough to follow through. When your job is to organize, it's like... It's it's uncommon that these deep tech companies fail because the technology doesn't work. They fail because once you end up with this organization of hundreds of people and hundreds of thousands of square feet of physical infrastructure, you haven't built the systems to manage that and becomes unwieldy, and then you can't connect strategy to execution. So we heavily lean towards things that have shorter iteration cycles, kind of etcetera, all else equal, but that's not a blanket rule. Like, there are no blanket rules in startups. You're looking at each new fact pattern that comes in as its own unique thing and then making decisions that make sense to you. And one of the harder lessons as a startup founder, one of the harder things, I think, to to really deal with is the fact that you cannot delegate your judgment. As the CEO, you must always make decisions that make sense to you no matter how much momentum or inertia the alternatives seem to have. So in school, if you're... Let's say there's somebody sitting next to you and you cheat on the test by looking over at them, your grade will... All else equal, your grade will be dragged towards the average of the class. That is not good enough to succeed in startups. You have to do things that are at the long tail that you're the successful companies are the exceptions. By becoming an average, that is not good enough. And so in order to succeed, your judgment has to be differentiatedly good. Now, the reality might be that you don't know if your judgment is good yet. And so one way or another, you will have to find out. And that means making decisions that make sense to you even when you are totally alone in that realization. That is the only way to get to to the really big outcomes. Now, it's not that often that everyone else will think one thing, and you'll be like, you're all totally wrong, but it is a really eerie feeling. Like, you'll get to a point four or five years into the company when there's hundreds of millions of dollars on the line, and there's some really high stakes decision, and only you can make it, and then you will look around for advice.
00:23:49 · Max Hodak
Because, like, in the beginning, you'll get lots of... Like, there's a bunch of things that are easily advised or easily figured out, but you'll get to a key point years in, and you'll look for advice, and there is nobody to ask. And at that point, you must have a really good sense of the limits and boundaries of your judgment. That is a very eerie feeling, and you have to be able to commit to it regardless. Now, the good news is that in my experience, it's very difficult to actually get stuck. You can get yourself into trouble. And the action space is always larger than it appears. No matter what happens, there's usually when you get I think it's very easy to try and anticipate all kinds of problems that you'll never actually run into. And then you go and do it, and then you get to a point where the system... Like, you run into some real limitation. There's always 100 ideas about how to make it better. This is sometimes phrased as action produces information. This idea is, I think, much deeper than it sounds. So in physics, there's this quantity called action. And so if I throw a ball and it follows a ballistic like a parabolic trajectory, that trajectory is totally set, like when it leaves my hand, unless it gets blown by wind. Some other action is exerted on it. It will follow this ballistic trajectory, which is, in this sense, like kind of an information minimizing trajectory. I can say it just followed. It was ballistic. That totally determines it. If something else happens, you had to spend some energy time to cause that cause that to happen. And so when... Whenever you exert an... Like, action into the universe, that creates information, like, in a like, in a very fundamental sense. And whenever you get stuck, you have to start injecting action, producing entropy. And this produces some fairly counterintuitive effects. I've seen situations where the company is stuck in a deep local minimum, and there's someone who is great in many ways, but it's just the wrong fit for what that company is at the time. And removing them, even though they individually are very strong, unblocks the company and allows it to kind of enter a new phase. When you get stuck, you have to start doing things. And so the thing underneath the object level content of what the product you are building is, you've all these support systems.
00:26:01 · Max Hodak, S1
The company does purchasing and accounting and recruiting and performance reviews and budgeting and safety and quality is the operating system of the company, and that has a huge impact on how far you can take it. So speed is determined by infrastructure. Speed determines success and failure. You just need to put more thought into these... Into getting these foundations right. If you do them right at the beginning, everything else is much easier. If you get them wrong, you'll end up spending $5,000,000 a month and feel like you have very little control over it. And then you're forced into coarser levers and harder decisions. Thank you for coming to my TED Talk. [00:26:38] S1: Do you have advice for people trying to choose between industry and academia, starting a company now versus getting a PhD first?