00:28:25 · Paul Todgham, CJ Gustafson
That's an example where if you can kinda make a pretty clear ROI, price for it. And and and, you know, don't be afraid to be kind of aggressive about it. When there's market skepticism, which there is, you know, some for voice AI. Right? Maybe due to bad experiences with earlier tech or just the optics of AI interviewing. We focus on, again, communicate the value, but then make trial as easy as you can. We're very clear about what our, you know, our pricing is and how much our implementation might be costing. But then for certain customers and certain dynamics, are we willing to give a free trial or a free pilot or a discount for a three month period just to get folks using it? Or start with one department before you roll it out elsewhere. I would say every product is different, starting with, again, what are the unit economics? And how close does this sort of tie to some some unit of value that the customer understands. [00:29:08] S1: You have such a clear value metric, I think, is what you're describing across those different products, which makes it easier to price. I mean, pricing something in the AI is never necessarily easy, but at least you understand how the end user looks at value or how they quantify it. I guess related to that, how do you begin to think about willingness to pay?
00:29:26 · Paul Todgham
[00:29:26] S0: On the plus side, I think we can communicate value really well. The challenge we often face is that we're talking to talent teams who are often managed as cost centers, by their CFOs. So I'm grateful actually to have a chance that if there are sort of CFOs listening to this, which is your primary audience, just to sort of say, it takes effort to kind of really build a strong partnership between your people function and your finance function to kind of really think about the ROI of your hiring and and not just, you know, again, how much cost, you know, the the cost equation only. But it's worth that effort because, again, our people are so important. The cost of a bad hire is so high. I'll start with just kind of that, just an appeal to your audience, which I'm I'm I'm grateful to have the chance to connect with today. In terms of willingness to pay, again, I think it depends particularly on on the tool. But, I mean, ultimately, people see a lot of the value in these tools. They just need to know that it's predictable. And, again, if it's incremental to what they were paying before, there's gotta be incremental value. So, you know, I talked about AI interviewing as one example, but another one that we've launched is AI note taking. You and I are having this conversation and someone's transcribing the notes. If it was with Greenhouse, we'd be doing a... You'd doing a structured scorecard, right, questions to test certain competencies. And your scorecard would be populated, you know, after the fact. It's a pretty easy thing to predict. Right? You can predict how many interviews a company will conduct in a given year. We have data on that. So forecasting usage of it is pretty straightforward. And then there's just a question about, you know, value and willingness to pay. And, again, you can go straight to kinda how much time do think someone's spending writing up their notes after an interview. What's that worth director and up level? That's worth an awful lot, and you can quantify that.
00:30:50 · CJ Gustafson, Paul Todgham
[00:30:50] S1: I like how you make it something that's relatable to the user because a lot of people would put their hand up and say, I I don't truly understand tokens, how to count them, what they're worth. You have to put it in somebody's own vernacular of how they run their department. [00:31:04] S0: We're all facing this, you know, both as users and and producers, I would say, if you're, you know, working at a software company. When I ask something of of my LLM of choice, I have no idea how many tokens they're gonna use. And sometimes they they give a really, really long answer for something that I thought I just wanted a quick, you know, a quick response. So it's an evolving world. I think that'll get better over time. But right now, there's sort of this anxiety, and there should be around... When I press enter and I chose Opus, let's say, you know, versus versus Haiku or, you know, you know, did I... Or or high versus medium, you know, did I make did I make the right choice for the the problem I'm trying to solve? [00:31:35] S1: Paul, you told me that a lot of the ways companies are measuring AI ROI, that's always a hard one to say, AI ROI, that that they don't hang. What are we potentially kidding ourselves about?
00:31:35 · CJ Gustafson, Paul Todgham
[00:31:35] S1: Paul, you told me that a lot of the ways companies are measuring AI ROI, that's always a hard one to say, AI ROI, that that they don't hang. What are we potentially kidding ourselves about? [00:31:45] S0: I don't think there's any CFOs or finance leaders who aren't talking about this right now. And again, I don't feel like I have a monopoly on the answer. I think that there's, you know, a bunch of risks you can kind of fall through. I mean, probably the single biggest is just, oh, we've just counted up all the hours saved. And we're kind of done. Right? Oh, we spent this money, we saved a bunch of hours, and just kind of move on from there. Boards want an answer. And people are just coming with a super a really superficial answer that doesn't stand up to a ton of scrutiny. I mean, I don't think we've necessarily got this right, but we're certainly working really hard to to truly understand all parts of the ROI equation, you know, as it comes to AI. And I do think it's different than, you know, maybe how we've looked at traditional software or IT investments in the past. [00:32:27] S1: How are you measuring ROI internally? I I would imagine that, like you said, no one has it fully figured out, but I think everyone has their own flavors of the questions they like to ask and the metrics they like to review.
00:32:37 · Paul Todgham
[00:32:37] S0: I worked with an AI to kind of develop my framework, you know, as I gave some ideas and pressure tested it. I had some healthy pressure testing back. But first, there's four things we're doing. First, just tracking AI costs. If you're going to understand the ROI, what is the I? How much are we actually investing? And how is that changing over time? There's some judgment there, by the way. It's not super definitive. Are you dealing with partial time allocations about people getting onboarded and enabled and stuff? There are some vendors where it's super clear, the LLMs. And there are some where there are vendors who've moved from seat based to hybrid. And the biggest thing I think we're doing that I feel like is kind of feel most conviction of is we've anchored on cost as a percent of revenue by function as our primary ROI metric. I believe that AI, if it's going to make us more productive, should make us growing more profitably as a business. The way to measure that would be your EBITDA percent margin and cost of percent of revenue is basically just the sum of that. But it's a it's a... It's something you can take to every single function. So I can, you know, I can't give EBITDA to the product team, but I can show them their cost of percent of revenue. I can show marketing their cost of percent of revenue. Over time, we should be seeing this metric go up. And it can go up in two ways. Right? It can go up by lowering the numerator, lowering your cost over time, or, you know, increasing the denominator, growing our revenue as a company faster. So although the first word is cost and it's coming from a CFO, so people mostly think I'm doing this just to reduce cost, it actually is about profitable growth. You have conviction that these initiatives we're doing are gonna increase your growth rate, and they're very favorable in the... In in that. The trouble with cost as a percent of revenue, it's a really big metric. There's a lot of different things that can affect it, not just AI. Right? So we've gone then sort of deeper into what, you know, the true AI activities and the AI metrics are just to help build conviction that these actions we're taking are ultimately gonna help improve our cost as a percent of revenue. So each function at Greenhouse is aligned on a a small number of AI impact metrics is what they're calling them.
00:34:24 · Paul Todgham
But they're basically like leading indicators or, you know, things that over time will lead to a better cost of percent of revenue. The easiest example to understand would be coders. Right? You know, r and d folk... Coders can code much more effectively or efficiently with AI than the past. So our R and D team is measuring, you know, both the kind of velocity and output, you know, how many PRs per per week or month are they generating, as well as a quality metric, our bug rate, which, you know, we want our efficiency to be going up or our product to be going up with no declining quality. And that's kind of the two metrics they're being held to. The fourth piece is sort of a set of impacts... AI initiatives. So we've asked each function to articulate, you know, what are your top three to five initiatives in each function that are helping drive those AI metrics and ultimately leading to lower costs as percent of revenue over time. So I kinda know from that level, what are the projects I'm looking at to say, hey. Is this on track? Is it meeting its goals? Are we focused on the right things where I have shared resources, let's say, in IT? Which of these projects should we devote their time to based on, you know, the perceived return? Then sort of the first level, am I seeing the metrics move that need to move? And then, ultimately, is it leading to, you know, a lower cost of a percent of revenue, which, you know, we've now built into our, you know, next year budget cycle? So this year for us was a lot about kind of leaning in, experimentation, enablement, and obviously next year is a little more doing those things and and and proving the ROI.