00:30:06 · Matt Stockton
[00:30:06] S1: Yeah. So it, you know, my view is that if you can find a way, like if you can find a way to use, to assemble the tools that exist in creative ways and kind of limit how much custom stuff you're building, that's ideal, right? So example would be, okay, if you can use one of the main harnesses like Codex or Cowork, and you can build a refined set of skills for your company that are aligned to what you're trying to do. And then you can have the suite of data sources, whether they're, you know, an MCP server, which, you know, allows you to connect to an API via the models. Like, if you can do that and then share the knowledge of, like, how that system that you've built fits into your workflow, like, what are the expectations of when people use the thing and why? That goes a long way, right? I think, you know, and that requires you to have access to the data sources, requires you to spend some time building out your intents and these skills. I think ultimately, you know, and this is probably a good thing, but like once companies are getting value out of that, they oftentimes hit something where, oh, we want to do something a little bit more custom. Like we want to run like reports on a schedule, or we have a data source that we want to integrate that, you know, doesn't have an MCP server. So like that's kind of when it's tricky because if you're like a smaller firm, you can't expect to have like an engineering team to do this. But I think, you know, sometimes you actually do have to kind of build something, right? And back to the bitter lesson idea too, is like, you have to be cautious here because things are like continually changing, So like you don't wanna build something that might be thrown away immediately, or you wanna build something in such a way that such that it can be like repurposed later, which is why I actually really like this concept of skills, because when it comes down to it, it's kind of just like writing down what you want in English, right? And it's a little bit bigger than that, but that's the simplification. And that's going to be, I think, durable over time, right, and useful.
00:32:29 · Khe Hy
[00:32:29] S2: I was about to ask you about skills. You're a big proponent of them. And I've found that many of the folks I work with, they can kind of get their first couple skills, you know, like, oh, you know, earnings call, like, transcript reviewer, kind of like the blocking and tackling. But once they kind of get those kind of core, and they tend to usually be summarization skills or synthesis skills, struggle, and myself too, to kind of expand the map of what's possible. And I know like OpenCLOF, for example, a lot of the code is actually skills files. And so how do you, how would you guide folks who are kind of earlier in their skills journey to move beyond just synthesis skills? Like whether it's a mental model or an actual, like, tactic that you would recommend for them?
00:33:18 · Matt Stockton
[00:33:18] S1: Yeah. You know, I still like to think about in terms of like, okay, what are you trying to accomplish? Right? Like, not necessarily like, what's the skills library I should build out? But, you know, back to Brett's point earlier is like, try something that you think the model will not do correctly. And hopefully you'll be pleasantly surprised. So I think it's more about expanding, first of all, expanding people's view on, like, what these things can do and then experimenting with it. Like, maybe you do something that's, like, more cross cutting in nature than what you're generally using these tools for. But it turns out too, like, if you go at it with an applied approach of just trying something and you get to a really good output, like, you can turn that into a skill, right? Like, you can ask the model to be like, Hey, like, let's do that again later. Can you help me turn that into a skill that I can reuse? So I like that applied approach versus, you know, just like once you kind of do have a recipe that works and you're like, This is what I want. Like, I think that the objective there is then to kind of make that durable and reusable, and then ideally, you know, share that knowledge back with your team, right? Like, embed that new skill or that workflow in the shared repository where other folks have access to it. Like, share that knowledge.
00:34:39 · Brett Caughran
[00:34:39] S0: Matt, have you found any best practices on skills creation? I mean, back to prompts, we had a speaker say beyond four to six pages, the chatbot starts to lose attention. There any equivalents that you've found in skills in terms of depth versus efficiency or other elements in creating skills that you've found helpful for people to think about?
00:35:09 · Matt Stockton
[00:35:09] S1: Yeah. I mean, I think, you know, these models, it's funny, they have a skills creator skill, which you can go in and use to create skills. The point of that is like these models obviously know what they expect to see within these skills. So I think that's useful to try. Ultimately, I think it's more about... I like to... This concept of like hill climbing, right? Where you try a thing, you look at its outputs, and you use your brain to say, Hey, is this good or bad and why? Right? And so, like, ultimately, when I'm building these things out, you kind of have to have this iterative loop. So maybe you build something that does something, produces an output, and then you say, okay, like, let me record myself talking about what's good and bad about this specific thing. Let me take that feedback, the output of the skill, the skill itself, and do work with the model to iterate the skill itself, right? I think kind of refining it is a pattern that I like to use, just trying to continually improve it. And ideally, you know, for some of these things too, it's really important to have kind of systematic evaluations. Like if you're kind of producing something, you ideally have a way to test its outputs, which is tricky if it's text, right? Because it's sometimes very subjective. But how can you specify what correct is and have a way to kind of continually test that your skills are doing the right thing over time?
00:36:44 · Brett Caughran, Matt Stockton
[00:36:44] S0: Have you thought about building evals? Because that's sort of hard as well too. Even like I'll eval things across three different LLMs, and I'll get a little bit of a different scorecard on my on my evals. [00:36:56] S1: Yeah. Yeah. It's definitely... It's it's tricky with with written text. Right? It's a lot easier if you're evaluating. Oh, is this number correct? Did I extract this specific sentence from this specific, section of the the filing? With with with text, I mean, can obviously do something... The technique is LLM as judge where you're using a LLM itself to review the outputs, but that's a little tricky because then, okay, what is the judge doing? How are you evaluating that? I mean, I think oftentimes, like, when I'm building these workflows for the space and the output is like a written report, you get a lot of utility by just reviewing those with the person that's consuming it. Like you say, okay, well, let's step through this memo and you tell me what's good and what's bad about this. And we'll record that and transcribe that and use that as input to try to figure out, okay, how do I adjust this skill? So that's, I mean, that's not a automated evaluation, but it's kind of an evaluation loop, which I think is really valuable, right? Like, because you're kind of going back to the person who's using the outputs and getting their feedback and trying to integrate that back into the upstream instructions.