Zero-Shot Learning
00:15:11 · Dev Tagare, Adarsh Hiremath
[00:15:11] S1: Yeah. [00:15:11] S2: Yeah. And let's say you queue up a a piece of material that is not desirable or one of the stakeholders flags as, you know, being incorrect. Can you go into the platform and update and suggest the diff? And do that over and over again for a bunch of use cases. [00:15:23] S1: Makes a lot of sense. So, like, we're we're sort of entering into the the context discussion, if you will. One thing that keeps coming up, Adarsh, is the ontology for context. Right? Different companies call kinda similar things differently. I may call Jira a buganizer, you may call Jira just a ticket or in my queue or something like that. Right? And the fields could be different. So in this space, how do you deal with the problem of semantic meaning of data or ontology as a problem space? And then how do you define success, in terms of, like, an on ramp, assuming the company's ontology Mhmm. And your interpretation of it? [00:16:04] S2: The thing that we have found is understanding semantic meaning in a very, very big comp... You know, the company dataset is actually more or less a solved problem. Mhmm. Right? Of course, there are different optimizations that you can make to make recall, you know, better and better. Mhmm. But for the most part, taking that data, splicing it appropriately [00:16:25] S1: Mhmm.
00:16:25 · Dev Tagare, Adarsh Hiremath
[00:16:25] S1: Mhmm. [00:16:26] S2: Putting it in a high dimensional space using whatever embeddings model you want, then retrieving over it is something that will give you very, very good results, especially when paired with a really, really smart model. So that understanding piece is a problem that we support, but it's not as interesting to us as the second problem, which is how do you actually create the graph that represents all the relationships between the different data sources of a company. Right? Like, it's less interesting to us to know what what linear means in a in a company's context or or Jira or Buganizer or something like that. Mhmm. It's how does that actually fit into the whole, you know, process of shipping a feature? Yep. Right? What happens after Jira? [00:17:04] S1: Yep. [00:17:05] S2: Where does the collaboration for human work happen? Is it Slack or something else? What does good collaboration look like? When you're reviewing a, you know, feature that's about to be released, what does that review process look like? And when you're releasing the feature, what does the feedback loop look like if there's a a big error, and you wanna actually find out that error, maybe push a fix automatically. Right? Like that whole set of different steps and tying them together and making sure the knowledge graph for that company's context is is actually something that represents relationships accurately is the harder part.
00:17:05 · Adarsh Hiremath, Dev Tagare
[00:17:05] S2: Where does the collaboration for human work happen? Is it Slack or something else? What does good collaboration look like? When you're reviewing a, you know, feature that's about to be released, what does that review process look like? And when you're releasing the feature, what does the feedback loop look like if there's a a big error, and you wanna actually find out that error, maybe push a fix automatically. Right? Like that whole set of different steps and tying them together and making sure the knowledge graph for that company's context is is actually something that represents relationships accurately is the harder part. [00:17:33] S1: And and in this, again, we we come back to the question of, like, time is not continuous. As an example, I may pick up a task today, I may do it over the next five days, abandon it randomly, pick it up in two weeks again, so on and so forth. And in your process of discovery, maybe you find this as a high impact task to have actually done. So help us understand how do you think about that process of discovery and then tying it back to disaggregated steps, if you will. That models to your point earlier, basically an agent trajectory. We're moving from agent is solving for a goal to agent is sticking to a trajectory to get to the goal side of paradigm. How do you think about that?
00:17:33 · Dev Tagare, Adarsh Hiremath
[00:17:33] S1: And and in this, again, we we come back to the question of, like, time is not continuous. As an example, I may pick up a task today, I may do it over the next five days, abandon it randomly, pick it up in two weeks again, so on and so forth. And in your process of discovery, maybe you find this as a high impact task to have actually done. So help us understand how do you think about that process of discovery and then tying it back to disaggregated steps, if you will. That models to your point earlier, basically an agent trajectory. We're moving from agent is solving for a goal to agent is sticking to a trajectory to get to the goal side of paradigm. How do you think about that? [00:18:19] S2: Yeah. So maybe breaking apart that specific thing of sometimes humans will do a task in one sitting, they might wait five days. There's a couple of reasons for why a person might do that. The one is that the actual task is not that important. Right? And you need to reprioritize and do something different, or you just don't have the bandwidth. Or or or the second thing is that you just actually didn't get around to it, even though it is important, which actually is also a bandwidth problem. The beautiful part of agents is that they don't have these constraints. Yeah. Right? They can prioritize. They can operate at exceptionally high throughput. So when there are a bunch of different tasks that can be taken [00:18:55] S1: Right.
00:18:55 · Dev Tagare, Adarsh Hiremath
[00:18:55] S1: Right. [00:18:55] S2: The agent should be able to do them all simultaneously, assuming they're high value tasks that would that would be important in an enterprise context. The piece that I think is unique and has not been solved is the continuous learning and updating. Right? How do you make sure that the agent doesn't have, like, a long feedback loop before it understands that it made a mistake or even understands that it could be doing better. Right? Yep. That continuous learning loop involves a bunch of different things. It involves triaging tickets to humans who know what good looks like, if that's appropriate. In some cases, it involves automatically understanding its failure from LLM as a judge. In some cases, you could actually even look at the telemetry of the API call, and then you can have that automatically improve the agent. So that continuous learning as a problem space, I find, is is very, very interesting and unsolved.
00:19:39 · Nancy Wang
[00:19:39] S3: You know, when I would say humans are doing workflows, right? You pause because you might not have actually all the context, even of Exactly. The Or like, hey, you realize I need to tag in so and so because they own this particular work stream. For the agent itself, right, because oftentimes we see, you know, hallucination. For example, internally, we have a no code agent builder that allows our GTM teams to pull in, like, context from across Drive, right, Slack, Notion, so on and so forth, for their presentations. For fields, oftentimes, that we find that the agent doesn't know, it very confidently just makes up data. Right? So, do you prevent that, where a natural human might pause, but the agent might just continue and finish that task?