Continual Learning: How AI Agents Get Better With Every Use | Arjun Karanam, Trajectory
频道: Sequoia Capital
视频: https://www.youtube.com/watch?v=eYrMF9Cht8A
原文语言: en
统计: 共 32 轮 · 主持人 2 · Arjun 20 · 提问者 8
[0:00] 主持人
Okay, last talk to bring us home, we have Arjun and Ronak, co-founders of Trajectory. They're doing a bunch of interesting research around what I think is one of the hotter topics, continual learning. Please welcome Arjun to the stage.
好，最后一场分享给大家收个尾。我们请上 Trajectory 的两位联合创始人 Arjun 和 Ronak。他们在做一堆很有意思的研究，围绕的是我认为当下最热的话题之一——continual learning。掌声欢迎 Arjun 上台。
[0:12] Arjun
Cool. What's up? How's it going everyone? Thank you so much. I'm really excited. I know I'm close to the last one, so hopefully there's like remaining attention span. I'm here. Ronak is also back there. So we're together. But let's get started. We're Trajectory and we're building the platform for continual learning. Who are we? I think we're working on like a really really cool mission and that's the most fun part. The second most fun part is I get to do this with my two best friends. Ronak over here worked at One Surf and Train Speed 1 before this and and Michael who worked on really cool robotics stuff at at Deep Mind. And what I wanted to start off with was the world view that we have and how that shapes why we think this problem is important. And so the world view that we have is that and it's not a hot take, we are living in an incredible time in human history here. Every single week it seems like another model is coming out that is leapfrogging the last one. It's undeniable that models are getting better and better. But we think that they're getting better on one axis and that is IQ. These models are smarter and smarter but always feels like when you're talking to them, it's their first day on the job. And so you take this analogy if you have a Terrence Tao in your pocket, that's great.
好，大家好吗？非常感谢。我挺兴奋的。我知道我这场排得比较靠后，希望大家还剩点注意力。我人在这儿，Ronak 也在后面那边，我们俩都到了。那就开始吧。我们是 Trajectory，我们在做 continual learning 的平台。我们是谁？我觉得我们在做一件特别酷的事，这是最好玩的部分。第二好玩的是，我能跟我两个最好的朋友一起做这件事。这位 Ronak，之前在 Windsurf，训练过 SWE-1；还有 Michael，之前在 DeepMind 做机器人，都是非常酷的东西。我想先从我们的世界观讲起，以及它怎么塑造了我们为什么觉得这个问题重要。我们的世界观是——这其实算不上什么惊人观点——我们正活在人类历史上一个不可思议的时代。几乎每周都有新模型出来，一个把上一个甩在身后。模型越来越强，这没法否认。但我们认为它们变强只是在一个轴上，那就是 IQ。模型越来越聪明，可你跟它们对话的时候，总感觉它今天是上班第一天。打个比方，你口袋里揣着一个 Terence Tao，那当然很棒。
[1:33] Arjun
But Terrence Tao day one at an accounting firm is probably not the best accountant there. But give him a few years or honestly maybe even like a few days, he'd probably be really good. And so we have this orthogonal axis of experience that is much more important than we think in conjunction with IQ. And that's what we're calling the experience gap and that is what we want to close as a company. Um And then but the question is like how do you how do you close that? Where does experience come from? Well, it's already out there right now. There's so many hundred million is a number, but so many tokens out there that are being generated by these agents. You view them or sometimes you don't even view them and they get thrown away. But this is all real work that they're doing and then people are acting upon them and then it's being thrown away. And so our take as a company as Gabe mentioned like our opinionated take is that uh this is a signal that we should be learning from. And this is also how humans get better. And in classic AI fashion like if the if humans do it probably a good way to mental model off of. Uh and so today's agents they're still slow, expensive, error prone over time. You implement them, they probably don't get better.
但 Terence Tao 到一家会计事务所上班的第一天，多半不会是那儿最好的会计。可给他几年——说实话可能只要几天——他大概就会做得非常好了。所以除了 IQ，还有一条正交的轴叫「经验」，我们认为它跟 IQ 配合起来的重要性被大大低估了。这就是我们说的 experience gap（经验鸿沟），也是我们这家公司想去填平的东西。问题是，怎么填？经验从哪儿来？其实它现在就已经在那儿了。几亿这个数字都不够形容——此刻正有海量的 token 被这些 agent 生成出来。你可能扫一眼，可能压根不看，然后它们就被扔掉了。但这些都是它们实实在在干的活儿，人们还基于这些结果去行动，然后就这么被丢掉了。所以像 Gabe 刚才提到的，我们公司的立场是：这是信号，我们应该从中学习。人也正是这么变强的。按 AI 圈的老传统——人类如果是这么干的，那多半是个不错的心智模型可以拿来借鉴。今天的 agent 依然慢、贵、容易出错，而且你把它部署上去之后，随着时间推移它基本不会变得更好。
[2:44] Arjun
We want to imagine what learning agents look like where as they are used by people, they get better and better over time. Uh and tangibly this has two good benefits. Right away it gets you much faster, better, cheaper models. Uh but more excitingly it gets you to this goal of systems that compound with use. And so I want to start off by talking about how we're approaching this problem uh and then uh some interesting stuff for you guys at the end. And so where it starts off and this kind of dovetails off of what Harrison was talking about before is it starts off with traceability. We need to first capture these interactions, capture this experience that's being thrown away. So that's step number one. Uh And then step number two and this is where where we start getting to our research. The way we view the company we're building is we're building a product to allow for continual learning, but we're doing cutting edge research under every single one of them to make this possible. And so you have all these interactions and the next thing we're building is this idea of a like a model spec. This idea that like you need to have a way to define what do I want my agent to do and have the agent learn against that. So we're doing really cool research here on how to extract user interactions, turn that into reward, extract traces, and turn that into exact specs of what you want your agent to do.
我们想象的是 learning agent 长什么样：随着人们不断使用，它会越来越强。具体来说这有两个好处。最直接的，是让你拿到更快、更好、更便宜的模型。更让人兴奋的是，它把你带向一个目标——系统会随使用而复利增长。所以我想先讲讲我们怎么解这个问题，最后再给大家一些有意思的东西。起点跟 Harrison 刚才讲的正好能接上，就是 traceability（可追溯）。我们得先把这些交互抓下来，把那些被丢掉的经验捕获住，这是第一步。第二步就开始进入我们的研究了。我们做公司的方式是：产品上做一个让 continual learning 成为可能的平台，但底下每一层我们都在做前沿研究。所以你有了这一堆交互，接下来我们做的是一个叫 model spec 的东西——你需要有办法定义「我到底想让我的 agent 做什么」，然后让 agent 照着这个去学。我们在这块的研究很有意思：怎么从用户交互里抽出 reward，怎么从 trace 里抽出你到底想让 agent 做什么的精确 spec。
[4:04] Arjun
And then okay, cool. You have what you want your agent to do. What do you do with that? Well, there's two surfaces. One is the models, right? You want your models to to learn from actual interactions here. And so we're doing really cool research on algorithms like SDPO, and and and and things along those lines where we're using RL to uh take these full long traces and improve models over time. But models are not the only thing. You have harnesses as well. And we're doing a lot of really cool research on when you have good feedback from people, does that go to a harness, or does that go to a model? An example here is if it's like a fact, like this person this like this company has been delisted, right? You probably don't want to train that knowledge into the model. It's probably context that should be available to the harness. And so that is how kind of how we're splitting things, and that is the product that we're building. Allowing you to go from real interactions all the way to specs of what you want your agent to get better at, to better models, and then better harnesses, and then being able to deploy those those models right away, and then start using our product. And so that's the world we want to get to.
好，你有了「想让 agent 做什么」，然后拿它干嘛？有两个落点。一个是模型——你希望模型能从真实交互里学。所以我们在做 SDPO 这类算法的研究，用 RL 把这些完整的长 trace 吃进去，随时间把模型变好。但模型不是唯一的落点，你还有 harness。我们在这块也做了很多有意思的研究：当你拿到人给的好反馈时，它该进 harness 还是该进模型？举个例子，如果它是一条事实，比如「这家公司已经退市了」，你多半不想把这个知识训进模型里，它更像是应该提供给 harness 的 context。这就是我们大致的切分方式，也是我们在做的产品：让你从真实交互一路走到「你想让 agent 在哪些地方变好」的 spec，再到更好的模型、更好的 harness，然后立刻把这些模型部署出去，接着继续用我们的产品跑起来。这就是我们想要抵达的世界。
[5:14] Arjun
If you have built agents, which I'm sure most of you guys have, you probably know that there are so many things that go wrong with every single part of these, and then how every single one of these is an unsolved problem. And so the way I wanted to frame the rest of this is I'm going to imagine that I am like straight from Aladdin, and I have like four wishes. And what are those wishes that I would ask to like make the agent ecosystem better. And so like we have agents today, we have this like gap, and there's a level where agents get to and after that we can really see the exponential effects of continual learning. And what are those in I'll walk through them step by step with the same four categories I talked about. What we would like to see on the traceability side. What we would like to see on the eval side, what we'd like to see on the harness side, what we'd like to see on the model side. So let's start with wish number one. Uh and how can companies get there faster? Uh traceability. So two two sub wishes here. Number one is around tracing the entire tree of what happens, uh including sub agents. Uh what a lot of companies do is they trace the main action that is happening, but they throw away tool calls or sub agents that are made. Um and that makes it very hard to learn from the whole thing that you've done. So that's wish number one. Uh sub wish number two uh is probably the most important in here and that you should build your product around how you can both capture this interaction data, but also elicit the right amount of feedback uh so that you can then capture it. Um and a really interesting thing here is I think the level one way of thinking about this is uh okay, let's have like a thumbs up, thumbs down and like let's capture that. And that sounds amazing in theory uh but it's incredibly noisy and if you've used any coding agent, you know you kind of just like accept everything that the agent does.
如果你搭过 agent——我相信在座大部分人都搭过——你肯定知道每一个环节都有一堆会出错的地方，而且每一个都还是没解决的问题。所以接下来我想这么讲：假设我是《阿拉丁》里出来的，我有四个愿望，那我会许什么愿来让整个 agent 生态变好。今天我们有 agent，中间隔着这么一道 gap，agent 得先到某条水平线，过了那条线我们才能真正看到 continual learning 的指数级效应。那这几个愿望是什么？我按前面讲过的同样四个类别一个个说：traceability 上我们希望看到什么，eval 上希望看到什么，harness 上希望看到什么，模型上希望看到什么。先说第一个愿望——公司怎么才能更快到位——traceability。这里有两个小愿望。第一，把整棵树都 trace 下来，包括 sub agent。很多公司只 trace 主流程发生了什么，但把 tool call 和 sub agent 全扔了，这就导致你很难从整件事里学到东西。这是第一个愿望。第二个小愿望可能是这里最重要的：你的产品设计本身就要围绕「怎么既捕获这些交互数据，又能引导出恰当分量的反馈」来做，这样你才捕得到。这里有个挺有意思的点：我觉得第一层的想法是「行，加个点赞点踩，把它记下来」。理论上听着很美，但它噪音大得离谱。如果你用过任何编程 agent，你就知道你基本上是 agent 干啥你都点接受。
[7:09] Arjun
And it's only like five commits later that you're like, oh crap, like this broke everything. Let me go and undo that. And so it's the corrective behavior, the like the edits, the undos and the retries that both need to be like elicited from the user, but also captured. So that's my wish on traceability. The second wish is around evals. And the mental model to use here is your the product that your user uses should be as similar as as close to what the eval is done in in for wise should be as close to where the training is done in. In an ideal world, these are all the same. There's no difference between these. And so this means that evals are, you know, drawn from traffic, how people are actually using your product, both how they're using it now and the things that they're requesting on the frontier that might not be possible, all super helpful. Second is making every task roll outable. This is a pretty big infra challenge, uh but it's this idea that hey, if a user has done one thing, can I somehow replay what they what the user has done? And that is an incredibly helpful boon to to moving forward in this agentic era. And then lastly, this goes with number one, grading through like the real harness, uh not making a variation of the harness, but like grading the real harnesses that people use in production.
然后要等到五个 commit 之后你才反应过来：完蛋，这把什么都搞坏了，我得回滚。所以真正有价值的是那些纠正性行为——编辑、撤销、重试，这些既要从用户那儿引导出来，也要被捕获下来。这是我在 traceability 上的愿望。第二个愿望是关于 eval 的。这里能用的心智模型是：你用户用的那个产品、你做 eval 的环境、你做训练的环境，三者要尽可能接近。理想世界里它们就是同一个东西，中间没有任何差别。这意味着 eval 应该是从真实流量里抽出来的——人们现在实际怎么用你的产品，以及他们在能力边界上提的那些当下还做不到的需求，这些都特别有用。第二点是让每个任务都能 rollout。这是个挺大的基础设施挑战，但核心想法是：用户做过一件事，我能不能把他做过的这件事重放一遍？在 agent 时代，这对往前推进是极大的助力。最后一点跟第一点是扣在一起的：要在真实的 harness 里打分，不要另做一个 harness 的变体，就用生产环境里大家真正在用的那个 harness 来评。
[8:32] Arjun
So that is wish number two. Uh wish number three, getting into the harness, uh pretty important. Uh number one is and and I think the mental model to use here is I think a lot of products have built their harnesses around models that existed a year a year and a half ago. Where the primary function of the harness was to prevent the agent from doing bad things. Which was really good when agents would randomly break or have misformatted outputs, but now we're very much in a let the agents cook world. Uh and so I think I would view building the harness as not enforcing specific flows that need to be done, but more what are the primitives that your product has, whether that's search tools or private information, and then viewing the agent as orchestrating those primitives and then letting it do that. So that's sub wish one. Uh sub wish two, and this is how we're building our product as well, is making the agent interface as close to the user interface as possible. Uh in an ideal world, every single thing that I can do on your UI, your agent can do via tool call as well. Uh I think that is a world that makes training and continual learning so so much easier. Uh and then lastly, this is a small one, uh making tool responses informative. I think it's easy to be like, "Okay, I just want to know the state." Like let's say I'm like calling a search tool. I want to know the state. I want to see if it worked. Uh and if it's writing to a a DB or something, it just writes to it.
这就是第二个愿望。第三个愿望进到 harness，这块挺重要。第一点，我觉得这里能用的心智模型是：很多产品的 harness 是围绕一年到一年半前的模型搭的，那时候 harness 的主要职责是防止 agent 干蠢事。在 agent 会随机崩掉、输出格式乱套的年代，这么做非常对。但现在我们已经明确进入了「放手让 agent 去跑」的时代。所以我会认为，搭 harness 不该是去规定必须走哪些固定流程，而是想清楚你的产品有哪些 primitive——搜索工具也好，私有信息也好——然后把 agent 看成是在编排这些 primitive，放手让它去编排。这是第一个小愿望。第二个小愿望，也是我们自己产品的做法：让 agent 的接口尽可能贴近用户界面。理想情况下，我在你 UI 上能做的每一件事，你的 agent 都能通过 tool call 做到。我觉得那样的世界会让训练和 continual learning 容易太多了。最后一点比较小：让 tool 的返回足够有信息量。人很容易觉得「行吧，我就想知道状态」。比如我调一个搜索工具，我想知道状态，想确认它成功了；如果是往数据库里写，那它写进去也就完事了。
[10:01] Arjun
And what we see often is like the response from that tool call will be uh done. Finished. That sounds like good in theory, but from an agent's perspective it's like incredibly confusing. And if you're trying to train or learn off of that, there's like no signal to learn off of. Right? You have no idea what actually was written. You have no idea what was actually was read. And so, that's the last sub wish under the harness is to make your tool responses really informative. So, that's wish number three. I think like canonically I only get three wishes, but let's imagine I have four. Uh and that's around the models. Uh and we've talked about that a lot today. Uh and that is uh I wish it was as easy as like just switching to an open weight model, but if you've tried that, you've probably seen there's 50 other considerations from security to safety to uh you know, different access provisioning stuff. So, start getting comfortable running on open weights cuz that is what unlocks the door to owning owning your weights and then continually improving on top of them. Um and then Gabe also talked about this this idea that like model routers are uh going to probably play a really large role in routing intelligence to the exact capability of the task they're at.
但我们经常看到的是，这个 tool call 返回的就是一句「done」「finished」。理论上听着挺好，可从 agent 的视角看，这简直一头雾水。而且如果你想拿这个去训练、去学，那压根没有信号可学。对吧？你根本不知道实际写进去了什么，也不知道实际读到了什么。所以 harness 下的最后一个小愿望，就是让你的 tool 返回真正有料。这是第三个愿望。按惯例我应该只有三个愿望，但咱们就假设我有四个。第四个是关于模型的，今天大家已经聊了不少。那就是——我真希望事情简单到「换个 open weight 模型」就行，但如果你试过，你多半发现后面还有另外五十件事要考虑：从安全到合规，再到各种权限配置。所以，开始习惯在 open weights 上跑吧，因为那才是打开「拥有自己的权重」这扇门的钥匙，然后你才能在上面持续改进。另外 Gabe 也讲过这个点：model router 很可能会扮演相当重要的角色，把智能路由到任务真正需要的那种能力上。
[11:15] Arjun
And so, experimenting with with routers is another wish I would wish cast into the world. And so, that is the high level here. And so, I think like when we work with companies, the first thing we often do is do an audit of of these and like where things are. And for the most part there's a lot of work to be done to get here and so we work with them to build it out and it would be my dream if we go to a company and they have a lot of this built out. Granted a lot of the work we're doing is also knowing that a lot of this we can't expect and so we're doing a lot of cool research on, you know, assuming that nobody has Evals, somebody doesn't have Evals, how can we overcome that? Or assuming that the traceability is off, like how can we learn from the signals that are there? And so that is that that is my my goals for the for the universe here. Uh just want to quickly go over uh I think this is like a very very core research and product problem and so we're really excited with the team we've put together but I think more importantly and then two of the folks who are here today uh we're working with extremely frontier customers. The folks who are really really pushing the boundaries of what agents can do in production both today and tomorrow.
所以「多拿 router 做实验」是我想向这个世界许下的另一个愿望。这就是整体的大图。我们跟公司合作时，第一件事往往是把这几块做一次盘点，看看现状在哪儿。大多数情况下，要做到这个程度还有很多活要干，所以我们会陪着他们一起搭。要是哪天我们走进一家公司，发现这些他们都已经搭好了，那就是我的梦想了。当然，我们做的很多工作也是建立在「这些没法指望」的前提上，所以我们在研究：假设一家公司完全没有 eval，我们怎么绕过去？假设 traceability 根本没开，我们怎么从现有的那点信号里学？这就是我对这个领域的期望。我还想快速过一下——我觉得这既是一个极核心的研究问题，也是一个极核心的产品问题，所以我们对组起来的这支团队非常兴奋。但更重要的是，我们在和最前沿的客户合作，其中两位今天也在现场，就是那些真正在把 agent 在生产环境里的能力边界往前推的人，今天的边界和明天的边界他们都在推。
[12:31] Arjun
Uh and so really excited to be working with them and another thing I want to show is we have Oh, that's my email. Uh we have a beta and so a really really core belief we have is that this capability, this ability to own your own intelligence, should not be something that you need to consult away a lot of the times. It should be expertise you build on your own because of how important it is for your product. Um and what that means for us is we want to build a product that empowers any company to own their own models and mainly get to continual learning from real interactions and so here's some some cool screenshots from our product. I'm I don't know if Nico is here but I'm supposed to show this Nico later today so this is like a sneak peek. Uh but here's Harvey. Here's the lab benchmark that that Gabe was talking about. Really really easy to import. Um training a model is incredibly easy. We're extremely proud of the interface we've put together here cuz if you've post trained, you know, there's like 50 million things that go wrong, 50 million knobs to turn, and it's all researcher intuition, trademark asterisk, whatever that means. And what we've tried to do is take all of how our researchers post train their models and use an agent behind the scenes to take care of most of those knobs and just only expose the ones you need to know to make post training as easy as possible.
所以特别高兴能和他们一起做事。另外我想给大家看的是——哦，那是我的邮箱。我们有个 beta。我们有一个非常核心的信念：「拥有自己的智能」这件事，不应该是那种你花钱请咨询公司帮你做掉的东西，它应该是你自己长出来的专业能力，因为它对你的产品太重要了。这对我们意味着，我们要做一个能让任何公司都拥有自己模型的产品，核心是从真实交互里做到 continual learning。这里放几张我们产品的截图。我不知道 Nico 在不在现场，我本来是说好今天晚点给 Nico 看的，所以这算提前剧透。这是 Harvey，这是 Gabe 刚才提到的那个 lab benchmark，导入非常方便。训练一个模型也极其简单。我们对这套界面特别自豪，因为如果你做过 post-training 你就知道，有五千万个地方会出问题，五千万个旋钮要调，全靠所谓的「研究员直觉」（注册商标，随便你怎么理解）。我们想做的是把研究员做 post-training 的整套经验沉淀下来，用一个 agent 在后台帮你搞定绝大部分旋钮，只把你真正需要知道的那几个暴露出来，让 post-training 尽可能简单。
[13:54] Arjun
So, really easy to train a model, see how the model's doing, evaluate, compare it, uh see if it's better than the model you were using before, and then deploy it. And so, in terms of actual work you put in, and that was not waiting for the model to train, probably like 15 minutes. And so, that is the world we're we're trying to get to to empower all these companies and hopefully more folks in this room. Uh So, yeah, that's our goal. We want to every company to own its own experience layer, and we're excited to build that alongside of some of y'all here, hopefully.
所以训练一个模型很容易，看模型表现、做评测、做对比，看它是不是比你之前用的模型好，然后部署。真正要你投入的操作量——不算等模型训完的时间——大概也就 15 分钟。这就是我们想抵达的世界，去赋能这些公司，也希望能赋能今天在座的更多人。所以，这就是我们的目标：让每一家公司都拥有自己的经验层（experience layer），也希望能和在座的一些朋友一起把这件事做出来。
[14:27]
[applause]
[掌声]
[14:28] 提问者
Thank you. It's a great talk. You know, I wonder like, you know, uh when you talk about continual learning, like uh what do you think about their trainable object um between model weights, harness, tools, application layers?
谢谢，讲得非常棒。我想问一下，你讲 continual learning 的时候，在模型权重、harness、工具、应用层这几个可训练对象之间，你是怎么看的？
[14:43] Arjun
Yeah.
嗯。
[14:43] 提问者
Uh how how do you think of them? How do how do you prioritize?
你怎么看待这几者？又怎么排优先级？
[14:46] Arjun
Yeah, I think I like a fun fun phrase is that if you ask like six researchers what continual learning is, you're going to get like seven answers, probably. Uh you can be very pure about this and be like, "Oh, it needs to be like a human, and it is just the weights that need to be adopting in real time, one shot." But, the way we view it is that the intelligence that your product is run off of is a system. It's a system that has many components, and true continual learning is something that optimizes across that system and optimizes based on what parts of the system needs to be updated uh for the information that you're learning. Like a mental analogy that I have is like when you are you know, saving things, right? We no longer think of like where in your RAM or where in your hard disk to save. That's the level that's abstracted based on what makes the most sense. We think about models versus harnesses versus context in the same way. It feels wrong that we're having to make the decision off of very little priors what to update based on what. This is a scientific problem that can be solved. Let's do that and then let's abstract that away. Cool. Oh, yes. Yes.
有句挺好玩的话：你去问六个研究员什么是 continual learning，大概会得到七个答案。你可以特别较真地说：「它必须像人一样，只有权重在实时地、一次性地更新才算。」但我们的看法是，你产品所依赖的那个智能是一个系统，这个系统有很多组件，真正的 continual learning 是能在整个系统层面做优化的东西——根据你要学的这条信息，判断系统的哪个部分该被更新。我常用的一个类比是存东西：今天我们已经不会去想这份数据该存在内存的哪儿、硬盘的哪儿了，那一层已经被抽象掉了，系统会按最合理的方式安排。我们看待模型、harness、context 也该是同一回事。现在这样，在几乎没有先验的情况下，让人去决定该更新什么、依据什么更新，感觉就是不对的。这是个可以被解决的科学问题，那我们就把它解掉，然后把它抽象走。好。哦，请讲。
[16:07] 提问者
Um, some of the speakers today talked about, you know, the importance of not training on your customer data.
今天有几位讲者提到，不要拿客户数据去做训练，这一点很重要。
[16:12] Arjun
Yeah.
嗯。
[16:12] 提问者
And I imagine like a big part of continual learning is you are learning from all these interactions. Um, how are you thinking about things like differential privacy? Like how do you actually make your system better with I imagine most of these application companies have customer specific data arrangements that make that very hard.
我猜 continual learning 很大一部分，就是从这些真实交互里去学。那你们怎么看 differential privacy 这类问题？我想大多数应用层公司跟客户之间都签了专门的数据使用协议，这会让事情变得很难——你们到底怎么在这个前提下把系统越做越好？
[16:27] Arjun
100%. I think that's an incredibly important problem. Uh, actually before this I was at Apple and this is a problem we worked on quite a bit as well. Um, and there's a lot of clever things you can do where you're not actually training on customer data, you're instead sampling distributions from your customer data and then synthetically generating your own and doing kind of like a mental models maybe like a cryptographic thing where you're like comparing those distributions to see is my data actually on distribution, but you're not training directly off of the customer data. Um, and so that that's totally right. There's a lot of interesting work there and it's like very central and already with our customers we're doing some interesting and and fun things in order to uh, to get over that. Yes.
完全同意，这是个极其重要的问题。其实我来做这件事之前在 Apple，我们当时也在这个问题上花了不少功夫。有很多挺聪明的做法，可以让你压根不用直接拿客户数据去训练：你从客户数据里采样出分布，然后自己合成生成一批数据；再用类似密码学的那套思路去比对两边的分布，判断我这批数据是不是落在真实分布上——但全程都没有直接拿客户数据做训练。所以你说得完全对，这块有很多有意思的工作，而且非常核心。我们跟现有客户已经在做一些挺有意思、也挺好玩的尝试，来绕过这个限制。
[17:13] 提问者
Hey, I'm Sid. I'm from Finch. Um, thank you. Um, I think episodic memory plays uh, a role in continual learning to some extent. For example, like if a user does something on our platform that corrects what an agent did before, we might want to update an agent's behavior. Does Trajectory have like an opinionated take about that in the platform?
你好，我叫 Sid，来自 Finch。谢谢分享。我觉得 episodic memory 在 continual learning 里多少也起了作用。比如用户在我们平台上做了个动作，纠正了 agent 之前的做法，我们可能就想据此更新 agent 的行为。Trajectory 在产品层面对这件事有没有比较明确的主张？
[17:33] Arjun
Yeah, I think Okay, on episodic memory and I'm just going to repeat here. On episodic memory and like what if I'm thinking about this correctly, how different types of feedback impact what is learned?
好，关于 episodic memory——我先把问题复述一遍给现场的人听。关于 episodic memory，如果我理解没错，问的是：不同类型的反馈，会怎么影响模型最终学到的东西？
[17:47] 提问者
Yes.
对。
[17:47] Arjun
Yes, okay. So many thoughts. One way to look at this is there are two different types of signals. There are signals that are just something went wrong. Like for example, if somebody like flames the agent says like, "You're really bad." But he doesn't follow anything up with that or just thumbs down or like drops off the session. Those are cases where you know something went wrong, but not what right looks like. Then you have things like the agent retried and then it got to the correct solution or you corrected it by saying this is a difference. And so that is like a difference we make. In the latter case, we're very, very confident about the reward associated with the with the output and and what we're assigning. With the former, we know to penalize that behavior, but we don't necessarily say like there's like a correct right answer. That's one way of looking at it. Another way of looking at it is there are different like there's like a hierarchy of like how pertinent the information is people-wise. There's some information that's probably globally accurate, globally true. Things like a tool call repeatedly failing when trying to call a certain tool. That's probably relevant to everybody.
好，这块我想法挺多的。一种看法是：信号分成两类。一类只告诉你「出错了」。比如有人对着 agent 一顿喷，说「你太烂了」，但后面没有任何补充；或者只点了个踩；或者干脆退出会话不玩了。这些情况下你知道出了问题，但不知道对的应该长什么样。另一类是：agent 重试了一遍，最后拿到了正确的解；或者你直接纠正它，告诉它正确的应该是什么。这两类我们是区别对待的。后一种情况里，我们对该给这个输出打多少 reward 非常有把握；前一种我们只知道要惩罚这个行为，但说不出正确答案是什么。这是一个切法。另一个切法是：信息的适用范围是有层级的。有些信息大概率是全局成立、对谁都对的，比如调某个工具时 tool call 反复失败——这多半对所有人都适用。
[18:58] Arjun
And so this should be trained into the model and the model should get better at learning this tool. A certain user is like, "I never want to use the sub agent. Please, please, please don't do it." That's probably not something you should train into the model and leave to the context, and what we were really excited about is this is the stuff Harvey was talking about where this is actually something that will probably happen on a per org basis, or per customer even further, a per customer basis. And so, that's another way to think about it as well. So, the hierarchy of like what the feedback is pertinent to.
这种就该训进模型里，让模型把这个工具用得更熟。但如果某个用户说「我永远不想用 sub agent，求你千万别用」——这个大概率就不该训进模型，交给 context 处理就行。我们特别兴奋的一点是，这正是 Harvey 刚才讲的那类东西：它很可能是落在每个组织的层面上，甚至再细一层，落在每个客户的层面上。所以这是另一个思考角度：按反馈的适用范围来分层级。
[19:30] 提问者
Got it. Thank you.
明白了，谢谢。
[19:31] Arjun
Yeah. Yes.
嗯，好。
[19:36] 提问者
Thank you. I think one of the themes of today has kind of been this idea that there are some, you know, tasks or part of your product where you kind of want to maybe like experiment more, go cheaper, add in more open source stuff, right? And then other areas where kind of being at the frontier and using a closed model or a closed framework is more useful. I guess like extending that analogy to continual learning, are there tasks or workflows, or like what is the shape of problem where something like this you've kind of seen really matters a lot, right? Versus where like just using, you know, statically trained frontier model or even a statically trained open source model, and then like you know, using the harness to inject relevant information into the context is sufficient to kind of achieve what the user usually wants to achieve.
谢谢。我觉得今天反复出现的一个主题是：产品里有些任务或环节，你会更愿意多做实验、用更便宜的方案、多塞点开源的东西；而另一些地方，站在前沿、用闭源模型或闭源框架反而更划算。把这个类比延伸到 continual learning——有哪些任务或工作流，或者说问题长成什么形状的时候，你们发现 continual learning 是真的关键？反过来，什么时候直接用一个静态训练好的前沿模型，甚至静态训练的开源模型，再靠 harness 把相关信息注入 context，就已经足够满足用户通常想达成的目标了？
[20:20] Arjun
Yeah. This is a great question. I think the level one answer is I think for the most part most tasks work for continual like, but the ones I'm most excited about are tasks that are on the frontier. The mental model I have that that we have for AI progress in in in a way is people will ask models, people will ask products with the expectation of what they think the product can do. Uh and maybe they'll ask at the edge of what the product can't do, and then they'll see it fail, or they'll see it do something wrong, and they'll retreat back and be like, oh, it's it wasn't good enough. I can't use it for this. I think like a great example is like, would I even dream of typing in some of the like the whack query that give the cursor two years ago, like definitely not. I was like way lower than this, but then over time I started querying it where the models were getting better and I was like, "Okay, maybe we can do bigger and bigger things." What we're most excited about is the way that has happened is like models have generally gotten better and we've been targeted pushing in certain directions. Uh but what we're seeing with some of our customers are cases where users ask for things, the model can barely do it, but then during training it learns how to do it and then the user can now do this thing they couldn't do before. That chain is what's really exciting about continual learning of like really pushing that frontier of what's possible based on what the user tries and cannot do.
这问题问得好。第一层答案是：大部分任务其实都适合上 continual learning，但我最兴奋的是那些正好处在能力边界上的任务。我们对 AI 进步的心智模型大概是这样：人们向模型、向产品提要求时，是按自己以为这产品能做到什么来提的。偶尔他们会往产品做不到的边界上试一下，然后看到它失败、或者做错了，就缩回去了——「哦，还是不够好，这事儿用不了它」。一个很好的例子是：两年前我会想到把今天丢给 Cursor 的那些离谱 query 打进去吗？绝对不会，那时我的预期比现在低得多。但随着模型越来越好，我慢慢就那么用了，心想「行，好像可以干越来越大的事了」。以前这件事发生的路径是：模型整体在变强，我们再有针对性地往某些方向推。而我们在一些客户那儿看到的是：用户提了个要求，模型勉强能做，然后在训练过程中它把这件事学会了，于是用户现在能做成一件他以前做不成的事。这条链路才是 continual learning 真正让人兴奋的地方——根据用户试过但没成的那些事，实实在在地把可能性的边界往外推。
[21:47] 主持人
Awesome. Thank you, Arjun. I think we're out of time. Uh really appreciate you doing this. It's awesome. Thank you.
太棒了。谢谢你，Arjun。时间差不多到了，非常感谢你今天来讲这些，真的很精彩。谢谢。
[21:52]
[applause]
[掌声]