When to Build Your Own Agent Harness | Harrison Chase, LangChain

频道: Sequoia Capital
视频: https://www.youtube.com/watch?v=HI2q3ci3Iuc
原文语言: en
统计: 共 23 轮 · 主持人 1 · Harrison 18 · 提问者 2

[0:00] 主持人

Harnesses. I think this is a very important topic. A lot of you are thinking through building your own harnesses right now. Um, I'm very excited to introduce Harrison. I first noticed Harrison on Twitter in 2022, back in the GPT-3 era. And Harrison was one of the first people thinking about, "Okay, we have these models. How can we build an entire harness around them so that they're not just um auto-complete uh tasks, but that they start acting as virtual collaborators or agents?" Um and Harrison, like the ecosystem has grown so much since 2022, and I've seen you you also grow a lot in terms of how you think about building agents, building harnesses, how to eval them, etc. Um so, I'm very excited to have you talk today. I think the talk's going to be both about harnesses and evals. And then format again will be 15 minutes or so of presentation content, uh 15 minutes of Q&A. Thanks for joining us, Harrison.

harness。我觉得这是个特别重要的话题，在座很多人现在都在琢磨要不要自己搭 harness。非常高兴请到 Harrison。我最早注意到 Harrison 是 2022 年在 Twitter 上，那还是 GPT-3 的年代。当时 Harrison 是最早一批开始思考这件事的人：我们手里有了这些模型，怎么在它们外面搭一整套 harness，让它们不只是干自动补全那类活，而是真的开始扮演虚拟协作者、扮演 agent？从 2022 年到现在，整个生态长大了太多，我也看到你在怎么构建 agent、怎么构建 harness、怎么做 eval 这些方面成长了很多。所以今天特别期待听你讲。我理解今天的内容会同时覆盖 harness 和 evals。形式还是老样子：大概 15 分钟演讲，15 分钟 Q&A。Harrison，谢谢你来。

[0:57] Harrison

Cool. Um my name's Harrison, co-founder CEO of LangChain. I want to talk about evals and harnesses in the context of kind of owning your own intelligence. So, when we talk about intelligence, we're normally talking about agents. What exactly makes up an agent? At LangChain, we think there's kind of like three main parts. There's a harness that orchestrates a model and some context. And if you're talking about owning your intelligence in general, you probably want to own all three parts of these. And so, owning the model, I'm not going to talk too much about. Lin was here from Fireworks and talking about open weight models and owning that. Uh big part of this is also the ability to switch models. Uh there used to be this concept of kind of being like cloud agnostic and being able to switch clouds uh back in the day. Same thing exists, but for models. You want to be able to switch to avoid lock-in, but also to just use the best model when it's available. Context, you want to own all the the context that your agent uses. Whether that is memory, uh whether that is semantic knowledge, um whether that is previous conversations. You can these can help personalize and guide the agent as it goes along. And then the last bit is the harness, and that's what I really want to focus on.

好。我叫 Harrison，LangChain 的联合创始人兼 CEO。今天我想在「把智能握在自己手里」这个框架下，聊聊 evals 和 harness。我们说 intelligence，通常指的就是 agent。那 agent 到底由什么构成？在 LangChain，我们认为主要是三块：一个 harness，它负责编排一个 model 和一堆 context。如果你真想掌握自己的智能，这三块你多半都想握在自己手里。model 这块我今天不多讲，Fireworks 的 Lin 刚才来讲过开放权重模型、讲过怎么把模型握在自己手里。这里很重要的一点还有换模型的能力。以前有个说法叫「云中立」，能随时换云厂商；现在同样的事发生在模型上。你希望能随时切换，一是避免被锁死，二是有更好的模型出来时马上能用上。context 这块，你希望 agent 用到的所有 context 都归你——不管是 memory、语义知识，还是过往的对话记录。这些东西能帮你把 agent 个性化，并在运行过程中给它引导。最后一块是 harness，这才是我今天真正想聊的。

[2:02] Harrison

So, how how do you how do you really own your harness? What does that even mean? What's the main job of a harness? The main job of a harness is to bring context to the model at the right point in time. And so, it does all the orchestration around the the fixed context, the the dynamic context. It brings it into the context window of the model, shows it something, gets some response, and then does something with that. And so, agents need to do all these different things in order to accomplish their jobs. There's a ton of domain-specific stuff that they need to do as well, but they need to interact with external systems. These external systems, when you interact with them, they emit more context that can get fed back into the agent into the loop. And so, the harness is the thing that really orchestrates all of this together. Agents at [snorts] their their kind of like simplest, when everyone talks about agents, what they really talk about is just an LLM running in a loop calling tools. Um and uh this this is a really simple, but really general architecture. Some request comes in, the the LLM makes some uh generation. That generation may include a tool to call. If it does, you invoke those tools, and you pass that observation back to the LLM. And and this is this is the core architecture behind pretty much every agent out there today.

那到底怎么才算掌握自己的 harness？这话什么意思？harness 的核心职责是什么？核心职责就是：在正确的时间点，把正确的 context 送到模型面前。所以它要围绕固定 context 和动态 context 做全部编排，把它们塞进模型的 context window，给模型看点东西，拿到回复，再拿这个回复去做点什么。agent 要完成任务，得干很多不同的事。里面有大量领域特定的活儿，但共性是它得跟外部系统交互。而你一跟这些外部系统交互，它们就会吐出更多 context，这些 context 又回流进 agent 的循环里。harness 就是把这一整套编排起来的那个东西。agent 说到最朴素的形态——大家一提 agent，其实说的就是一个 LLM 在循环里不停调工具。这个架构非常简单，但非常通用：来一个请求，LLM 生成一段内容，这段内容里可能包含要调用的工具；如果有，你就去执行工具，再把 observation 回传给 LLM。今天外面几乎所有 agent，内核架构都是这个。

[3:12] Harrison

But they're all different in like slightly different ways. And so, on the left here, this is kind of like uh the the the base core kind of like loop. But there's a bunch of different things that you can do in your particular harness at different stages. And so, this is uh over here, this is So, so we built LangChain, which is a really really base minimal harness, and that's LangChain over here. And then this is Deep Agents. Deep Agents is kind of like our model-agnostic and and more general-purpose version of of Quad Code. And so, it does more things. It connects to file systems. Uh it has skills. It has sub-agents. It's built on top of this really simple harness, but we customize it by using these uh these levers over here. So, you can run particular code snippets before the agent's invoked, before each model call. You can kind of like wrap these model calls, you can wrap the tool calls, and you can and you can customize this core uh simple loop in a lot of really powerful ways just by just by using kind of like small what we call kind of like middleware constructs. There's other ways to customize the harness as well, but this is kind of emerged as uh there's a concept of hooks and plugins in a lot of the coding agents as well, and that's essentially what they're doing. They're taking this base loop that's running, and they're adding little hooks or plugins at various points to let you customize it.

但它们彼此又都有些细微差别。左边这个是最基础的核心循环。可是在你自己的 harness 里，你可以在不同阶段插入很多不同的东西。这边这个是我们做的 LangChain，一个极简的基础 harness，就是这边这个。然后这个是 DeepAgents。DeepAgents 可以理解成我们做的、模型无关且更通用的 Claude Code 版本。所以它干的事更多：接文件系统、有 skills、有 sub-agents。它就搭在那个极简 harness 之上，我们靠这边这些「操纵杆」把它定制出来。你可以在 agent 被调用之前、在每次模型调用之前跑一段特定代码；你可以把模型调用包一层，也可以把工具调用包一层。就靠这些我们叫做 middleware 的小构件，你能把这个简单的核心循环定制出非常强的花样。定制 harness 还有别的路子，但这套写法逐渐成了共识——很多编码 agent 里都有 hooks 和 plugins 的概念，本质上干的就是这件事：拿住那个在跑的基础循环，在各个节点上挂点 hook 或 plugin，让你能改。

[4:26] Harrison

And so, a lot of the stuff that you can do when you can customize, this is all done by that concept of middleware, by just modifying that core loop. So, the agent's still running in a loop, it's still doing that same simple architecture, but through that you can give it access to a sandbox, you can give it access to a file system, you can give it access to sub-agents, you can give it access to to memory, you can have summarizations. So, summarization, if if if we go back to this thing, summarization would come in before the model. Before the model's invoked, you check if the context is too long, and then you summarize it. And so, that that you can add into this core loop through this concept of middleware, same with context offloading, um which which is a way of basically taking large tool calls and dumping them. That kind of wraps the tool call. And so, the point is there's there's this really simple kind of like general architecture of an agent. All of these more advanced agent harnesses are basically doing that loop, but adding in a bunch of stuff while still running the core loop. And so, as you think about kind of like building or customizing your own harness, these are the different places that you can insert things into. You can add your own summarization step, you can add your own handling of particular tool calls, and that's one way that you can customize kind of like the agent to your particular domain and the harness to your particular domain.

所以你能做的很多定制，都是通过 middleware 这个概念、通过改那个核心循环实现的。agent 还是在循环里跑，还是那个简单架构，但通过 middleware 你可以给它接一个沙箱、接一个文件系统、接 sub-agents、接 memory，也可以加摘要。摘要这件事，回到刚才那张图，它发生在模型调用之前：模型被调用前，你先检查 context 是不是太长了，太长就压缩一下。这就是通过 middleware 加进核心循环的。context offloading 也一样，它的做法基本是把体积很大的工具调用结果卸载出去，相当于在工具调用外面包了一层。所以重点是：agent 有一个非常简单、非常通用的架构；所有那些更高级的 agent harness，本质上跑的都是这个循环，只是在跑核心循环的同时往里塞了一堆东西。所以当你考虑自建或定制 harness 时，这些就是你可以插东西进去的位置——你可以加自己的摘要步骤，可以加对某类工具调用的特殊处理，这就是把 agent、把 harness 调教到贴合你自己领域的一条路。

[5:41] Harrison

The other way that you can customize the harnesses that the agent runs in is by having a more explicit kind of like cognitive So, this used to be the way that a lot of people would build agents in kind of 2023, 2024 because the models weren't good enough to run in a loop. And so, in order to get it to do particular things, you would have these very bespoke cognitive architectures. And so, this one over here is for a deep research example where it would generate some some sub questions, fan them out, and then go and execute them. And then this one over here is for a code review bot. And you can see that there's these very kind of like the bespoke steps. A lot of this has gone into the harness now. And by the harness, I mean it's still this core loop. These might be added as particular kind of like modifications to that core loop. But, for a lot of really particular kind of like flows, we do see people still using cognitive architectures like these to really guide it in particular ways. One thing that we recommend to people is to start with a general harness. That's the easiest to get started. It's going to be quickest to time to value. And then as you kind of narrow in on the use case that you want to be excellent at, you can start to add more of these kind of like gates and checks around it to to to guide it into particular ways.

定制 agent 运行环境的另一条路，是用更显式的认知架构（cognitive architecture）。2023、2024 年那会儿大家基本都这么搭 agent，因为当时模型还不够好，扔进循环里跑不起来。所以为了让它干成特定的事，你得写一套非常定制的认知架构。这边这个是 deep research 的例子：先生成一堆子问题，扇出去，再逐个执行。那边这个是 code review 机器人。你能看到里面都是很定制化的步骤。这类东西现在很多都被吸进 harness 里了——我说的 harness 仍然是那个核心循环，这些步骤可能是以对核心循环做特定改造的形式加进去的。但对某些非常具体的流程，我们确实还看到有人在用这样的认知架构，把 agent 往特定方向硬引导。我们给大家的建议是：先从通用 harness 起步，这最容易上手，出价值最快；等你把想做到极致的那个场景收窄了，再往上加各种关卡和校验，把它引导到特定路径上。

[6:55] Harrison

One question that we get a lot is when to kind of think about building your own harness versus using an off-the-shelf harness. Um a lot of the off-the-shelf harnesses are uh work with particular models. So, the off-the-shelf harnesses include things like Claude Code or Claude Agent SDK, which works with Anthropic models, Codex, which works with OpenAI models. I think this is a big open question in the industry. My answer generally is the more in distribution you are of what the models are trained on, then the better the off-the-shelf harness will be. As soon as you start to move further and further out of distribution, then the then you'll probably want to tune your harness in some way. There's different ways to tune the harness as well. So, the models may be in distribution on particular things that you are doing on an out of distribution task. So, what I mean by that is if you think about something like like legal AI, which Gabe just talked about, and I think he mentioned how they have their own harness, there are things that are in legal AI that are still in distribution of the the the main models. So, for example, editing files is something that the main models have all been RL'd on. And they've actually all been RL'd in very particular ways. So, OpenAI and and Claude models edit files in in different ways in their harnesses, and as a result, their models are actually best at editing files in different ways.

我们被问得最多的问题之一是：什么时候该自建 harness，什么时候直接用现成的 harness？很多现成 harness 是绑定特定模型的，比如 Claude Code 或者 Claude Agent SDK，配 Anthropic 的模型；Codex 配 OpenAI 的模型。我觉得这在行业里还是个很大的开放问题。我的一般性回答是：你越是落在模型训练分布之内（in distribution），现成 harness 的效果就越好；一旦你越走越偏、跑到分布之外，那你多半就得对 harness 做些调校。调校 harness 的方式也有好几种。还有一点是：你在做的整体任务可能是 out of distribution 的，但其中某些具体动作对模型来说仍然是 in distribution 的。我举个例子，比如 Gabe 刚讲的法律 AI——我记得他提到他们有自己的 harness——法律 AI 里有些环节对主流模型来说仍然在分布之内。比如编辑文件，所有主流模型都被 RL 训练过这个动作，而且训练方式还各不相同。OpenAI 的模型和 Claude 的模型在各自 harness 里编辑文件的方式就不一样，结果就是它们各自最擅长的编辑文件方式也不一样。

[8:16] Harrison

Now, the the models themselves are out of distribution on this larger task of legal AI, but they're in distribution on this task of editing files. So, if you think about building a harness that works there, you'll probably want a custom harness, but you'll want it to use the edit file tool that is in distribution for the the the model that you're using. So, one of the things we do in Deep Agents, for example, so Deep Agents is our customizable harness, we actually have this concept of model profiles, where for things that are in distribution of models like editing files, we basically switch between different edit file implementations depending on which model's being used. And so, I think that's an example of customizing the overall harness when it's out of distribution for a task, but keeping smaller in distribution parts as close to the model layer as as possible. The second big part of what I want to talk about is evals and observability. And so, I think as you're experimenting with all parts of an agent, whether it's the model or the harness or the context, you're going to want to know what's going on inside of this system, and you're going to want to be able to evaluate it. And so, these are useful tools that you can use, again, not just for custom harnesses, but also for custom models.

所以，模型在「法律 AI」这个更大的任务上是 out of distribution 的，但在「编辑文件」这个动作上是 in distribution 的。那你要在这个场景里做 harness，你多半需要一个定制 harness，但你会希望它调用的编辑文件工具，正好是你所用那个模型分布之内的那一套。举个我们在 DeepAgents 里的做法——DeepAgents 是我们那套可定制的 harness——我们做了一个叫 model profiles 的东西：对于编辑文件这类落在模型分布内的动作，我们会根据当前用的是哪个模型，在不同的编辑文件实现之间自动切换。我觉得这就是一个很好的例子：整体任务 out of distribution 时定制整个 harness，但里面那些 in distribution 的小零件，尽量贴着模型层原样保留。今天我想讲的第二大块是 evals 和可观测性。我觉得当你在折腾 agent 的各个部分时——不管是模型、harness 还是 context——你都会想知道这个系统内部到底发生了什么，也会想有办法评估它。所以这些工具很有用，而且不只是对定制 harness 有用，对定制模型同样有用。

[9:29] Harrison

So, there was a great Twitter article that Satya wrote uh 2 weeks ago, um where he talked about a lot of these concepts. And there's three quotes in particular that kind of stood out for me. One, create your private evals because eval defines what good looks like inside the organization. Two, retain ownership of your organization's memory, traces, feedback, though that bold is mine, decisions, and institutional context. And then three, you create your own continuous learning loop hill climbing machine that will allow your AI investments to compound the value of your firm. And so I think these speak to the importance of evals and observability and the learning loop that they power in in really owning your intelligence and compounding it. So how exactly do they do that? So evals. Gabe was here talking about how they built benchmarks for the legal domain. I think every company what when they're building a mission-critical agent, they will build benchmarks for that agent. Um you can use it to define and catch regressions or you can hill climb on that benchmark. Again, either by adjusting the harness or adjusting the model. Things that we see becoming the industry standard for defining these benchmarks is Harbor. Harbor is an open-source eval runner. It's uh created by the makers of Terminal Bench 2, which is one of the industry standard benchmarks for benchmarking coding agents, and it's become pretty popular for a variety of domains.

两周前 Satya 在 Twitter 上写了一篇很好的长文，讲了很多这方面的概念。其中有三句话我印象特别深。第一，建立你自己的私有 evals，因为 eval 定义了在你这个组织内部「什么叫做得好」。第二，把组织的记忆、trace、反馈（这个加粗是我加的）、决策和机构性知识的所有权握在自己手里。第三，你要打造自己的持续学习闭环、自己的爬坡机器，让你在 AI 上的投入能对公司价值形成复利。我觉得这几句话正好点出了 evals、可观测性，以及它们驱动的学习闭环，对于真正掌握并复利放大你的智能有多重要。那具体怎么做到？先说 evals。Gabe 刚才讲了他们怎么给法律领域造 benchmark。我认为每家公司在做关键任务级别的 agent 时，都会给这个 agent 造一套 benchmark。你可以用它来定义和捕捉回归，也可以拿它来爬坡——不管是调 harness 还是调模型。我们看到在造这类 benchmark 上正在成为行业标准的工具是 Harbor。Harbor 是一个开源的 eval runner，作者就是 Terminal Bench 2 那个团队——那是评测编码 agent 的行业标准 benchmark 之一——现在 Harbor 在很多领域都挺流行。

[10:48] Harrison

What it lets you do, so this is so this is Frontier Bench, which is another uh coding benchmark. You get this nice benchmark and you can compare different agent harnesses, different models, different reasoning efforts, and you can get this nice benchmark and you can see how all these different harnesses and all these different models do on your task. And so having a benchmark for your task will become really really important when you're trying to define it. What exactly is Harbor? It's pretty simple. At a high level, it consists of agent that you you run an agent against the data set. A data set has a bunch of different tasks. Generally, they're run in sandboxes because they are a lot of these different tasks and you might want to parallelize them. And as we talk as I talk about in a little bit, each task has its own kind of like environment. So this is what a Harbor task looks like. So, on the right, you can see that it has an environment. This is where you define the environment that the agent runs in. A lot of these longer running, more stateful agents need to interact with their environment. And so, you basically spin up a sandbox, give it its own environment that's defined in a Dockerfile, and run it there.

它能让你做什么？比如这个是 Frontier Bench，又一个编码类 benchmark。你能拿到这样一张漂亮的成绩表，去对比不同的 agent harness、不同的模型、不同的推理强度（reasoning effort），一眼看出这些 harness 和这些模型在你的任务上各自表现如何。所以当你想把任务定义清楚时，为你自己的任务准备一套 benchmark 会变得极其重要。Harbor 到底是什么？其实挺简单。从高层看，它由 agent 组成——你拿一个 agent 去跑一个数据集。数据集里有一堆任务。这些任务一般跑在沙箱里，因为任务数量多，你可能想并行。而且待会儿我会讲到，每个任务都有自己的环境。这就是一个 Harbor 任务长什么样。右边你能看到它有一个 environment，你在这里定义 agent 运行的环境。这类长时运行、更有状态的 agent 需要跟环境交互，所以你基本就是起一个沙箱，给它一个用 Dockerfile 定义好的独立环境，让它在里面跑。

[11:48] Harrison

There's then uh a solution, which is basically this this is uh a kind of like golden solution that you use to sanity check it, so it's not that interesting. Test is more interesting. This is basically the verifier for the the agent run. The test scripts can do anything. They can run code, they can run unit tests, they can run another LLM as a judge, they can run an agent as a judge. You basically define how the agent is scored in this test. And then instruction.md is the prompt that the agent is given. And that's kind of like the core of Harbor. You define these tasks, which are bundled up things that can be run in a sandbox, and then you run a bunch of them against agents. And agents again consist of models and harnesses, and you score how how they do. When you do all of that, what do you get? You get some nice results that you can compare. So, this is LangSmith, the the platform that we build for evals and observability. And so, you can see here a bunch of different experiments. Uh we have a great integration with Harbor. You can see the the feedback scores. In this case, it's a single reward function. You can also track latency and tokens. So, when you're benchmarking agents, you probably don't just care about accuracy. You also probably care about latency and and and cost. And so, you'll want to track all of those.

接下来有个 solution，本质上是一份「标准答案」解法，用来做 sanity check，所以没那么有意思。test 更有意思，它其实就是这次 agent 运行的验证器。测试脚本可以干任何事：跑代码、跑单元测试、跑另一个 LLM 当裁判、跑一个 agent 当裁判。你在 test 里定义 agent 怎么被打分。然后 instruction.md 就是给 agent 的 prompt。这基本就是 Harbor 的核心：你定义这些任务，每个任务打包成一份能在沙箱里跑的东西，然后拿一堆任务去跑 agent；而 agent 又由模型和 harness 组成，你给它们的表现打分。这些都跑完，你得到什么？你会得到一份可以横向对比的漂亮结果。这个是 LangSmith，我们做的 evals 和可观测性平台。这里你能看到一堆不同的实验。我们跟 Harbor 有很好的集成。你能看到反馈分数，这个例子里是单一的奖励函数。你还能追踪延迟和 token 用量。因为给 agent 做 benchmark 时，你多半不只关心准确率，也关心延迟和成本，所以这些你都会想追踪。

[12:55] Harrison

And then for a particular experiment that you run, these are these would be the different tasks that are in a in a in a Harbor data set. Talking about a little bit about observability. Uh observability sounds basic, but I think it's really important and really underrated for agents, actually. So, when agents mess up, they mess up because an LLM call goes wrong. Why might it go wrong? It might go wrong for one of two reasons. One, the model is not good enough. Two, the context that the LLM received isn't good enough. And so, I actually think it's the the second one that more often than not causes issues. And so, having really good observability into what is going into the context window of the the model and then how that context is accumulated, what steps were run, what tools were run, how does that context get there? All of that is really important for debugging your agent when it when it goes wrong. So this is one view of observability that we have. This is intended to be a more kind of like user-friendly view where we actually represent it. This is similar to what you might see in kind of like Claude code. We we kind of like hide some of the tool calls so you can see seven tool calls up there.

然后针对你跑的某一次具体实验，这些就是 Harbor 数据集里一个个具体的 task。再稍微聊聊 observability（可观测性）。observability 听上去很基础，但我觉得它对 agent 来说真的很重要，而且被严重低估了。agent 出岔子，根子上是某一次 LLM 调用出了问题。为什么会出问题？无非两个原因：一是模型不够强，二是这次调用拿到的 context 不够好。而我的判断是，多数情况下毛病出在第二个上。所以你必须能非常清楚地看到：到底有什么东西进了模型的 context window，这些 context 又是怎么一步步攒起来的——跑了哪些步骤、调了哪些工具、这段 context 是从哪冒出来的。agent 出错要 debug 的时候，这些信息全都至关重要。这是我们做的一种 observability 视图，偏「给人看」的友好呈现，跟你在 Claude Code 里看到的挺像——我们会把一部分 tool call 折叠起来，所以上面能看到「七次 tool call」这样的提示。

[13:57] Harrison

And so we try to make it really easy to kind of like skim through this. Most agent paths these days come in the form of trajectories. Trajectories are basically you can think of them as the list of messages that you see kind of like Claude code running. So when you run Claude code or another agent, you type in a human message, it then makes a bunch of tool calls. Those are all messages under the hood and then it responds and then you type in another human message. That's kind of like this message trajectory that is becoming more and more of a central part of of these agents. But that's not enough to fully debug it and so we also have this full kind of like trace and you can click into particular things and see exactly what goes on inside the model. And this type of observability is pretty important for knowing what's going on. Evals and observability really let you set up this data flywheel and compound the intelligence as you as you start to use the agent, as your users start to use the agent and you start to get feedback. So this is this is a slide that one of our team members presented at Swyx's AI engineering fair actually around a recipe for continuously improving agents. At a high-level it's really simple. You build an agent, you start running it, you collect lots of traces, you then curate the trace data, and then you run experiments on on that data that you create. And so it's really simple, but of course there's a lot of complexity under the hood. So so one thing that's really important for this is is feedback. Getting feedback either from the environment or from synthetic source. So from the environment, one thing that I think is really underestimated in agent design is actually UX design of how you present the agent to your users. If you present it in a really intelligent way, you can actually end up getting a lot of feedback from them. They may not click thumbs up or thumbs down explicitly. No

我们想让人一眼扫过去就能看懂。现在 agent 的运行路径大多以 trajectory 的形式呈现。trajectory 你可以理解成你在 Claude Code 里看到的那一长串消息：你输入一条 human message，它接着发起一堆 tool call——底层这些也都是 message——然后它给出回复，你再输入下一条 human message。这种消息 trajectory 正越来越成为这类 agent 的核心部分。但光靠它还不足以彻底 debug，所以我们还提供了完整的 trace，你可以点进某一环，看清模型内部到底发生了什么。这种 observability 对搞明白系统在干嘛相当关键。eval 加 observability，真正能帮你转起数据飞轮：随着你自己开始用、你的用户开始用、反馈开始回流，agent 的能力会不断复利式积累。这一页其实是我们团队一位同事在 swyx 办的 AI Engineer 大会上讲过的内容，一套持续改进 agent 的配方。往大了说非常简单：造一个 agent，跑起来，收集大量 trace，然后把 trace 数据筛选整理，再基于整理出来的数据去跑实验。听着简单，底下当然一堆复杂度。其中特别关键的一环是反馈——要么来自环境，要么来自合成来源。先说环境这块，我觉得 agent 设计里最被低估的一件事，其实是 UX 设计，也就是你怎么把 agent 呈现给用户。如果呈现得足够聪明，你能从用户那儿拿到大量反馈。他们未必会明确去点赞或点踩，几乎没

[15:36] Harrison

one really doing that. But if you if you design the UX in a clever way, you can get some of that feedback. The other thing you can do is you can start to get synthetic feedback. So you can run what we call kind of like online evaluators over these traces to judge things. So Gabe was talking about an experiment that we did with Harvey where we we significantly reduced the cost of some of these LLM as a judge type thing. So if you imagine running Opus over every single trace that comes into your system, that's going to rack up a big bill. And so you want a really cheap and fast way of doing this. So we've fine-tuned some SLMs for actually doing this, but you can of course use off-the-shelf models with custom prompting to do it. Or you can just use code if some of the things that you want to test are are simple enough. Um so this is the the the full a part of it. Um the curating the trace data, feedback is a big part there. And then the other thing is is when you use that data to update what happens, you can update any part of the agent with this with this kind of like system. Um so you can update the harness by doing harness engineering. You can update the model by do a by doing fine-tuning on that. You can update the context by doing memory.

什么人真会那么干。但只要 UX 设计得巧妙一点，你还是能捞到一部分反馈。另一条路是拿合成反馈。你可以在这些 trace 上跑我们所说的 online evaluator，让它对各种维度做判定。Gabe 刚提到我们和 Harvey 做过一个实验，把这类 LLM-as-a-judge 的成本大幅压了下来。你想想，如果系统里每进来一条 trace 都拿 Opus 跑一遍，账单会非常吓人。所以你需要一个又便宜又快的做法。我们为此微调了一些 SLM，当然你也完全可以用现成模型配定制 prompt 来做；如果要测的东西足够简单，直接写代码判定也行。这就是其中一块的全貌——整理 trace 数据，反馈在里面占了很大比重。另一块是：当你拿这些数据去做更新时，这套机制其实可以更新 agent 的任何一个部分。你可以通过 harness engineering 更新 harness，可以通过 fine-tuning 更新模型，也可以通过 memory 更新 context。

[16:45] Harrison

Um and and so the part that we are that we think most about at LangChain is the harness engineering part of that. And so I want to show a really quick demo of of one of the things that we added to help with that. But I I think Trajectory is talking next on some fine-tuning that can be done. Uh and and so it's a very similar process where you run the agent, get some traces, use that data in some way to improve the system. What is the system? It's these three pieces. Any of these can be updated in some way. Um and so yeah, this is the full end-to-end um uh uh flow that you might want to do. One of the things that we think about is how can you automate this as much as possible because this is tricky and takes a lot of time. Um and so that's one of the things that we've been thinking about for the past few months. I want to do a quick demo of what we call LangSmith Engine, uh which is basically an agent that sits on top of your traces and does all this work. So if we look at what that work was, you know, you've got these traces, it's it the work from there is curating the traces and running some experiments, suggesting fixes to one of the three things. And as I mentioned, we mostly focus on the harness engineering bit. Um so uh in the demo I want to show uh what this looks like and how it represents that. So hopefully this will work. If not, it's not that big of a deal.

而在 LangChain，我们想得最多的就是 harness engineering 这一块。我想快速演示一下我们为此加的一个东西。不过我记得 Trajectory 接下来会讲 fine-tuning 那条路。流程其实很像：跑 agent、拿 trace、用这些数据以某种方式改进系统。系统是什么？就是刚才那三块，任何一块都可以被更新。所以这就是你可能想跑的完整端到端流程。我们一直在想的一个问题是：这件事怎么才能尽可能自动化？因为它很棘手，而且极其耗时间。这也是我们过去几个月一直在琢磨的事。我想快速演示一下我们叫 LangSmith Engine 的东西——本质上就是一个坐在你的 trace 之上、把这些活儿都干了的 agent。回头看那些活儿到底是什么：你手上有一堆 trace，接下来就是筛选整理这些 trace、跑一些实验、针对那三样东西之一给出修复建议。前面说了，我们主要聚焦在 harness engineering 这块。下面我就演示一下它长什么样、怎么把这些呈现出来。希望能顺利跑起来，跑不起来也不是什么大事。

[17:59] Harrison

Perfect. Okay. So this is LangSmith. This is a bunch of traces we have coming in. Um we have this tab called engine over here. Um this is an agent. It runs in the background. It creates what we call kind of like issue boards. So this is the part of curating data. It will look for It will It will It will basically under the hood is a coding agent that has access to our LangSmith CLI. The LangSmith CLI, you can filter traces for feedback and other things like that. So we give it a nice big prompt and some sub-agents that help it basically go out and explore this data and identify issues and see what common things are. And then it will create these issues right here. And so here um it's created an issue. It gives a description of it. It has It links to the traces so I can go see some supporting evidence. And then down here, I guess this is very simple changes to the prompts. Um but here it's updating part of uh of the context in this case. Um here it's also updating some instructions. Um and we can see here that it's adding uh some code to go into the harness as well. And so this is uh something we launched in the past few uh months and I think speaks to this data flywheel, which again is a very simple thing. Run agent, get traces, see patterns, fix. Um and and this is our attempt at automating it. Um, that's all I've got. Happy to take any questions on harnesses or evals.

完美。好，这就是 LangSmith，这是一批进来的 trace。这边有个叫 Engine 的标签页。它是一个 agent，在后台跑，会生成我们叫 issue board 的东西——这对应的就是整理数据那一步。它底层其实是一个能调用我们 LangSmith CLI 的 coding agent。用 LangSmith CLI 可以按反馈之类的条件去筛 trace。我们给它一段很长的 prompt，再配几个 sub-agent，让它自己去把这堆数据翻一遍、找出问题、看看有哪些共性。然后它就会在这里生成这些 issue。比如这条，它建了一个 issue，写了描述，还链到了对应的 trace，我可以点进去看佐证。再往下，这次是几处很简单的 prompt 改动——这里它在改 context 的一部分，这里在改一些指令，这里还能看到它往 harness 里加了一段代码。这是我们过去几个月上线的东西，我觉得它正好体现了这个数据飞轮。飞轮本身特别简单：跑 agent、拿 trace、看模式、修。这就是我们把它自动化的一次尝试。我就讲这些，关于 harness 或者 eval，欢迎提问。

[19:19] 提问者

Um, this is great, by the way. Really appreciate the whole whole presentation. Uh, Engine itself is an agent, right? That's given a prompt and go search over things. Uh, have you run Engine on Engine?

顺便说一句，这个真的很棒，整场分享我都很喜欢。Engine 本身就是个 agent，对吧？给它一个 prompt，让它去到处搜。你们有没有拿 Engine 去跑 Engine 自己？

[19:31] Harrison

We have it running, yeah. So, we got slight So, Engine also hooks up to Slack and sends it kind of like reports about itself. Um, and yeah, that's how we that's how we dogfood it. Yeah. We also uh we also created uh uh what we call kind of like issue bench for Engine, which again is like a harbor a harbor formatted uh uh benchmark, basically, that we're constantly benchmarking different models and different harnesses on. Um, and so, it's I think Gabe talked about this a little bit, but one of the benefits of having a benchmark is you can you can benchmark it on a bunch of different harnesses and see what they're good and bad at. So, we we uh I think a few weeks ago we ran our our own kind of like deep agents and then Codex and then Claude Code on this. And we saw that Codex was doing a really interesting thing, where it would write itself a bunch of small scripts to run against these traces, and it was doing that really aggressively and actually allowing it to perform really well. So, we we did a sprint to do what we call the kind of like codis- codexification of Engine and basically take that learning and and bring it into kind of like the the core Engine harness. And so, I think that's another uh a benefit of having a benchmark is you can just run a bunch of different things on it and see how they actually perform and then bring those things back into your core kind of like agent harness.

跑了，一直开着。而且 Engine 还接了 Slack，会把关于它自己的报告发过来，我们就是这么 dogfood 的。我们还给 Engine 做了一个叫 issue bench 的东西，同样是 Harbor 格式的 benchmark，我们不断拿不同的模型、不同的 harness 在上面跑分。Gabe 刚才也提到一点：有 benchmark 的一个好处是，你可以把一堆不同的 harness 都放上去跑，看看各自擅长什么、不擅长什么。几周前我们就把自家的 Deep Agents、Codex 还有 Claude Code 全放上去跑了一轮。结果发现 Codex 有个很有意思的做法：它会给自己写一堆小脚本去跑这些 trace，而且用得非常凶，恰恰是这一点让它表现特别好。于是我们专门做了一个 sprint，我们管它叫 Engine 的「Codex 化」，把这个心得搬进 Engine 的核心 harness。所以我觉得这是有 benchmark 的另一个好处：你可以拿各种东西上去跑，看它们真实表现如何，再把好东西吸收回自己的核心 agent harness。

[20:45] 提问者

Super cool talk. Um, to which extent do you think that like harnesses will converge into one thing and users will be educated to do that and the models will be best for that versus diversifying here, every company has their own uh way of doing things uh optimized for them?

讲得太棒了。你觉得 harness 最后会收敛成一种形态、用户被教育成按那一套来、模型也针对那一套做到最好；还是会走向分化，每家公司都有自己一套、按自己的场景做优化？

[21:02] Harrison

Yeah. Yeah, really good question, one that we think a lot about. And um, I I chatted with Eno from Factory, who also kind of like thinks a lot about this. Um, and and I think there's some stuff we talked about this. Um, I I think uh I think there's I I think the model I I don't know is the is the honest answer. I think uh some things that I've seen is that the general purpose harnesses have gotten good enough to work for a lot of basic tasks at least when you're getting started. So, I would recommend getting started with like an off-the-shelf harness, whether it's Deep Agents or Codex or Cloud Code or something like that. Because I think the models are now good enough and the things that we've learned about what these makes these models good, access to file systems, sub-agents, things like that, those are those those are kind of like good enough. Um, I think we often see that the more out of distribution you get, the more you're going to want to customize the harness. And it's a scale, right? So, like it uh it it it it at the extreme end of a scale, you might want to build a complete kind of like cognitive architecture that that is really focused on things. A A reason Another reason you might want to do that, by the way, is kind of like for predictability and control. And so, we have a lot of customers in financial services where they want where they need kind of like predictability. And so, they we show them something like Deep Agents and they're like, "Woah, woah, woah. That's way too like scary an agent for us. We want like more of this kind of like custom cognitive architecture where we can really control things." Um, but but then on the other end, uh you know, you could just use an off-the-shelf harness and there's things in the middle like hooks or middleware that you can use to kind of So, so it's a spectrum as well. The more out of distribution you get, the more custom harness you're going to want to have.

嗯，非常好的问题，我们也想了很多。我跟 Factory 的 Eno 聊过，他也一直在琢磨这件事，我们聊到一些东西。老实说，我不知道答案。我观察到的一些现象是：通用 harness 已经足够好，至少在起步阶段能应付很多基础任务。所以我建议先从现成的 harness 起步，Deep Agents、Codex、Claude Code 之类都行。因为模型现在确实够强了，而我们摸清的那些让模型变强的要素——能访问文件系统、能开 sub-agent 之类——也基本够用了。但我们经常看到，你的场景越 out of distribution（越偏离常见分布），你就越会想去定制 harness。这是个连续的光谱：最极端那一头，你可能想从头搭一套完整的、针对性极强的 cognitive architecture。顺便说，另一个想这么做的理由是可预测性和可控性。我们有不少金融服务行业的客户，他们必须要可预测。我们把 Deep Agents 这类东西给他们看，他们会说：「打住打住，这 agent 对我们来说太吓人了，我们更想要那种能把每一步都控住的定制 cognitive architecture。」但另一头，你也完全可以直接用现成的 harness；中间还有 hook、middleware 这些手段可以拿来调。所以它同样是个光谱：越 out of distribution，你就越需要定制的 harness。

[22:41] Harrison

Um, and then there's other like weird things where like again, like um, I I I think uh both OpenAI and Anthropic are getting really good at coding, but they've landed on different ways to kind of like edit files um that are like, you know, that are like actually pretty different. Um, you know, I I think they have some benchmark and I think he was uh he he thought that one way was just better than the other way just like strictly superior and and so that's like like so I think the model labs will kind of converge and that they all seem to be kind of like really good at coding the harnesses will converge to to kind of like being really good at coding if they keep on going down that path but they're at the same time there are these like really small differences and I don't really know how to explain those either and and right now I think those show up most concretely in small things but you could imagine what what if one lab really goes down kind of like bio and those harnesses become really good at kind of like bio agent things then then then the harnesses themselves start to diverge and so I I I don't know is the answer to the fast moving space that's why evals and observability are important and I think and and I think we to measure all of that.

另外还有些挺怪的现象。比如我觉得 OpenAI 和 Anthropic 在编码上都做得非常好，但他们在「怎么改文件」这件事上走了完全不同的路子，差别其实还挺大的。他们各自有一些 benchmark，我记得他的看法是其中一种做法就是比另一种好，属于严格意义上的更优。所以在我看来，模型实验室这边多半会收敛——大家看起来都很擅长编码；如果沿着这条路一直走，harness 也会朝「特别擅长编码」这个方向收敛。但与此同时，又确实存在这些非常细微的差异，我也解释不清楚。目前这些差异主要体现在很小的地方，但你可以设想一下：万一某家实验室在生物方向猛扎下去，它的 harness 在生物类 agent 任务上变得特别强，那 harness 之间就会开始分化。所以我的答案还是「不知道」。这个领域变化太快，这也正是 eval 和 observability 重要的原因——我们得有办法把这些都量出来。

[23:49]

Cool. Awesome. Thank you guys.

好，太棒了。谢谢大家。

[23:52]

[applause]

[掌声]