Long-Horizon Agents Need Experiments, Not Just Prompts — Erina Karati
频道: AI Engineer
视频: https://www.youtube.com/watch?v=x4e5O9zN0TE
原文语言: en
统计: 共 11 轮 · Erina 11
[0:12] Erina
Hi everyone, I'm Arena, former engineer at Microsoft and Supercell. And today I want to talk about auto research in a multi- aent AI village. I will use a video game like AI Village as a running example here, but the broader question is one I think many AI engineers are starting to run into. How do we evaluate and improve agents that carry state over a long period of time? Before I get into the auto research layer, I want to talk a bit about project paradox. We developed project paradox at supercell's AI innovation lab. Me and my teammate are natural money. We built a modular AI framework that allows any developer to plug in intelligent autonomous agents within a video game that can interact, compete or cooperate with other players or agents as well and place them uh and make them into dynamic game companions. Now to give examples of what these agents can do, the agents can move with intent. They can go to any location or person and they're guided by their own memories, emotion, or curiosity. These agents can interact with the world. They can pick up objects, drop them anywhere, and they're also aware about the context in their own environment, such as the objects or other characters or agents as well. I would also like to note that game developers can also add new actions for these agents to accomplish within our framework as well. Instead of just dropping or uh placing objects, agents can also obviously react to what's happening around them. And these events that happen around them affect their own beliefs and emotions on the fly as well. And of course, it wouldn't be complete if agents can't start conversations, right? agents can in this scenario approach other agents or even the player as well and this makes the game feel more alive. And of course these conversations are stored within their memory and is according to their own um and affect their own emotions and beliefs or goals as well.
大家好，我是 Erina，之前在 Microsoft 和 Supercell 当工程师。今天我想聊聊在一个 multi-agent（多智能体）AI 村庄里做自动研究（autoresearch）。我会拿一个像电子游戏那样的 AI 村庄当贯穿全场的例子，不过背后那个更大的问题，我想很多 AI 工程师都开始碰到了：那些要长时间携带状态的 agent，我们该怎么评估、怎么改进？在讲自动研究这一层之前，我想先简单介绍一下 Project Paradox。Project Paradox 是我们在 Supercell 的 AI 创新实验室里开发的。我和我的队友 Arunachalam Manikandan 一起搭了一个模块化的 AI 框架，任何开发者都可以把智能的自主 agent 接进电子游戏里，这些 agent 能和其他玩家或 agent 互动、竞争或者合作，成为动态的游戏伙伴。举几个例子说说这些 agent 能做什么：它们能带着目的移动，可以去任何地点、找任何人，驱动它们的是自己的记忆、情绪或好奇心。这些 agent 能和世界互动：捡起物品、把东西放到任何地方，还能感知自己所处环境里的上下文，比如周围的物品、其他角色或 agent。另外我想说明一下，游戏开发者还可以在我们的框架里给这些 agent 添加新的动作让它们去完成。除了放下或摆放物品，agent 当然也能对周围发生的事做出反应，而周围发生的这些事件也会实时影响它们自己的信念和情绪。当然，如果 agent 不能主动发起对话，那就不算完整了，对吧？在这个场景里，agent 可以主动走向其他 agent，甚至走向玩家，这让游戏显得更有生气。而且这些对话当然都会存进它们的记忆里，也会影响它们自己的情绪、信念或目标。
[2:43] Erina
And al together these agents make our multi- aentic framework. Um yeah yeah one second so the architecture was intentionally stateful behind this. The first important part was per agent memory. Each agent has its own memory namespace backed by rag. So memory did not bleed between agents. Second, we tracked emotion as a small vector. So after an event or conversation, the system could update values like joy, sadness, fear, anger, or disgust. Third, agents had belief scores towards other agents and the player. You can think of this as a trust matrix. Basically like after the interaction happens, the LM basically decides whether the trust score should go up, down, or whether it shouldn't change at all. And fourth, every memory receives an important score. Um to to explain this better, like let's say you had dinner a few days ago, you probably wouldn't remember what you had for dinner, right? But um if someone was murdered a few days ago, you definitely remember that. So the agent will evaluate or the LM will evaluate uh an important score of an event and if it crosses a threshold, it will store that specific memory uh in a separate cache so that important context can be retrieved better later on.
这些 agent 合在一起，就构成了我们的 multi-agent 框架。嗯，对对，稍等一下。这背后的架构是有意设计成有状态的。第一个重要部分是每个 agent 各自的记忆。每个 agent 都有自己的记忆命名空间，底层由 RAG 支撑，所以记忆不会在 agent 之间互相串。第二，我们把情绪记录成一个小向量，这样在一次事件或对话之后，系统就能更新喜悦、悲伤、恐惧、愤怒、厌恶这些数值。第三，agent 对其他 agent 和玩家都有信念分数，你可以把它想成一个信任矩阵。基本上就是每次互动发生之后，由 LLM 来决定信任分该升、该降，还是完全不变。第四，每条记忆都会得到一个重要性分数。为了解释得更清楚一点：比如说你几天前吃了顿晚饭，你大概不会记得那天晚饭吃了什么，对吧？但如果几天前有人被谋杀了，你肯定会记得。所以 agent——或者说 LLM——会评估一个事件的重要性分数，如果超过某个阈值，就把这条记忆单独存进一个缓存里，这样之后就能更好地检索到重要的上下文。
[4:28] Erina
And here's an example of it just working. Um, we going to ask one of the characters to go on a picnic with us. Here, uh, our character Blossom um, decides to pick up a pastry and go to the picnic area because we asked her to do so. Keep in mind during the conversation in the background, she plans all of these sequence of actions to accomplish. And one when we talk to her afterwards, she will also reply within context as well. Yeah. But this is where an interesting problem actually started. As you saw in the last example, like for shortterm game play, this our architecture worked pretty well. like a character could make a plan, move around, talk and remember the recent interaction and respond to us or other characters as well. But over longer horizons, this is where we notice the social consistency start to get weaker. So in this example, we have one agent spreading a rumor about a sale on mangoes to another agent and that agent receives that information and goes and tells another agent about it. Later on, after a number of events that occurred in between, when the player asks one of the agents about the mangoes, it doesn't exactly store that context that we were expecting or it doesn't give us the context that we kind of wanted to. And this is where things are starting to get messy naturally. Like the system may remember the rough topic but lose the source of the topic. A rumor may become certain instead of just a rumor like the agent might state it as a fact or um an agent might know a fact but fail to execute fail to remember it while creating a plan for its actions. So the question here became how do we improve a multi-agentic system over longunning social behavior and not just over one response.
这是它实际跑起来的一个例子。我们会请其中一个角色跟我们去野餐。这里，我们的角色 Blossom 决定拿上一块点心，走去野餐区，因为我们让她这么做。注意，就在对话进行的时候，她已经在后台把要完成的这一连串动作都规划好了。之后我们再跟她说话时，她的回应也能贴合上下文。嗯。但有意思的问题恰恰是从这里开始的。正如上一个例子里看到的，对短时段的游戏过程来说，我们这套架构效果挺好：角色能制定计划、四处走动、说话，记住最近的互动，并回应我们或其他角色。但时间周期一拉长，我们就发现社交一致性开始变弱了。在这个例子里，一个 agent 向另一个 agent 散播了一条芒果打折的传言，那个 agent 收到消息后，又跑去告诉了另一个 agent。后来，中间又发生了好些事，等玩家去问其中一个 agent 芒果的事时，它并没有存下我们预期的那个上下文，或者说没给出我们想要的那个上下文。事情自然就从这里开始变乱了。比如系统可能还记得大致的话题，却丢了这个话题的来源；一条传言可能变成了确定的事，而不再只是传言，agent 可能会把它当成事实说出来；又或者 agent 明明知道某个事实，却在给自己的行动做计划时没能想起来。于是问题就变成了：怎么在长时间持续的社交行为上改进一个 multi-agent 系统，而不只是改进某一次回答？
[6:44] Erina
And this is where we wanted to bring in auto research. As you all know, uh few months ago, Karpathi posted out auto research and this this made us immediately very curious. Uh perhaps we can make the system run experiments uh on itself and can we use this for our system as well. So what we understood is instead of manually tuning a prompt or watching one nice demo, we could define sen a scenario suit, run the agents, collect traces, score the behavior and change a small policy surface and only keep the changes that actually improve the score. And this is where we're trying to bridge project paradox with auto research. So at this point basically our multi-agentic framework project paradox is more like a lab bench and auto research becomes the experimental loop around it. And importantly this is not only about improving rag retrieval. The broader framing is optimizing the agent protocol like how do agents write memories, retrieve them, communicate uncertainty, update trust attribute sources and replan around new facts. basically um yeah in this context uh oh yeah in this context art research is not another agent in the village like I said it's a meta system outside the village the villagers have local perspectives of course they only know what they saw heard remembered or inferred because there isn't a common memory database in between them.
这就是我们想引入自动研究的地方。大家都知道，几个月前 Karpathy 发布了 autoresearch，这一下子就勾起了我们的强烈好奇：也许我们可以让系统在自己身上跑实验——那能不能也把它用到我们的系统上？我们的理解是：与其手动调 prompt，或者只看一次漂亮的演示，不如定义一套场景套件，把 agent 跑起来，收集 trace（运行轨迹记录），给行为打分，改动一小块策略面（允许自动研究改动的那一小块），并且只保留真正让分数提高的改动。这就是我们想把 Project Paradox 和自动研究连接起来的地方。到这一步，我们的 multi-agent 框架 Project Paradox 基本上更像是一个实验台，而自动研究就成了套在它外面的实验循环。重要的是，这不只是为了改进 RAG 检索。从更宽的角度看，我们是在优化 agent 协议：agent 怎么写入记忆、怎么检索记忆、怎么传达不确定性、怎么更新信任、怎么标注信息来源，以及怎么围绕新事实重新规划。基本上，嗯，对，在这个语境下，自动研究并不是村子里的又一个 agent，就像我说的，它是村子外面的一个元系统。村民们当然都只有局部视角：他们只知道自己看到的、听到的、记住的或推断出来的东西，因为他们之间并没有一个共享的记忆数据库。
[8:37] Erina
Information only travels once uh other agents communicate them properly. The auto research layer has a different job here. It reads the full traces of a run, compares what happens against the scenario ground truth, uh scores the behavior and proposes a constrained change to the agent protocol or cognitive policy. Then it reruns the scenario and asks society level behavior like did society level behavior get better. This is the key shift we were trying to look for. So we were no longer evaluating one answer. We were evaluating an entire run. And this is what one of the loops would look like. Like first we define a control scenario which I'll elaborate a bit more about later. For example, one agent learns a public fact or one agent hears a rumor. Uh that could be a controlled scenario. Then we run the simulation. During the run, we collect structured traces, observations, conversations, memory rights, retrievals, belief updates, whatever is relevant to us in that case, we collect. Then we score this behavior. Did the information spread as we expected it to? Did the source attribution survive? Such as, does the agent remember who started the rumor? Did uncertainty stay uncertain? Did agents act on what they actually knew? And then the auto research layer here proposes a small policy change. And this is important. It should not rewrite the whole application of course. it should only edit a controlled policy surface and then we rerun. If the score improves and the guardrails hold, we we keep the improvement and if not we simply just revert back.
只有其他 agent 把信息好好传达出去，信息才会流动。而自动研究层在这里的任务不一样：它会读取一次运行的完整 trace，把实际发生的事情和场景的标准答案（ground truth）做对比，给行为打分，然后对 agent 协议或认知策略提出一个受约束的改动。接着它重跑这个场景，看社会层面的行为——社会层面的行为有没有变好？这正是我们想要的关键转变：我们不再评估某一个回答，而是在评估一整次运行。一轮循环大概是这样的。首先，我们定义一个受控场景，这个我待会儿再展开讲。比如，某个 agent 得知一个公共事实，或者某个 agent 听到一条传言，这都可以是一个受控场景。然后我们跑模拟。运行过程中，我们收集结构化的 trace：观察、对话、记忆写入、检索、信念更新——在那个场景下跟我们相关的，我们都收集。接着给这些行为打分：信息有没有按我们预期的那样传播？来源归属有没有保住，比如 agent 还记不记得传言是谁起的头？不确定的东西是否仍然保持不确定？agent 是否按照自己真正知道的东西来行动？然后，自动研究层会提出一个小的策略改动。这一点很重要：它当然不应该重写整个应用，只应该改动一块受控的策略面，然后我们重跑。如果分数提高了、护栏也守住了，我们就保留这个改进；如果没有，就直接回滚。
[10:43] Erina
And talking about controlled scenarios, the reason why uh scenario design matters is that social behavior is otherwise a bit fuzzy. uh in general in the sense if you just let the agents in our environment wander around it might look cool and you might get nice interactions but it's actually very hard to evaluate on whether the system actually improved. So this is why we believe you need controlled scenarios. For example, one scenario could test a public fact diffusion. Let's say agent A learns uh the bakery will close tomorrow. Do the right agents learn it? Do they remember who said what? Do they rem do they change their plans based on this fact? Another scenario could test rumor uncertainty. Agent let's say agent A hears that agent C might leave the village. When this rumor spreads, does might leave suddenly become is leaving or does it stay as might leave? Like does it become a fact or does it still stay as a a rumor? Another scenario could test replanning. The group has a plan, but one agent learns, let's say, the route they wanted to take is blocked. Do agents update this and communicate this uh with each other to avoid uh a improper plan or scale actions? The point is not that these exact scenarios are universal here. The point we're trying to make is that long horizon agent behavior needs scenario suits.
说到受控场景，场景设计之所以重要，是因为不这么做的话，社交行为就有点模糊。一般来说，如果你只是让环境里的 agent 到处闲逛，看起来可能挺酷，也可能出现不错的互动，但其实很难评估系统到底有没有变好。所以我们认为你需要受控场景。比如，一个场景可以测公共事实的扩散：假设 agent A 得知面包店明天要关门。该知道的 agent 都知道了吗？它们记得谁说了什么吗？它们会根据这个事实改变计划吗？另一个场景可以测传言的不确定性：假设 agent A 听说 agent C 可能要离开村子。这条传言传开时，「可能会离开」会不会突然变成「要离开了」，还是依然保持「可能会离开」？也就是说，它是变成了事实，还是依然只是一条传言？还有一个场景可以测重新规划：一群 agent 定好了计划，但其中一个得知，比如说，他们原本要走的路被堵了。这些 agent 会不会更新这个信息、互相通气，从而避免不当的计划或过时的行动？重点不在于这几个具体场景放之四海而皆准，我们想说的是：长周期的 agent 行为需要场景套件。
[12:32] Erina
And talking about our Mango example again, after running one of our auto research loops, this time after uh a a long pro period of time, when the player finally asked one of the agents about the sale on mangoes, we did find that u the the agent was able to respond within context this time like compared to last time. Um yeah and for this talk the form the exact formula we believe is less important than the shape of the scorecard. Uh you do not want a single vague met metric like agent quality. This will hide all the interesting failures. Instead you want a balanced scorecard. For diffusion, you might measure reach like how many agents know the fact after end steps. For provenence, you measure source retention among agents who know it. How many remember it, where it came from, etc. For rumors, you can measure uncerny preservation and false surn rate. For planning, you can measure action consistency and time to replan. And for privacy, you can measure containment. This matters because optimizing only one metric can create bad behavior. Because let's say if you only optimize for diffusion, the agents may learn to overshare everything. And let's say if you only optimize for memory recall, you might create noisy or steal um like memories. So this scorecard is what keeps the system honest and prevents the auto research agent from gamifying the system to just increase one specific score.
再回到芒果的例子：跑完一轮自动研究循环之后，这一次，过了很长一段时间，玩家终于去问其中一个 agent 芒果打折的事，我们发现，跟上次相比，这次 agent 能贴合上下文来回答了。嗯，就这次演讲而言，我们认为具体公式没有记分卡的整体结构重要。你不会想要一个单一的模糊指标，比如「agent 质量」，那会把所有有意思的失败都掩盖掉。你要的是一张均衡的记分卡。扩散方面，可以测触达，比如 N 步之后有多少 agent 知道这个事实。溯源（provenance）方面，测知道这件事的 agent 当中的来源保留：有多少还记得它是从哪儿来的，等等。传言方面，可以测不确定性保留和虚假确定率（false certainty rate）。规划方面，可以测行动一致性和重新规划所需的时间。隐私方面，可以测 containment，也就是信息有没有被控制在该知道的范围内。这很重要，因为只优化一个指标可能会催生坏行为。比如说，如果只优化扩散，agent 可能会学会什么都往外说；如果只优化记忆召回，你可能会制造出一堆嘈杂或过时的记忆。所以正是这张记分卡让系统保持诚实，防止自动研究 agent 为了拉高某一个分数去钻系统的空子。
[14:32] Erina
The other important engineering lesson that we learned over this project is that um it's important to keep the editable surface really small. The auto research layer should not have permission to randomly rewrite the whole codebase. Instead, it's really important to freeze the harness, the scenarios, and the metrics. So, we're only exposing the part of the system that we actually want to optimize. Here in project paradox for us that meant things like memory writing policy, retrieval policy, communication prompt, belief, trust rules, source attribution, replanning triggers, etc. This gives the search pro process room to improve behavior, but also prevents it from gaming the evaluation directly as we mentioned before. And this is the difference between the LM writing random patches versus the LM actually searching within a controlled policy space. And here are here are examples of the kind of changes I want this kind of loop to search over. If if source attribution disappears, the policy change might be preserve source in memory and uh write uh memory rights and summaries. If rumors harden into facts, the policy change might be stored confidence mark firstand versus secondhand and require hedging when retelling uncertain claims.
我们在这个项目里学到的另一个重要工程经验是：可编辑的范围一定要非常小。自动研究层不应该有权限随意重写整个代码库。相反，非常重要的一点是把 harness（包在模型外面、管提示词/工具调用/循环的那层程序）、场景和指标都冻结起来，这样我们只暴露系统里真正想优化的那一部分。在我们的 Project Paradox 里，这指的是记忆写入策略、检索策略、沟通用的 prompt、信念、信任规则、来源归属、重新规划的触发条件等等。这样既给搜索过程留出了改进行为的空间，也像前面说的那样，防止它直接钻评估的空子。这就是 LLM 随手乱打补丁，和 LLM 在受控的策略空间里真正做搜索之间的区别。下面是几个例子，说明我希望这类循环去搜索什么样的改动。如果来源归属丢了，策略改动可以是：在记忆写入和摘要里保留来源。如果传言固化成了事实，策略改动可以是：存储置信度、标注是一手还是二手信息，并要求在转述不确定的说法时留有余地。
[16:10] Erina
If if facts if public facts stay local, the policy change might be classify useful public facts differently and make agents proactively share important source evidence. The key is that these are small changes to the agent protocol, but they can have larger effects on a society level behavior for multi-agentic systems. This is also where I kind of want to be careful about our claims here because with we believe without repeated current loop results like I wouldn't say the system just generally improved. We're trying to say this is the right kind of surface to expose to an auto research layer uh loop because it is small enough to control but it's still rich enough to change the social behavior to some extent at least. And the biggest lesson for me perhaps was that memory is not enough here. You can add a rag memory to an agent and still not get the current long-term uh horizon behavior that you were looking for. Um because agents need to sometimes know where that information came for uh came from. You need to preserve whether it was firsthand, secondhand, verified or uncertain. Sometimes you need to separate raw episodic memories from what the agent currently believes too. And you need to test behavior through scenarios not not just through vibes. So the other lesson is that uh roll back also is not optional. When you optimize social behavior, a change can improve one thing and damage another. So, a policy that spreads public facts uh faster might also leak private information. A policy that increases recall might increase stale memory usage. So, the loop should basically be like a ratchet. Try a change, score it, keep it only if the scorecard improves and guard rails whole.
如果公共事实只停留在局部、传不开，策略改动可以是：对有用的公共事实另行分类，并让 agent 主动分享重要的来源证据。关键在于，这些都是对 agent 协议的小改动，但对 multi-agent 系统来说，它们能对社会层面的行为产生更大的影响。这里我也想对我们的结论谨慎一点：我们认为，在没有反复多轮的循环结果之前，我不会说这个系统就是全面变好了。我们想说的是，这正是适合暴露给自动研究循环的那类面：它小到足以控制，又丰富到至少能在一定程度上改变社交行为。对我来说，最大的教训可能是：光有记忆是不够的。你可以给 agent 加上 RAG 记忆，却依然得不到你想要的那种长周期行为。因为 agent 有时候需要知道这条信息是从哪儿来的。你得保留它是一手的、二手的、已核实的，还是不确定的。有时候你还得把原始的情景记忆（episodic memory）和 agent 当前相信的东西区分开。而且你得通过场景来测试行为，而不只是凭感觉。另一个教训是：回滚也不是可有可无的。优化社交行为时，一个改动可能改善了一件事，却损害了另一件事。比如，一个让公共事实传得更快的策略，也可能会泄露私密信息；一个提高召回的策略，可能会增加对过时记忆的使用。所以这个循环基本上应该像一个棘轮（只进不退）：试一个改动、打分，只有在记分卡变好、护栏也守得住的时候才保留。
[18:19] Erina
And we we definitely believe this is not only relevant for game agents cuz although I gave you an example using a game village um we believe like let's say for example support agents support agents need to know which policy update comes from where right and whether it supersedes an older answer. Personal assistants for example need to remember commitments that they previously made and h make corrections if uh if the user u wants to change those personal commitments. Research agents need pro uh provenence citations contradiction handling and hypothesis updates. Coding agents need longunning context across issues, files, teammates and changing requirements. Workflow agents need access controls, handoffs, and replplanning when the world changes. All of these systems have the same underlying problem. They maintain state over time. And that state affect affects future action. So they need control scenarios and behavioral scorecards is what we are proposing. So again in brief, a recipe for long horizon agents. If there's one practical recipe I want you to take away, freeze the harness, define scenarios, log traces, score behavior, and expose only a small policy surface. Search over these changes, keep only changes that survive your measurement. And this is an engineering pattern that we believe would uh make sense for longunning agents. The real question we believe is across controlled runs, does the system behave better?
我们确信这不只适用于游戏 agent。虽然我举的是一个游戏村庄的例子，但我们认为，比如说客服 agent，它们需要知道哪条政策更新是从哪儿来的，以及它是否取代了旧的答复。再比如个人助理，需要记住自己之前做过的承诺，并在用户想修改这些个人承诺时做出更正。研究型 agent 需要溯源、引用、矛盾处理和假设更新。编程 agent 需要横跨 issue、文件、队友和不断变化的需求的长期上下文。工作流 agent 需要权限控制、交接，以及在外部世界变化时重新规划。所有这些系统都有同一个底层问题：它们要长期维护状态，而这个状态会影响未来的行动。所以我们的主张是：它们需要受控场景和行为记分卡。再简要总结一下长周期 agent 的配方。如果只希望你带走一个实用配方，那就是：冻结 harness、定义场景、记录 trace、给行为打分，并且只暴露一小块策略面。在这些改动上做搜索，只保留经得住你测量的改动。我们认为，对长时间运行的 agent 来说，这是一个说得通的工程模式。真正要问的问题，我们认为是：在一次次受控运行中，系统的表现有没有变好？
[20:13] Erina
To close, project paradox start as an attempt to make game agents feel alive in a 3D world. But the deeper engineing problem was not animation or dialogue for us. It was the state such as which agent knows what, which agent told whom, what is true, uncertain or outdated, and do agents act on what they remember. Otter research. Otter research gave us a way to approach this a bit more systematically. Not by trusting one demo and not by endlessly handtuning prompts, but by running control experiments and keeping only the changes that survived our measurement. Long horizon agents need experiments and not just prompts. And I hope that's the takeaway that you get from this talk. And yes, please do connect with us. We'd love to talk if you have any questions. Thank you so much for listening. Yeah.
最后总结一下：Project Paradox 最初是想让游戏 agent 在 3D 世界里显得鲜活。但对我们来说，更深层的工程问题不是动画，也不是对话，而是状态：哪个 agent 知道什么，哪个 agent 告诉了谁，什么是真的、什么不确定、什么已经过时，以及 agent 会不会根据自己记得的东西去行动。自动研究。自动研究让我们能更系统地去处理这个问题——不是靠相信某一次演示，也不是无休止地手调 prompt，而是跑受控实验，只保留经得住我们测量的改动。长周期 agent 需要的是实验，而不只是 prompt。希望这就是大家从这次演讲里带走的收获。也欢迎大家来联系我们，有任何问题都很乐意聊。非常感谢大家的聆听。嗯。