RL Environments Explained: How AI Agents Learn Real-World Work | Brendan Foody, Mercor
频道: Sequoia Capital
视频: https://www.youtube.com/watch?v=a00xIn5kwhM
原文语言: en
统计: 共 35 轮 · 主持人 2 · Brendan Foody 21 · 观众提问 4 · 提问者 7
[0:00] 主持人
Um, Record, I think you guys grew from a 1 to a 2 billion dollar revenue run rate in the last 4 months or so. Um, so this company's off to the races and I think you were just so front and center to how companies are thinking about uh post training their own models, uh building their own intelligence. So, thank you for joining us for this conversation. Um, format-wise what we're going to do is we've 15 minutes or so of content from Brendan. He's going to talk about uh RL environments in particular, which I think is a, you know, new frontier topic. It'll be fun to fun to explore. And then we're going to leave 15 minutes or so at the end for Q&A again. So, uh please keep please keep questions back pocket. I will turn it over to you, Brendan.
嗯，Mercor——我记得你们在过去四个月左右，营收年化跑速（run rate）从 10 亿美元一路涨到了 20 亿美元。所以这家公司真的是一路狂奔。而且我觉得，在「企业该怎么给自己的模型做 post-training（后训练，即在通用基础模型之上继续训练，长出属于自己的能力）、怎么打造属于自己的智能」这件事上，你们站在最前排。所以非常感谢你来参加这场对话。形式上我们是这样安排的：先由 Brendan 讲 15 分钟左右的内容，他会重点讲 RL environment（强化学习环境），我觉得这是个挺新的前沿话题，聊起来会很有意思。然后最后再留 15 分钟做 Q&A。所以问题先揣在兜里，一会儿再问。下面交给你了，Brendan。
[0:39] Brendan Foody
Sweet. So, I'll be talking about RL environments. Starting out, I figured it's helpful to give a little bit of the background on the history of the data market and how that history ties into Record's origin story. Where things really started in 2020 in the era of crowdsourcing data for behavior cloning. So, this was mainly supervised fine-tuning data, inputs and outputs, and RLHF data where you would have a annotator select from a couple of model responses which they preferred. And we were able to make all this progress in fine-tuning GPT-3, making progress towards ChatGPT and GPT-4 in the crowdsourcing era of agentic data. But, what we saw changing in the market, especially as we uh headed into 2024, was this giant transition away from the low-skilled crowdsourcing era of behavior cloning data and moving towards the agentic era of data. Of how do we find the highest-skilled experts in the world that can work collaboratively in teams to build frontier evals and RL environments for the next generation of models models. All the software engineers, lawyers, doctors, bankers, et cetera that could measure the frontier of intelligence and help to use that to improve model capabilities. And so, Mercor grew up with our first big project being deep research. I guess the first prominent RL agent scaling up dramatically with all of the frontier labs to become the primary agentic data vendor to all of the leading labs and also all of the leading application layer companies ranging from Harvey, Cera, Cognition to Ramp. And what's been really exciting over the last 12 months especially is how RLVR within the agentic data paradigm has evolved to also include RL environments with these rich apps and worlds that teach agents how to use all of the tools on our laptops that we use every day. So, I'll be talking about that and of course how this technology that started in the frontier labs is now getting disseminated to the application
好嘞。我今天要讲的是 RL environment。开场我想先补一点背景——数据市场这些年是怎么演变的，以及这段历史怎么和 Mercor 的创业故事扣在一起。
真正的起点是 2020 年，那时候是「众包数据做 behavior cloning（行为克隆，让模型模仿人类的示范动作）」的年代。当时主要就两类数据：一类是 supervised fine-tuning（SFT，监督微调）数据，也就是一问一答的输入输出对；另一类是 RLHF（基于人类反馈的强化学习）数据，标注员从模型给出的几个回答里挑一个自己更喜欢的。在那个众包时代，我们靠这些数据完成了 GPT-3 的微调，一路推进到 ChatGPT 和 GPT-4。
但我们看到市场在变，尤其是走进 2024 年之后，出现了一次巨大的转向：从低技能众包的行为克隆数据时代，转向 agentic（智能体）数据时代。核心问题变成了——怎么找到全世界技能最顶尖的专家，让他们组队协作，为下一代模型构建前沿的 eval（评测集）和 RL environment。就是那些软件工程师、律师、医生、银行家等等，他们能衡量智能的前沿到底在哪儿，再把这份衡量转化成模型能力的提升。
Mercor 就是在这个背景下长起来的。我们第一个大项目是 Deep Research——我想那大概算是第一个真正出圈的 RL agent。之后我们和所有前沿实验室一起把规模急剧放大，成为所有头部实验室的主力 agentic 数据供应商，同时也服务所有头部的应用层公司，从 Harvey、Sierra、Cognition 到 Ramp。
而过去 12 个月最让人兴奋的，是 agentic 数据这套范式里的 RLVR（可验证奖励的强化学习，说白了就是用能自动判分的标准来给模型打分做 RL）已经进化到把 RL environment 也包进来了——那些内容极其丰富的应用和世界，能教会 agent 怎么使用我们每天在笔记本上用的所有工具。所以我今天会讲这个，当然还会讲：这套原本诞生在前沿实验室的技术，现在是怎么扩散到应用层的。
[2:53] Brendan Foody
layer and all of the products that all of you are building as you work on your company. So, high level on what an RL environment is is that it includes three parts. The first part is the worlds. So, this includes all of the messages, slides, docs, sheets, etc. that correspond to everything you would have in a real project or company that you're working on. The second part is the apps which is high fidelity clones of popular applications, Salesforce, ServiceNow, Microsoft 365, etc. that agents can interact with via MCP, CLI, or Kua. And then the third part is the tasks where we have prompts and verifiers. Verifiers could be rubrics or unit tests that can be used either for eval or training. And the barrier for frontier labs to automate everything that you can do on your laptop using Claude is how do they cover the full distribution of all of the worlds, all of the apps, and all of the tasks in the economy. And so, there's been this enormous scale out in order to do that. Um, where humans have been really central to how we build these environments, obviously with models in the loop meaningfully. And so I put a graph here of the amount of expert hours that um, of throughput in from our talent network over the last 24 months. Um, and it's a a pretty crazy trajectory with respect to um, 2.5 million hours um, in Q2 alone with growth sort of accelerating on the amount of expert time uh, used to build out all of these environments. The reason being of course as I mentioned we need to scale out the environment distribution across every category in the economy. Many of you might know GDP val where there's 205 domains in the Bureau of Labor Statistics across all the different jobs, but then you have to think through how do we have all of the apps corresponding to all of those jobs, all the different scenarios, all the tasks.
……扩散到应用层，扩散到在座各位在自己公司里做的那些产品上。
先从高层次说，一个 RL environment 由三部分组成。第一部分是 worlds（世界），也就是所有的消息、幻灯片、文档、表格等等——对应你在一个真实项目、一家真实公司里会接触到的一切。第二部分是 apps（应用），也就是高保真复刻的主流软件，Salesforce、ServiceNow、Microsoft 365 等等，agent 可以通过 MCP、CLI 或者 CUA（computer use agent，直接像人一样操作图形界面的智能体）跟它们交互。第三部分是 tasks（任务），里面包含 prompt 和 verifier（验证器，判断模型做得对不对的那套东西）。verifier 可以是 rubric（评分细则），也可以是单元测试，既能拿来做 eval，也能拿来做训练。
前沿实验室想把你在笔记本上用 Claude 能做的一切都自动化，卡点就在于：怎么覆盖整个经济体里所有 worlds、所有 apps、所有 tasks 的完整分布。为了做到这一点，出现了一次规模极其庞大的扩张。人在这里面非常核心——当然模型也实打实地参与在流程里。
我这里放了一张图，是过去 24 个月我们人才网络投入的专家工时吞吐量。这条曲线相当疯狂：光第二季度就有 250 万小时，而且用于构建这些 environment 的专家时间还在加速增长。
原因刚才说了：我们得把 environment 的分布铺满经济体里的每一个门类。很多人可能知道 GDPval，它基于美国劳工统计局的职业分类，覆盖了 205 个领域、各行各业。但你还得继续往下想：这些工作对应的所有应用、所有场景、所有任务，怎么才能全都造出来。
[4:49] Brendan Foody
Is this enormous build out. Only humans can measure the frontier in most domains, not every domain. There are rare exceptions like math where you have a really clean simulation environment and so uh, the model's able to learn from whether it got the right answer, but in most domains like building a slide deck uh, the model has an incredibly hard time identifying reliably where it made its own mistake. It's as if you would be asking a human to grade their own homework. And so that's why it's really valuable to have a human create a rubric similar to how a professor would create a rubric to grade an essay or a TA would grade that slide deck. Similar to the way that a lot of us learn it's in large part from the feedback we got from those around us rather than uh, purely plugging things into a calculator or clean simulation. Um, and then building these verifiers is hard cuz anytime you're building the slide deck you need to understand the full problem space of what are the 10 different slide decks that, you know, could be a good path to go down? What are the dozens of mistakes you could possibly make? And how do you build a comprehensive verifier that captures this full solution area of what's possible? And so, what I'll walk through is a sample RL environment.
这是一次极其庞大的建设。在大多数领域，只有人才能衡量前沿在哪儿——不是所有领域，也有少数例外，比如数学，它有一个非常干净的模拟环境，模型能从「答案对不对」里直接学到东西。但在大多数领域，比如做一份 slide deck（幻灯片），模型极难可靠地判断自己到底哪里错了。这就好比让一个人给自己的作业打分。
所以让人来写 rubric 就特别有价值——就像教授会写一份评分细则去批一篇论文，或者助教用它来评那份幻灯片。这其实跟我们大多数人的学习方式很像：我们主要是从周围人的反馈里学，而不是纯靠把东西丢进计算器或者某个干净的模拟器。
而且构建这些 verifier 很难。你每做一份幻灯片，都得吃透整个问题空间：有哪十种不同的做法都可能是条好路子？可能犯的几十种错分别是什么？怎么造一个足够全面的 verifier，把这一整片可能的解空间都罩住？
下面我就带大家看一个 RL environment 的样例。
[6:01] Brendan Foody
Excuse me, to also break this down for all of you. Um and part of the reason that this is so cool, which I'll get to in a moment, is that we developed a lot of this technology in collaboration with the labs. These are, of course, ones that we have open-sourced and published to the world, but now that's all starting to get disseminated to the application layer companies that are building and owning their own intelligence. As they realize that the three core pillars of their AI strategy are their compute, their algorithms or researchers, and the data sets they build. And data's often the most differentiating factor. And so, this is one that we published, um as a legal environment, where we have lawyers from top law firms like Latham and Watkins write out a scenario of a real project that they worked on in their big law job. And then they create a full outline for a data room that corresponds to all of the different uh messages, emails, files, size of files. I cut off the full data room cuz it's it's very extensive. Um and of course, there's a lot of model in the loop with how they effectively populate this. Similar to how a software engineer now should not be coding by hand entirely themselves, they should probably be orchestrating agents in how to do this very productively. Um and then we render that data room into the apps, the clones of Google Workspace you can see in this scenario, and have prompts uh to roll out model trajectories against this. And so, in this one, it's evaluating the maximum total liability for Star Tanker uh Tankers International Limited compared to Cooper Jefferies Energy Corporation under the Oil and Petroleum Act, uh considering all of the context uh from this real scenario in the data room. And then as I mentioned, similar to how a professor would create a rubric to grade an essay, they have these key rubric criteria that correspond to what are the characteristics of a of a accurate model
不好意思——我把它拆开讲给大家听。这件事之所以酷，有个原因我一会儿说：这套技术里有很多是我们和实验室合作开发出来的。我今天要展示的当然都是已经开源、公开发布的部分，但现在这一切正开始扩散到应用层公司——那些想构建并拥有自己智能的公司。因为他们意识到，AI 战略的三根支柱是：算力、算法（或者说研究员），以及自己构建的数据集。而数据往往是最具差异化的那一环。
这是我们发布过的一个法律领域 environment。我们请来自 Latham & Watkins 这类顶级律所的律师，写出一个他们在大所工作中真实做过的项目场景。然后他们为这个场景做一份完整的 data room（尽调资料室）大纲，对应里面所有的消息、邮件、文件、文件大小等等。完整的 data room 我截掉了，因为实在太长了。当然，他们真正往里填内容的时候有大量模型参与——就像今天的软件工程师不该再全靠手敲代码，而应该去调度 agent 高效完成工作一样。
然后我们把这个 data room 渲染进 apps 里，这个场景里你看到的是 Google Workspace 的复刻版，再配上 prompt，让模型在上面跑出 trajectory（轨迹，即模型完成任务的一整串操作过程）。这一题是：在《石油与成品油法案》下，评估 Star Tankers International Limited 相对于 Cooper Jefferies Energy Corporation 的最高总责任额是多少，同时要结合 data room 里这个真实场景的全部上下文。
然后就像我说的，跟教授写 rubric 批论文一样，他们会给出这些关键评分项，对应「一个准确的模型回答应该具备哪些特征」。
[7:56] Brendan Foody
response. And making sure that these rubric criteria avoid reward hacking and effectively align with the uh when you roll out 100 trajectories, making sure all of those scores are accurate is incredibly technically challenging. And so there's an enormous amount of research, agenda quality control, training on the data, etc. that goes into how you solve that problem and then ultimately produce these high-quality verifiers and leaderboards um that give you an aggregate model score across how well all of the different models are um doing on a particular domain. And as we can see, one of the big changes over the last few months is that GLM 52 and Chimera K3 are on the leaderboard. And so that is a huge opportunity for all of you because that gives us the foundation to actually achieve frontier intelligence and all of the specific applications uh and verticals that you're focusing on um that's not too far away. And so to give a little bit of context on what that looks like, um I'll share an example of post-training on Apex Agents, which is the data set that I uh or the sample I just showed before, where we have 1,800 tasks in this example. This was post-training run of GLM 47, but we're redoing a lot of them for Chimera K3, so we'll have updated results for you all soon. Where you can see the jumps just on 1,800 tasks with about 500k in compute are pretty dramatic. Um corporate law going from 4.7% to 26.6%, um but notice that this is just Apex Agents data set we gave it, and it actually generalized incredibly well to GDP valve and uh Apex V1, which doesn't have these data rooms.
……一个准确的模型回答应该具备哪些特征。而要保证这些 rubric 评分项不会被 reward hacking（模型钻规则空子骗高分）钻空子、并且在你跑出 100 条 trajectory 时这 100 个分数都打得准——这在技术上极其有挑战。所以背后有海量的研究工作、agentic 质检、在数据上做训练等等，才能把这个问题解掉，最终产出这些高质量的 verifier 和排行榜——排行榜会给出一个综合的模型得分，看各家模型在某个特定领域上表现如何。
可以看到，过去几个月一个很大的变化是，GLM-5.2 和 Kimi K3 也上了这个排行榜。这对在座各位是个巨大的机会，因为这意味着我们已经有了基础，能在你们各自聚焦的具体应用和垂直领域里真的做出前沿智能——而且那一天并不遥远。
给大家一点具体感受。我分享一个在 APEX Agents 上做 post-training 的例子——就是我刚才展示的那份样本所属的数据集，这个例子里有 1,800 个任务。这次 post-training 跑的是 GLM-4.7，不过我们正在用 Kimi K3 重做很多实验，很快会给大家更新结果。你能看到，仅靠 1,800 个任务、大约 50 万美元的算力，提升就相当猛：公司法从 4.7% 涨到 26.6%。但请注意，我们喂进去的只有 APEX Agents 这一个数据集，而它居然极好地泛化到了 GDPval 和 APEX v1 上——后两者里根本没有这些 data room。
[9:42] Brendan Foody
Even just seeing nominal gains and some other benchmarks as well. And so, we're doing a lot of this work of working with customers like Harvey, who I know will present on stuff later to help build out the environments corresponding to their specific domain so that they can build frontier intelligence within that. And I think Andrew talked about how Cursor was a great first example of how an application layer company could build a industry-leading model that, you know, built an enormous amount of value for their customers. And I believe that over the next 12 months, there is going to be dozens of examples just like that where companies own their own intelligence and that is the key source of the modes that they're building. And actually, Josh and I talked about this the other day as well. Um a couple of examples of ways to curate high-quality data sets. The general three that we see most that I'm happy to talk about and send people links to is first by task is the most common. Where people would say, "I really like this data shape of environments in law and we will pay $2,000 per task to scale this up." And as an example, certain frontier labs might buy 50,000 tasks a month from us.
甚至在另外一些 benchmark 上也看到了一定幅度的提升。
所以我们现在做了大量这类工作，和 Harvey 这样的客户合作——我知道他们稍后也会来分享——帮他们把自己所在领域对应的 environment 建起来，好让他们能在那个领域里做出前沿智能。
我记得 Andrew 讲过，Cursor 是一个很好的先例：一家应用层公司做出了行业领先的模型，为客户创造了巨大价值。我相信未来 12 个月里会冒出几十个同样的例子——公司拥有自己的智能，而这就是他们护城河的关键来源。前两天我和 Josh 也聊到过这一点。
再说几个策划高质量数据集的方式。我们见得最多、也很乐意展开讲、可以给大家发链接的主要有三种。第一种也是最常见的：按任务计价。客户会说「我很喜欢法律领域这种形态的 environment，我们愿意每个任务付 2,000 美元，把它扩起来」。举个例子，某些前沿实验室一个月可能从我们这儿买 5 万个任务。
[11:03] Brendan Foody
And so, it tends to be pretty dramatic scale. And these tasks would generally be very complex. Some would even take humans up to a month to complete that given task. Sometimes it would take just a few hours. And these would be sort of custom per task pricing. Second is off-the-shelf data where we have we've invested hundreds of millions of dollars in building our own data sets that we sell to multiple customers. All these new neo labs are generally airing more towards off-the-shelf data because it doesn't make sense for 10 different labs to all be building their own uh data sets. Um there's a lot of value to building something once that can uh then be applied to everyone. And then the final which um we see a little bit of, but is uh less of our focus anymore, is just providing um the experts so that customers are able to uh or organize the experts on their own um in just an hourly model. Um so we we do a little bit of that when people like that was how Harvey got started with us hiring uh some lawyers, um but it generally moves towards more of these uh scaled offerings of data over time. So, that's a little bit of the background of how to build RL environments, what they are, and I'm really excited about all this technology that we have uh that has previously been limited to the frontier labs all making its way to all of you. And so, happy to answer any questions um about that.
所以量级往往相当夸张。而且这些任务通常非常复杂，有的甚至要人花上一个月才能做完一个，有的可能只要几个小时。这一类是按任务定制报价。
第二种是现成数据（off-the-shelf data）。我们已经投了好几亿美元构建自己的数据集，然后卖给多个客户。那些新兴的 neo lab 普遍更倾向买现成数据，因为让 10 家实验室各自重复造一遍同样的数据集并不划算。做一次、所有人都能用，这里面价值很大。
第三种我们现在做得比较少、也不再是重心了，就是单纯提供专家，让客户自己去组织这些专家，按小时计费。我们还是会做一点点——比如 Harvey 最早跟我们合作，就是从雇几位律师开始的——但随着时间推移，一般都会走向前面那两种规模化的数据交付方式。
以上就是关于 RL environment 是什么、怎么造的一点背景。让我特别兴奋的是，这套原本只有前沿实验室才用得上的技术，现在正一步步交到你们手里。所以，很乐意回答大家的任何问题。
[12:34] Brendan Foody
Sweet. Go ahead.
好嘞。请讲。
[12:37] 观众提问
Hey, uh um I'm Ali from Astro Guide. My question is it's kind of open-ended, but simple. How do you price data? Like how do you value data?
嗨，我是 Astro Guide 的 Ali。我的问题有点开放，但很简单：你们怎么给数据定价？或者说，你们怎么衡量数据的价值？
[12:47] Brendan Foody
So, there's a so many different ways. I mean, the most natural would be our customers care about model improvement, right? And so, our customers have a given goal of they want to you know, be at the frontier on a given leaderboard. And so, we're able to work backwards from how much is that worth to them and how much should we charge uh per task, how many tasks do we think would get them to that goal. And so, when we think about uh a company like Nvidia, they're probably willing to pay, you know, a billion dollars to have a frontier open-source model. And so, there's a lot of complexity of like how do we you know, price all the different ingredients that go into um making that happen. The other way that we price when it lens we look at it through is also our cost structure to make them where of course when we have a task that takes 10 hours of human time and we're paying the human $150 an hour there might be a $1500 cost basis and so then it becomes a question of what margin do we want to run on top of that based on how differentiated and frontier that specific task is. But it's it's super wide range. Like we have tasks that range from $50 to $10,000. So.
方式实在太多了。最自然的一种是：我们的客户在意的是模型有没有变强，对吧？客户会有一个明确目标——他们想在某个排行榜上冲到前沿。那我们就可以倒推：这件事对他们值多少钱、我们每个任务该收多少、需要多少个任务才能把他们送到那个目标。
比如说像 Nvidia 这样的公司，为了拿到一个前沿的开源模型，他们大概愿意付十亿美元。那这里面就有很多复杂的问题：促成这件事的各种原料，我们该怎么一一定价。
另一个我们会用的视角是自己的成本结构。当一个任务要占用人 10 个小时、我们付这个人每小时 150 美元，成本基础就是 1,500 美元。接下来的问题就变成：基于这个具体任务有多差异化、有多前沿，我们想在上面跑多少毛利。
但区间真的非常宽。我们手上的任务从 50 美元到 1 万美元都有。
[14:05] 观众提问
How do you think about data quality? You mentioned like you know utilize human expert to label data and how do you how do you compare the human label data and you know the frontier lab you know frontier alliance the judgment data? How do you compare them to from your opinion?
你们怎么看待数据质量？你刚才提到用人类专家来标注数据，那你们怎么去比较人类标注的数据，和前沿实验室那边那种偏好判断类的数据？在你看来这两者该怎么比？
[14:23] Brendan Foody
So first question was how do we think about quality? Second one was sort of how do we compare the like judgment of preference labels to the auto graders?
所以第一个问题是我们怎么看待质量，第二个问题大概是：偏好标注这类人的判断，和 auto grader（自动评分器）之间怎么比，对吧？
[14:33] 观众提问
Yeah.
对。
[14:33] Brendan Foody
So the core spot which is generally when people say data quality they're referring to two things. First is realism and secondly is accuracy of verifiers. On realism it's just like they want to automate everything in the economy that corresponds to corporate law in this case right? And so it's like how do we make sure that this actually reflects the real distribution of what we would see in a real lawyer's environment. And that's one of the reasons that experts create outlines and help to guide all the processes of the data curation. Realism of the environment the apps the tasks everything is incredibly important and and also granularly understanding the taxonomy that drives that realism across the entire distribution that you're looking for. The second part of it relates to the other way people think about quality, which is the accuracy of the verifiers. Cuz the way that you would train one of these models is you might roll out 100 trajectories of Kimi K3 and then use this rubric to score all of those trajectories. And as you can imagine, there's like so many different paths that a model can go down. And so you want to make sure that the way this rubric is doing the scoring is the same as if we were to just have human stack rank those 100 trajectories. And so what we do for that is a process called trajectory analysis, where we roll out 10 trajectories of the model that we're focused on improving, um, and then score all of those, and have some combination of agentic quality control systems and some human review go through to make sure that, um, all of the scores align with, uh, the goals.
核心是这样：一般人说「数据质量」的时候，指的是两件事。第一是真实性（realism），第二是 verifier 的准确性。
先说真实性。客户想自动化的，是经济体里对应「公司法」这一整块的工作，对吧？那问题就是：我们怎么保证这份数据真的反映出一个真实律师的工作环境里会看到的真实分布。这也是为什么要让专家来写大纲、并全程引导数据策划的整个流程。environment 的真实性、apps 的真实性、tasks 的真实性，全都极其重要；同时还要非常颗粒地搞清楚，是什么样的分类体系在支撑你要覆盖的那一整片分布的真实性。
第二部分对应人们理解质量的另一个角度：verifier 的准确性。因为训练这类模型的做法是，你可能会跑出 Kimi K3 的 100 条 trajectory，然后用这份 rubric 给这 100 条打分。你可以想象，模型能走的路径实在太多了。所以你要确保这份 rubric 打出来的分，和「直接让人把这 100 条 trajectory 从好到坏排个序」的结果是一致的。
为此我们有一套流程叫 trajectory analysis（轨迹分析）：把要重点提升的那个模型跑出 10 条 trajectory，全部打分，然后用 agentic 质检系统加上一部分人工复核，逐条确认所有分数都和目标对得上。
[16:15] Brendan Foody
Um, and sometimes you can also use, uh, human feedback evals or preference labels to as a eval for your auto grader, um, is the other way related to that that you're able to solve for it. Go ahead.
另外有时候你也可以拿人类反馈的 eval、或者偏好标注，来当作你 auto grader 的评测集——这是跟这个相关的另一条解法。请讲。
[16:30] 观众提问
Um, how much do you think like synthetic data generation plays into all of this, especially like creating these large data rooms?
你觉得合成数据（synthetic data）生成在这整件事里占多大分量？尤其是在造这些庞大的 data room 的时候。
[16:38] Brendan Foody
So the fascinating thing is I think that there's been a lot of misinterpretation of what people mean when they say synthetic data because, like, RLVR is a bet on synthetic data. It's basically let's roll out a bunch of synthetic model trajectories rather than having the humans write the SFT, and then let's score all of them, and let the models learn from all of these like, uh, synthetic model trajectories. So I think that's the first way that synthetics get used. The second way is that models play giant role in the way that we populate environments and create tasks in the same way that, um, a lawyer that is writing a legal memo should definitely be using Claude or ChatGPT to do that. The experts that are building out these data rooms should definitely be using Claude, ChatGPT, or whatever model to help them do that. And there's a lot of ways that the model can make them more efficient. But the reason that humans are still an essential component of the process that's incredibly differentiated is that you need humans almost definitionally to measure what is beyond the frontier of the model capabilities. Like the models you you can't just tell the model like come up with the legal environment and then like tell me which of your legal memos are like good and bad. It's super noisy and there's not like clear signals associated with that.
有意思的地方在于，我觉得大家说「合成数据」的时候，含义被误解得挺厉害。因为 RLVR 本身就是在押注合成数据啊——它本质上就是：与其让人去写 SFT 数据，不如让模型自己跑出一大堆合成的 trajectory，然后把它们全部打分，让模型从这些合成的模型轨迹里学习。这是合成数据被用到的第一种方式。
第二种方式是，模型在我们填充 environment、创建 task 的过程中扮演了巨大角色——就像一个要写法律备忘录的律师，今天肯定应该用 Claude 或者 ChatGPT 来写；同样地，那些搭 data room 的专家，也肯定应该用 Claude、ChatGPT 或者别的什么模型来帮忙。模型能在很多环节让他们效率高得多。
但人之所以仍然是这个流程里不可或缺、而且极难被替代的一环，原因在于：几乎可以说是按定义来讲，你必须靠人才能衡量「什么东西超出了模型能力的前沿」。你不可能直接跟模型说「你来造一个法律 environment 吧」，然后再说「你自己告诉我你写的哪几份法律备忘录好、哪几份差」。那噪声极大，根本得不到清晰的信号。
[17:57] Brendan Foody
You need something that has capabilities beyond the frontier of that model to do so reliably.
你得有一个能力超出那个模型前沿水平的东西，才能可靠地做到这件事。
[18:05] 提问者
Thank you for doing this. Um my question is RL environments seem like they're all the rage now and maybe have been for about a year. I had I hadn't really been hearing about them prior to that and it was all human expert labeling. And so kind of why why why is it all about RL environments now? Is that the most relevant thing for application companies to be thinking about? And is there any something after RL environments?
谢谢你来做这场分享。我的问题是，RL environment（强化学习训练环境，说白了就是给模型搭一个能动手操作、又能被打分的仿真工作场景）现在好像特别火，大概火了有一年了。在那之前我基本没怎么听人提过这个词，大家谈的全是人类专家标注。所以我想问，为什么现在什么都在讲 RL environment？这是应用层公司眼下最该琢磨的事吗？还有，RL environment 之后又会是什么？
[18:29] Brendan Foody
Um so I'll I'll start with why it's become the rage and maybe some of the differences also between the deep research paradigm and the like envi- the sort of environments paradigm as we saw it in 2025. And then I'll talk about looking forward what we see evolving in the data landscape. Specifically, I think that the reason the deep research environments were the first was because like deep research had tool use with search. So search was the tool in the environment that the model would work in. But the experts would not necessarily be populating apps. So it was sort of a lighter version of an RL environment where they would just like create rubrics corresponding to this. And again, like I only talk about this stuff cuz it's a couple of years old at this point, so it's no longer uh super confidential. And then for the like trend of apps in 2025, I think that really became giant because people realized that the primary bottleneck to making the models useful was how they started to use both all the context in the code base and all of the tools on our laptops, right? And so if we want this in the user distribution of usage, then we need to get it in the data distribution that the models are learning from.
我先说说它为什么会火起来，顺带讲讲 deep research 那套范式，和我们在 2025 年看到的 environment 范式之间有什么不一样，然后再往前看，聊聊数据这块我们判断会怎么演变。具体说，我觉得 deep research 类的 environment 之所以最先出现，是因为 deep research 的工具调用只有搜索这一个工具——搜索就是模型在那个环境里唯一要用的东西，专家并不需要真的去把一堆应用填满。所以那算是 RL environment 的一个轻量版：专家只要针对任务写出对应的 rubric（评分细则，就是一条条列清楚什么算做对、什么算做错）就行了。另外多说一句，我敢公开讲这些，是因为它已经是两三年前的事了，早就不算什么机密。至于 2025 年那波「拿真实应用当环境」的趋势，我觉得它之所以爆发，是因为大家意识到：模型好不好用，最主要的瓶颈在于它怎么把代码库里的全部上下文、还有我们笔记本上的各种工具真正用起来。所以，如果我们希望这些东西出现在用户实际的使用分布里，就得先让它出现在模型学习的数据分布里。
[19:42] Brendan Foody
Uh and so there's going to continue be to be this giant scale up of diversity across all three of these categories on a going forward basis, but there's going to be some changes um to maybe name two of those changes that we're thinking about the most. The first one is ultra long horizon. Like right now agents mostly aren't trained to do things that are over 10 hours, and we need to start building tasks for things that might take a human 100 hours or even 1,000 hours to do. And so that's going to be a giant shift. And then the other large shift that we're seeing is introducing virtual co-workers, which corresponds to that. Um like one my favorite questions to ask people when they're thinking about their data distribution is what percentage of tasks that they do in their job require interacting with other people. Uh and most people would say like 60% or 70%. Some people say a lot more, some people say a little bit less. Um but then if you map that on to what percentage of evals measure how well the models can interact with other people, it's like 1%, maybe Tau bench has a little bit of this. Um and so there's this giant realism gap associated with how you actually measure how well agents engage in social interaction um throughout uh all of the different people and other agents that they need to work with uh in their jobs.
所以往后看，这三大类环境的多样性还会继续大规模扩张，但中间也会有一些变化。我说两个我们想得最多的。第一个是超长时间跨度。现在的 agent 基本没被训练去做超过 10 小时的事，而我们得开始构造那种人类要花 100 小时、甚至 1000 小时才能做完的任务，这会是一次巨大的转变。第二个大转变是引入「虚拟同事」，跟前面那点是配套的。我最喜欢问别人的一个问题是：你工作里有多大比例的任务需要跟别人打交道？大多数人会说 60% 或 70%，有人说更多，有人说少一点。但你再对照着看，有多大比例的 eval（评测集）在衡量模型跟人协作的能力？大概 1%，可能只有 Tau-bench 沾了一点边。所以这里存在一个巨大的「真实感落差」——agent 在工作里要跟形形色色的人、还有别的 agent 打交道，可我们几乎没有办法真正衡量它在这种社交互动上做得怎么样。
[21:07] 主持人
Good.
好。
[21:08] 提问者
You talked about rubric generation, which is like is it like a bespoke access of verify that you put task? And from what I understood, that's like bottleneck by experts. Have you found any success of like being able to scale that up with your models giving you like some heuristic or even like post any models for
你刚才讲到 rubric 的生成，这是不是相当于给每个任务定制一套专门的验证标准？照我的理解，这一块的瓶颈就卡在专家身上。你们有没有成功地把它规模化过——比如让模型先给出一些启发式的判断，甚至干脆让模型来接手一部分……
[21:26] Brendan Foody
We have found that you can make it a lot more efficient if you have a like AI copilot that's able to work with the expert in creating the task and the verifier. Um so, the expert can talk to the trajectory and understand exactly what's happening and where it's going wrong. The challenge is just that if you're trying to improve FABLE, FABLE cannot reliably write out the like rubric criteria for where it's making mistakes. It might get like half of them right and half of them wrong, and that amount of noise is unworkable from a training standpoint. Um and so, that's the reason that the the process that requires humans the most is the task creation. Uh like a lot of the environments, we can get like use a lot of synthetic. Uh it's helpful to have humans write the outlines they're familiar with the environment and and uh and they're grounded in reality, a realistic distribution, but with the task that's those tend to really require humans. Um with with rare exceptions, um in code or if you're sort of distilling from like if you have a model that's worse than Kimikaze 3, then it can definitely learn from tasks Kimikaze 3 is creating. So, there are there are some exceptions if you're doing it that way.
我们发现，如果有一个 AI copilot 陪着专家一起去创建任务和 verifier（校验器，负责判定模型跑出来的结果到底对不对），效率能高很多。专家可以直接对着 trajectory（模型跑一遍任务留下的完整操作轨迹）提问，搞清楚每一步到底发生了什么、哪儿出的错。难点在于：如果你要提升的是 FABLE 这个模型，FABLE 自己没法可靠地写出「我是在哪里犯的错」这种 rubric 条目，它可能对一半错一半，而这种程度的噪声，从训练的角度看是完全没法用的。所以最离不开人的环节其实是任务创建。environment 本身有不少可以靠合成来解决——让熟悉那个环境的人来写大纲会更好，因为他们对现实有感觉，给出的分布更贴近真实——但任务这块是真的很需要人。当然也有少数例外，比如代码领域；或者你在做蒸馏，如果你手上的模型比 Kimi K3 弱，那它完全可以从 Kimi K3 出的题里学到东西。所以走这条路的话，确实存在一些例外。
[22:47] 提问者
Hey, this is uh Nikhil from uh Cribl. So, when we think about RL environments for certain provable domains like cyber defense or incident response, where there's the model is or the agent is trying to find a flaw in an existing system, Do you use uh humans for just authoring that environment or setting it up or do you also use that for grading? Is there Is there a way to scale that up?
你好，我是 Cribl 的 Nikhil。我们在想，RL environment 用在那些结果可验证的领域时——比如网络安全防御、事故响应，也就是 agent 要去找出现有系统里的漏洞——你们是只用人来编写、搭建这个环境，还是打分（grading）也靠人？这块有办法规模化吗？
[23:11] Brendan Foody
I I actually think cyber is one where you don't necessarily always need humans for the verifiers cuz you can have an attacker and a defender agent. Um and uh I think you're right in saying that for cyber you can have humans more so or or architect what is a realistic like environment um and sort of set up the environment um cuz you do need a lot of diversity. Uh but then it's less human intensive with respect to uh building verifiers.
我其实觉得网络安全恰恰是那种不一定非要人来做 verifier 的场景，因为你可以让一个攻击方 agent 和一个防守方 agent 互相对打。至于你说的另外一点，我觉得没错：在安全领域，人更多是去设计什么才算一个真实的环境、把环境搭起来，因为你确实需要足够的多样性。但在构建 verifier 这一环上，对人的依赖就小得多了。
[23:43] 提问者
Thanks.
谢谢。
[23:46] 提问者
How can you tell when you're limited by the base model?
怎么判断自己是被基座模型（base model）卡住了？
[23:50] Brendan Foody
What do you mean by that?
你这话具体是指什么？
[23:51] 提问者
Well, ostensibly you're using the same data set for all these models here and they somewhat land around the same final performance on this list here. But maybe if you try to smaller model, which is maybe a good place to start, it would land much lower. Um what is the cause there? Is it just parameter count or
就是说，表面上你在这些模型上用的是同一份数据集，而它们在这张榜上最后跑出来的成绩都差不多。但如果你换一个更小的模型试试——这可能是个不错的切入点——它的成绩可能会低很多。那这背后的原因是什么？纯粹是参数量的问题，还是……
[24:08] Brendan Foody
So the parameter count will definitely play a role in how effectively the model does insofar as how trainable it is. I think that um the main thing to look at is generally the gap between the like pass at 16 and the pass at one. If you have a model where you roll out 16 trajectories and it gets all of them totally wrong, the it's sort of like hopeless that the model is going to learn from that um for the most part. Maybe you roll out another 100 trajectories and it gets one of them right. Um versus if you have um the ideal case is that you have um pass at one it fails, but then pass at 16 when you roll out 16 trajectories it gets it right once or twice, and then the model is able to learn very effectively from that. So, that's generally the heuristic we would use for how strong the base model needs to be to effectively learn from a given data set. Cool. Maybe Oh, final question really quick.
参数量肯定会影响模型学得有多好，也就是它到底有多「可训练」。但我觉得主要该看的，是 pass@16 和 pass@1 之间的差距。如果你让模型 rollout（同一道题让模型独立跑很多遍，每一遍留下一条完整的解题轨迹）16 条轨迹，结果全都错，那基本上就没戏了——模型很难从这里面学到东西。当然你也可以再 rollout 100 条，看看能不能蒙对一条。理想情况是：pass@1 是失败的，但 rollout 16 条里能对上一两次，这时候模型就能学得非常有效。所以「基座模型得多强，才能从某份数据里真正学到东西」，我们基本就用这个经验法则来判断。好，那可能……哦，最后再快速问一个。
[25:19] 提问者
What advice do you have for companies as they partner with you on data that they should rely on themselves as part of their RL post-training versus relying What's the complementary
对于那些在数据上跟你们合作的公司，你会建议它们在 RL post-training（后训练，指预训练之后再用强化学习等方式，把模型调成真能干活的样子）里，哪些部分必须自己扛、哪些部分交给外部？两边怎么互补？
[25:32] Brendan Foody
Well, I think this is why a giant portion of our business is custom data, where it's like we have teams that are siloed and fully exclusive to critical customers to make sure that we build out the best data sets in the world that they own. And that allows them to maintain their competitive advantage associated with this, while also benefiting from all of the infrastructure that we've built. And I think there are some companies that try to like build out all of the talent network and infrastructure in-house, but I think if you look at what the frontier labs do and the best models do, it's a pretty good indication that there's so many economies of scale from working with a partner that has all of these economies of scale, the platform, the talent network, etc. So, anyways, thanks for all having me.
这也正是为什么我们业务里有很大一块是定制数据：我们会派出完全隔离、只服务某个关键客户的专属团队，确保给他们做出全世界最好的数据集，而且这份数据的所有权归他们。这样他们既守住了由此而来的竞争优势，又能享受到我们已经搭好的全部基础设施。确实有些公司想把人才网络和基础设施全都自建，但你看看前沿实验室、看看最好的那些模型是怎么做的，就很能说明问题：跟一个已经具备规模效应的伙伴合作——平台、人才网络等等都是现成的——能省下大量成本。总之，谢谢大家今天请我来。
[26:18]
[applause]
[applause]