Multiplayer agentic engineering — Arjun Singh, Superconductor

频道: AI Engineer
视频: https://www.youtube.com/watch?v=OL7kfezynJM
原文语言: en
统计: 共 20 轮

[0:01]

[music]

[音乐]

[0:12]

All right. Hey everyone, I'm Arjun Singh. Today I'm going to talk to you about multiplayer agentic engineering or how to enable your whole team and your best agents to work together. If you go to the talks or go around the expo, you're going to see that a lot of people are talking about putting the agents at the center of everything. Makes sense, they're really powerful, they're really cool. But you don't see a lot of people talking about the people. Like this is all for us to make us our lives better, our more our more productive, whatever. And so we're going to really focus on how the people fit into these agentic workflows. Just a little bit about us first. So our team has worked together building software for over a decade. Um my co-founder Sergey and I, we met in the PhD program at Berkeley. I worked on robotics, he worked on computer vision. And during that we co-founded a company called GradeScope. Some of you may have used it. It's used by millions of students worldwide at thousands of universities, helping instructors grade their students' work. And um pretty much the entire team working on Superconductor used to work together on GradeScope. And so we've had a team that's worked together productively from first user to acquisition, working on something new together again.

好，大家好，我是 Arjun Singh。今天我想聊聊「多人协作式的 agent 工程」，也就是怎么让你的整个团队和你最好的 agent 一起干活。你去听这些演讲、逛一圈展台就会发现，很多人都在讲把 agent 放到一切的中心。这也说得通，agent 确实很强、也确实很酷。但很少有人在讲「人」。可这一切归根到底是为了我们自己——让我们的生活更好、效率更高，不管是什么。所以今天我们会重点讲人到底怎么嵌进这些 agent 工作流里。先简单介绍一下我们。我们团队一起做软件已经超过十年了。我和联合创始人 Sergey 是在伯克利读博时认识的，我做机器人，他做计算机视觉。读博期间我们一起创办了一家叫 GradeScope 的公司，在座可能有人用过。全球几千所大学、几百万学生在用，帮老师批改学生的作业。而现在做 Superconductor 的这拨人，基本上就是当年 GradeScope 的原班人马。所以我们这个团队从第一个用户一路做到被收购，一直高效地在一起共事，现在又一起做一件新的事。

[1:15]

And I think it's kind of an interesting experiment because you know, over the past year we've very aggressively integrated agents in our workflows and we've kind of surfaced all the different bottlenecks and friction points that come up and how to do that productively and keep collaborating the way we used to but with the new power of agents. So today I'm going to talk to you about how, you know, we we the lessons we learned from kind of solving those friction points and solving those bottlenecks. And in the talk description I mentioned five lessons and I'm going to be an engineer and start from zero and add a sixth one in there. Um the first one I'm going to start with is just to be model and harness agnostic. So there's a few reasons for that. The best model and harness can change weekly. It could change cuz a new one comes out. It could change cuz the best one got taken away. Um things happen and you don't want that to disrupt your entire team's flow. The other thing is that open weight models are actually pretty good now. We've been really happy with GLM 5.2. They're much cheaper. Um and you want to be able to kind of explore with them and integrate them without again having to change your entire workflow.

我觉得这其实是个挺有意思的实验。因为过去一年里，我们非常激进地把 agent 塞进了自己的工作流，把各种瓶颈和摩擦点全都摸了一遍——怎么才能既高效，又能保持我们原来那种协作方式，同时还享受到 agent 带来的新能力。所以今天我要讲的，就是我们在解决这些摩擦和瓶颈的过程中学到的东西。演讲简介里我说了五条经验，但我要发挥一下工程师本色，从 0 开始编号，再加一条，凑成六条。第一条：不要绑死在某个模型和某套 harness 上。原因有几个。最好的模型和 harness 可能每周都在变——可能是出了新的，也可能是原来最好的那个突然没了。这种事随时会发生，你不希望它把整个团队的节奏搅乱。另一个原因是开源权重模型现在其实已经相当能打了。我们用 GLM 5.2 就挺满意的，而且便宜得多。你得能够去试、去接进来，而不用因此推翻整套工作流。

[2:16]

The last thing I'll mention on this is that the incentives of the people selling you tokens aren't really aligned with yours. You're here for a reason. You're you're you're So you're doing things for a reason. You're trying to make your customers' lives better, make your product better, delight your customers. They want to sell you more tokens. And you might be happy to pay for as many tokens as it takes, but you don't want to pay for more than that. And so again, kind of being able to switch between things lets you stay in control of all of that. And so, um you know, as I go through the talk, I'll mention a couple of places where our product makes it easy for us, but whether you use us or not, I'm just going to leave things with you that I think are really important um for you to be able to work collaboratively effectively. So, the next one is to turn every human interface into an agent and human interface. So, you know, typically when people are working with coding agents, they're on their laptop, kind of stuck on that laptop. Nobody else can talk to that agent. So, the first place people go to kind of expose more interfaces for them is Slack. Cloud has a Slack bot. Coda has a Slack bot. We have a Slack bot. It's really cool. You can say, "Hey, Coda bot, do XYZ." It does it. Somebody else can talk to it.

最后再说一点：卖你 token 的那些人，跟你的利益并不完全一致。你来这儿是有目的的，你做这些事是有目的的——你想让客户过得更好，让产品更好，让客户满意。而他们想的是多卖你点 token。你可能确实愿意为「该花的 token」买单，但你不想花超出这个量的钱。所以说，能在不同方案之间随时切换，就能让你始终掌握主动权。接下来我讲的过程中，会提到几个我们产品让这件事变简单的地方，但不管你用不用我们的东西，我更想留给你的是我认为真正重要的那些点，能帮你们更有效地协作。第二条：把每一个人机界面，都变成「人 + agent」共用的界面。通常大家用编码 agent 的时候，人是坐在自己笔记本前面的，agent 也被困在那台笔记本里，别人没法跟它说话。所以大家第一个想到的扩展入口就是 Slack。Claude 有 Slack bot，Codex 有 Slack bot，我们也有。挺酷的，你可以说「嘿 Codex bot，去做某某事」，它就去做了，而且别人也能跟它对话。

[3:24]

But it's not enough. Cuz now we've taken it from trapped on somebody's laptop to kind of trapped in Slack. And a lot of work happened in Slack, so that's better than than than nothing, but certainly not all work happens in Slack. So, what we really wanted was to be able to work with the same session from every relevant interface. Could be Slack, could be our app, could be GitHub, could be elsewhere. And so, one possible flow is you start and collaborate on a session in Slack. And then maybe you continue in a kind of more engineer-focused environment in the desktop app or the mobile app. And then, you can finish it up in GitHub. And the important thing here is the exact same agent session. So, it's like the agent didn't forget what you did in one place in Slack when you go and talk to it from GitHub, it's the same session. It's got the same context. And the second lesson builds on top of that, which is to make the agent work visible and collaborative across the team. And so, obviously Slack makes it more collaborative. Um But here, we've got that kind of app view. And you know, Sergey made this ticket. I've been talking to the same ticket. Our growth person hopped in as well. So, you can kind of see at the top here all the different people that interacted with this. So, I can see who's getting notified about this session, who's seen it. Um that's especially important when you have work triggered by non-technical people.

但这还不够。因为我们只是把它从「困在某个人的笔记本里」变成了「困在 Slack 里」。确实有很多工作发生在 Slack，所以这比什么都没有强，但显然不是所有工作都在 Slack 里发生。我们真正想要的，是能从任何相关的界面接着操作同一个 session——可以是 Slack，可以是我们的 App，可以是 GitHub，也可以是别的地方。举个可能的流程：你在 Slack 里发起并协作一个 session，然后在桌面端或移动端 App 这种更偏工程师的环境里继续推进，最后在 GitHub 上收尾。关键在于，全程是同一个 agent session。也就是说 agent 不会因为你从 Slack 换到 GitHub 就忘了之前干过什么——同一个 session，同一套上下文。第三条是在这个基础上叠加的：让 agent 的工作在团队里可见、可协作。Slack 当然让协作更容易了。但这里你看到的是 App 视图：Sergey 建了这个 ticket，我也一直在跟这个 ticket 对话，我们做增长的同事也进来了。所以你在顶部就能看到都有谁参与过这件事，谁会收到这个 session 的通知，谁看过。当工作是由非技术同事发起的时候，这一点尤其重要。

[4:39]

Right? So, it's like, you know, our customer support person created a ticket. It's working really well. I want to understand like has this been vetted by an engineer or not? You can kind of see who's involved really easily. And then, if I'm reviewing something, I can just pop in and say like, "Hey, why did you do it this way?" And again, because it's the same agent session, I don't need to wait for Sergey to kind of get my notification on GitHub and respond to me. The answer to the question is almost certainly in this thread. I don't also don't want to read the entire thread. So, I can just ask ask the agent. Or how we most often kind of make the work visible is with artifacts. So, it doesn't matter where the work started or where it's finishing, the agent can show you the work it's doing as screenshot or video or other. And you can see it from everywhere. So, again, you don't have to worry about like, "Oh, where is that thing? I got to go to GitHub to see the image or got to go to Slack to see the image." It's just everywhere. Work is visible everywhere. Collaborate from anywhere. The third lesson

对吧？比如我们做客服的同事建了一个 ticket，跑得还挺好，但我想知道：这东西到底有没有工程师把过关？你一眼就能看清都有谁介入过。然后如果我在做 review，我可以直接插一句：「嘿，你当时为什么这么做？」而且因为是同一个 agent session，我不用干等着 Sergey 收到 GitHub 通知再回我——答案几乎肯定就在这个 thread 里。但我也不想把整个 thread 从头读一遍，那我直接问 agent 就行了。我们让工作可见最常用的方式是 artifact。不管这件事从哪儿开始、在哪儿结束，agent 都能用截图、视频或者别的形式把它做的事展示给你，而且你在哪儿都能看到。所以你不用再操心「那个东西在哪来着？我是得去 GitHub 看图，还是去 Slack 看图？」——它到处都在。工作到处可见，随时随地都能协作。第三条经验（编号第三）——

[5:36]

[clears throat]

[清嗓]

[5:36]

um I'm going to talk about here is to turn every external signal into code that your team can quickly evaluate. And I I to show this live, but the Wi-Fi is not quite there. So, I'm going to show you something from yesterday. But, what do I mean by external signal? So, it could be Slack conversation, could be a meeting you have with a customer, an onboarding call, a sales call. Could be an internal team meeting, could be something from Sentry or bug tracker, a bug report from a customer, an email, feature request. And right now what's happening is like all that stuff is already exists. It's in all those different systems. People hook them together with MCPs. So, now your coding agent can check the email or check Notion or whatever it might be. But, like how do you how how does it know what to work on, right? It's like it's still kind of stuck everywhere. And so, like some humans are involved in like kind of taking stuff from one place and telling it solve email number 48 or ticket number 6,000. But, that's still a lot of coordination. And so, what we do is we we have several different ways to automatically ingest these signals, prioritize what to what to do with it and and kind of act on them.

我要讲的是：把每一个外部信号，都变成你的团队能快速评估的代码。我本来想现场演示，但这里的 Wi-Fi 有点不给力，所以我给你们看昨天录的。什么叫「外部信号」？可以是一段 Slack 对话，可以是你和客户开的会、一次 onboarding 通话、一次销售电话；可以是内部团队会议；可以是 Sentry 或者 bug 追踪系统里的东西，客户报的 bug，一封邮件，一个功能需求。现在的现实是：这些东西全都已经存在了，只不过分散在各个系统里。大家用 MCP 把它们接起来，于是你的编码 agent 能去查邮件、查 Notion 或者别的什么。但问题是——它怎么知道该做哪件事？东西还是散在各处。结果就是得有人从这儿捞一条、告诉它「去处理第 48 封邮件」或者「去处理 6000 号 ticket」。这依然需要大量的人工协调。我们的做法是提供好几种方式，自动把这些信号吸收进来，判断优先级，然后直接动手处理。

[6:41]

And my favorite one, the most fun one, is what we call our meeting bot. And so, I'm going to switch switch over to my browser here for a second. And uh Okay. So, we've got a booth at the expo and we had the meeting bot running all day yesterday. So, this is a 4-hour meeting of a Google Meet. You just kind of invite the bot to Meet or Zoom or Teams or whatever it might be. And it listens all day. And it created all sorts of stuff as it was listening. If it finds existing work, it'll link to it. All right? So, it's not going to just like create new work if it's something already working on. Some of this is, you know, people testing the meeting bot out and telling it to do some weird things, um or interesting things, or just creative ideas. But, a lot of it's actually just like really good ideas that come out of people looking at what we're doing, asking questions, having new ideas of what to do with it. And so, it's kind of nice cuz the last idea that was here was someone saying, "Hey, like when I work with coding agents, I want to make sure that the agent has clear criteria to evaluate whether it did a good job on the work before it tells me that it's done. And um so they had that idea, the bot just picked up on it. None of us did anything manually.

我最喜欢、也最好玩的一个，是我们叫做「会议 bot」的东西。我切到浏览器给大家看一下。好，我们在展台有个位置，昨天一整天都开着这个会议 bot。这是一场 4 小时的 Google Meet。你只要把 bot 拉进 Meet、Zoom、Teams 或者随便什么会议里，它就一整天在那儿听。它一边听，一边生成了一大堆东西。如果它发现已经有相关的工作在做了，就会直接关联过去——不会重复建一份新的。这里面有一部分是大家在测试这个会议 bot，故意让它干点奇怪的事，或者有意思的事，也有些只是随口的创意。但相当一部分其实是真的好点子——来自别人看我们在做什么、提问、然后冒出的新想法。挺妙的一点是，这里最后一条想法，是有人说：「我用编码 agent 的时候，希望它在告诉我『做完了』之前，先有一套明确的标准来判断自己这活到底干得好不好。」他就是随口提了这么个想法，bot 自己就把它捡起来了，我们没有任何人做任何手工操作。

[7:49]

It created this ticket and started working on it. And then I was able to just say, "Hey, take a screenshot of what what you did." And here's that screenshot. And it kind of modified our ticket form to add these two new fields of acceptance criteria. Now, I'm going to Am I going to ship this one exactly how it is? Like, no, probably not. But it's a new idea, it's concrete, I can play with it, I can go and actually like use the live preview and like see if this improves performance. And so it takes like this you know, you know, your hundreds or thousands of ideas that are everywhere and it helps you kind of move with the speed of what your customers are asking you for and what they're thinking. And um it's really fun because every time we have an onboarding your customer call or or team meeting, we almost always have dozens of new ideas that are prototyped, but more importantly, at least a few shippable PRs with a very minimal intervention. So we talk, stuff comes out, we look at it, we ship it. It's so much fun. Put this back. So the next thing I'm going to mention is that, you know, the these three things that I've talked to you about really rely on having your workflow, your code base, your project set up to work in an isolated cloud environment.

它自己建了这个 ticket，然后就开工了。之后我只需要说一句「嘿，把你做的东西截个图」，这就是那张截图。它把我们的建单表单改了，加上了两个新的字段，也就是验收标准。那我会原封不动地把这个直接 ship 出去吗？不会，多半不会。但这是一个新点子，而且是具体的，我可以拿它去折腾，可以打开实时预览，实际看看这是不是真的提升了效果。所以它做的事情是：把散落在各处的成百上千个想法收拢起来，让你能跟上客户的诉求和他们的思考节奏往前跑。而且这真的很好玩——每次我们开客户 onboarding 通话或者团队会议，几乎都能收获几十个已经做成原型的新点子；更重要的是，在几乎不用人插手的情况下，至少有几个是能直接 ship 的 PR。我们聊天，东西自己冒出来，我们看一眼，然后就发出去了。太有意思了。我把这个收回去。接下来我要说的是：前面讲的这三件事，其实都依赖一个前提——你的工作流、你的代码库、你的项目，得能在一个隔离的云端环境里跑起来。

[9:04]

So that way the agents aren't trapped on an individual's machine. So there's several reasons why this is important. So the first one is to eliminate what some people are calling lid anxiety. You want to be able to close your laptop. You've probably seen people running around the office with their laptops open while stuff is working or at the airport or you know, there's some posts on Twitter or whatever about um you know, like people having their laptop tethered to their phone in their cars as they're driving home. This was actually probably the impetus for for me and for a few people on our team to even start working on this. You You last year I started working on Cloud code with cloud code a lot. I had a I think at the time like 6-month-old. I like didn't want to like be tied to my laptop or have that stress. I was like, I I don't I I don't ever want to think about whether I can like step away from a laptop or not. And so, we moved everything to the cloud. Things are working always. It eliminated that problem for us that people have been talking about for the past year. It's really helpful. It's important to me. But I don't think that's the most important reason to do this. I think the most important reason to do this actually was was touched on in the previous talk, if you were here for it.

这样 agent 就不会被困在某个人的机器上了。这件事重要有好几个原因。第一个是消除有些人说的「合盖焦虑」。你得能随时合上笔记本。你八成见过有人抱着开着盖的笔记本在办公室里跑来跑去，因为活还在跑；或者在机场；再或者 Twitter 上那些帖子——有人开车回家的路上，把笔记本用手机热点连着放在车里。老实说，这基本就是我和团队里几个人当初动手做这件事的直接原因。去年我开始大量用 Claude Code 干活，当时我孩子大概六个月大。我实在不想被一台笔记本拴住，也不想扛那种压力。我当时想的是：我永远不想去纠结「我现在能不能离开笔记本」这种问题。所以我们把一切都搬到了云上。活始终在跑。这个困扰了大家一年的问题，对我们来说被彻底解决了。这确实很有用，对我个人也很重要。但我不认为这是把它搬上云最重要的理由。我觉得最重要的理由，其实上一场演讲已经提到了，如果你们刚才也在的话。

[10:12]

I think you should only give your access give your agents access to only what they need. Right? So, if you think about what's happening, you have a bunch of developers with these agents running on their laptop. Their laptops, unless you have like impeccable hygiene, probably have a bunch of stuff on it that you don't want the LLMs or agents to have access to. And yeah, like everybody's working on these sandboxes and approval flows. And so, you but but but really you're in one of two camps. You're either approving a bunch of stuff or you're hoping that your auto approval flow or your YOLO mode or whatever is configured properly and your sandbox is configured properly and doesn't read a bunch of stuff on your laptop that it shouldn't have. And, you know, as as the previous talk mentioned, like these agents are getting more autonomous. They're getting really resourceful. They They're trying to please you and do what you said. And so, when you say, "Hey, you know, wipe this agent database." and it finds a token on your laptop that it can use and it thinks it's working with staging, but actually it's production and now it just deleted everything. I'm not trying to say this is happening constantly, but it still happens. And for us, the peace of mind of just like letting anybody run with these experiments and ideas and prototypes and and and real code without having to worry about this is is is is really worthwhile.

我认为你只应该给 agent 它真正需要的那部分权限。你想想现在的情况：一堆开发者，各自在自己的笔记本上跑着这些 agent。而这些笔记本上——除非你的卫生习惯好到无可挑剔——多半装着一堆你压根不想让 LLM 或 agent 碰到的东西。是，大家都在搞沙箱、搞审批流。但说到底你只会落在两个阵营里：要么你在不停地点「批准」，要么你在祈祷你的自动批准流程、你的 YOLO 模式、你的沙箱配置都是对的，别去读你笔记本上那些它不该读的东西。而且正如上一场演讲说的，这些 agent 正变得越来越自主，也越来越足智多谋。它们一心想让你满意，想把你交代的事办成。于是当你说「把这个数据库清掉」的时候，它在你笔记本上翻到了一个能用的 token，它以为自己连的是 staging，结果那是 production，然后它就把东西全删了。我不是说这种事天天在发生，但确实还是会发生。对我们来说，能让任何人放开手去跑实验、跑想法、跑原型、甚至跑真实代码，还完全不用担心这类问题，这份安心真的太值了。

[11:24]

To go one step further on that, um it's not just, "Hey, make sure they don't have the credentials that they shouldn't have." It's also make sure they can't exfiltrate your code or your projects or your secrets or your content to somewhere they shouldn't be able to. And so you have a configurable network sandbox and you say, "Look, these are the places you're allowed to access, these are the ones you can't access." And anytime it tries to access something that it shouldn't, it just pops up and says, "Hey, tried to access something. Do you want to give it access? Maybe you're trying to integrate a new vendor and you need documentation." And you can do it on a per ticket basis or for the whole project. And so again, that peace of mind of like people can do things. If they need new access, it's easy to grant it. And um we're not going to leak a bunch of important data uh by uh running agents in yellow mode. And the last thing I'll mention about that is that this is the key for allowing your non-technical team members to trigger real work. Right? Your non-technical people don't have development environments set up on their computers. But we've gotten our support people or growth people to actually be meaningfully impacting the product by just talking to the users, seeing bugs, experiencing themselves, and just go to Slack or the rabbit cell and say, "Hey, fix this."

再往前推一步：这事儿不只是「别让 agent 拿到它不该有的凭证」，还得保证它没办法把你的代码、项目、密钥或者内容偷偷传到不该去的地方。所以我们做了一个可配置的网络沙箱——你来定哪些地方允许访问、哪些不允许。只要它想访问不该访问的东西，就会弹出来问你一句：「它刚才想访问某个地址，要放行吗？」比如你正好在接一家新的第三方服务、需要查文档。你可以只针对这一个 ticket 放行，也可以对整个项目放行。这又回到那种安心感：大家可以放手干活，需要新权限的时候授权也很容易，同时我们也不会因为让 agent 跑在 YOLO 模式下就泄露一堆重要数据。关于这点我最后再补一句：这正是让非技术同事也能触发真实开发工作的关键。对吧？非技术的人电脑上压根没配开发环境。但我们让做客服的、做增长的同事真真切切地在改进产品——他们只要跟用户聊天、看到 bug、自己踩到坑，然后去 Slack 或者我们的应用里说一句「把这个修了」。

[12:40]

They fix it. Screenshots are shown. Engineer gets it, gets merged. Without that, they'd have to put it in linear and linear would eventually pick it up and a PM would triage it or whatever. None of that here. You just ask for it and it's done. Now, the reason people didn't do this you know, up until somewhat recently, like this was really painful. Getting your full thing set up in this like kind of sandbox environment used to be really, really painful. But agents have gotten better. We have our own environment setup assistant that kind of takes your project and gets it to work in one of these sandboxes. But honestly, whether you use this or not, I highly recommend you get your project working this way and you can just get Cloud Coder or CodeX to do this for you. You don't have to use us, but you know, we think it's the best way. And the last lesson is to benchmark agents on your code base. So, um the way we do this, we select pull requests that represent great engineering work. It could be agent created, could be human created, could be a hybrid, doesn't matter. You pick the agents you want to use and benchmark. And then you get a quality versus cost and time breakdown on your code base.

agent 就真把它修好了，还附上截图。工程师接手一看，合并进去。要是没有这套东西，他们就得去 Linear 提个单，等 Linear 那边慢慢排到，再由 PM 分诊什么的。这里完全不用。你张嘴要，事就办完了。那为什么直到最近大家都没这么干？因为以前太痛苦了。把整套项目在这种沙箱环境里跑通，过去真的非常非常折磨人。但 agent 变强了。我们自己也做了一个环境配置助手，能把你的项目改造到能在这类沙箱里跑起来。不过说实话，不管你用不用我们的东西，我都强烈建议你把项目弄成这种跑法——你完全可以让 Claude Code 或者 Codex 帮你做。不一定非得用我们，只是我们觉得这是最好的路子。最后一条经验是：在你自己的代码库上给 agent 做基准测试。我们的做法是，先挑出一批代表优秀工程水准的 pull request——agent 写的、人写的、人机混合的都行，无所谓。然后选好你想用、想评测的那几个 agent，最后你就能拿到一份基于自己代码库的质量 vs 成本、质量 vs 时间的拆解。

[13:46]

Now, why do you want to do this? There's There's many reasons, but one is that if you're kind of going off the public benchmarks, we bench or terminal bench or other stuff like those tasks may have absolutely nothing to do with your task. Like swe bench is all in Python, we're Ruby on Rails. It is not the case that the benchmarks are identical for them. There's trends that do compare, but the results can be very, very different. And I'm going to swap over to my browser one more time here. So, this is These are results on our code base of all these different harnesses. This one here is quality versus cost. This one here is quality versus time. I'll start with this one. You can see some trends here. Right, you can see that the Anthropic agents have just been consistently getting better, but not really any faster. The Codex agents and cursor are actually pretty fast and quite good. The open stuff has been getting better and better over time, but they're kind of slow. This is for our code base again. I'm not trying to make any general claims here. By cost, the Anthropic stuff is clearly just so much more expensive for us. And the Codex stuff has been cheaper for us. And so this causes us to change our behavior. We still use the different models. There's different use cases for them. We like the variety. We still use all these things. But when we saw these results, they kind of matched our vibe check. We wanted to kind of like have hard data, too. We switched our default to Codex at that time.

那为什么要做这件事？理由很多，其中一条是：如果你只看公开榜单——SWE-bench、Terminal-Bench 之类——那上面的任务可能跟你手头的活儿八竿子打不着。比如 SWE-bench 全是 Python，我们是 Ruby on Rails，两边的评测根本不是一回事。趋势上确实有可比性，但结果可能差得非常非常远。我再切一下浏览器。这是各种不同 harness 在我们代码库上跑出来的结果。这张是质量 vs 成本，这张是质量 vs 时间。先看这张。能看出一些趋势：Anthropic 的 agent 一直在稳步变好，但速度基本没长进；Codex 系列和 Cursor 其实相当快，效果也不错；开源那批随时间越来越好，但偏慢。再说一遍，这只是我们自己的代码库，我不是在下什么普适结论。从成本看，Anthropic 那套对我们来说明显贵太多了，Codex 对我们就便宜。所以这直接改变了我们的行为。我们还是会用不同的模型，它们各有各的适用场景，我们也喜欢这种多样性，这些东西我们都还在用。但当时看到这组结果——它正好跟我们的直觉体感对上了，我们本来就想再拿点硬数据——我们就把默认切成了 Codex。

[15:06]

And Fiable came out, and it was great. Kind of switched our default to that for like the few days we had it, and then it went away and switched back to Codex. But the most important thing is like because we're agnostic, like none of that had any meaningful disruption on our work. Like we're able to just kind of switch back and forth really easily. So, the next day something new comes out, see if it's good, and go. And the last thing I want to mention around that is like I don't know if this resonates with you all, but I have a lot of friends that like, "Okay, and I heard MiniMax is good. I heard, you know, GLM is good, and Kimmy K2 is good, but like haven't had the time to try it out, and everyone keeps telling me I need to cuz it's so much better and faster and cheaper." And you kind of have that anxiety for a little while, and then like finally you take the two hours to try it, and it's like, "Oh, actually like didn't really work for us. So, like what's, you know, I just wasted those two hours." It kind of eliminates that. It helps you kind of stay on the cutting edge really like seamlessly. Let me go back. So, what that kind of turned into us for us is, you know, essentially 100% like 99.9% of our pull requests are like heavily agent generated.

后来 Fable 出来了，效果很好，我们那几天又把默认切成了它，然后它下线了，我们就切回 Codex。但最重要的是：因为我们对模型是中立的，这些来回切换对我们的工作完全没造成什么实质干扰，切来切去非常轻松。所以第二天再出个新东西，试试好不好用，接着干就是了。关于这点我最后想说一句，不知道在座有没有共鸣——我有不少朋友是这样的：「听说 MiniMax 不错，听说 GLM 也行，Kimi K2 很强，但一直没空试，身边所有人都跟我说必须试，说它又好又快又便宜。」你就这么焦虑一阵子，最后终于抽出两个小时去试，结果发现「哦，其实在我们这儿根本不好使，那我这两小时不是白花了」。这套东西正好能消掉这种焦虑，让你非常顺滑地一直待在最前沿。我切回来。所以这一整套下来，我们现在基本上 100%——严格说 99.9%——的 pull request 都是大量由 agent 生成的。

[16:07]

We know that quality and reliability and security are really important, so we still have humans look at everything. We have agents help with it all, but everything's human reviewed. Um you know, for our our our our relatively small team, we had 1 and 1/2 billion tokens over the past month. And you can kind of see what we're saying about Claude here. It's a little small, so I apologize, but we had 3,300 Claude code runs that cost $10,000 in tokens daily. We have plans, so we didn't spend $10,000 on it. And Codex had four times as many sessions, and it was cheaper overall. And so again, the vast majority of our work currently is merged through Codex. We still use the other models. More and more is happening through GLM 5.2. Going to invest in that. And the one thing that we're really excited to do going forward with this benchmarking is automatically like like you've probably heard about people, you know, routing tasks to the right models and all that, but how do how does like some third party know what to route for your code base? Like this is a way that you can know what's going to work best for which task for your for your project, and we're going to kind of automatically routing that for you.

我们很清楚质量、可靠性和安全性都非常重要，所以所有东西还是要人过一遍眼。agent 会全程帮忙，但每一处都经过人工 review。就我们这么个不大的团队，过去一个月用掉了 15 亿 token。这里也能看到我们刚才说的 Claude 的情况——字有点小，抱歉——我们跑了 3,300 次 Claude Code，按 token 算每天要花 1 万美元。我们买了套餐，所以并不是真的掏了 1 万美元。Codex 的会话数是它的四倍，总体反而更便宜。所以再说一次，我们目前绝大多数工作是通过 Codex 合并进去的。其他模型我们也还在用，越来越多的活儿开始跑在 GLM 5.2 上，我们打算在这上面加大投入。而这套基准测试接下来最让我们兴奋的一件事是自动化——你们大概听过「把任务路由给合适的模型」这类说法，但一个第三方凭什么知道该怎么给你的代码库做路由？这套东西能让你自己知道：在你的项目上、对哪类任务、哪个模型最好使，接下来我们会帮你自动完成这个路由。

[17:12]

So, I'm going to leave you with a few recommendations. So, first, um get your code base and agents working in a sandbox. It unlocks a lot of different things, a lot of different workflows, everything I've talked about and more. Second, integrate agents into the relevant human interfaces so your team and your agents can work together and don't have to like context switch and copy context back and forth. Obviously, we think Superagent is the best way to do it, but plenty of people are home rolling things, hacking things together. Figure out how to make this happen because if not, the friction is just really high. And lastly, find a way to benchmark and become model agnostic so you're not tied to anybody and you can just constantly stay at that right part on the frontier of cost, speed, quality. So, thank you so much. Um, we've got a booth in the expo. Please feel free to come by. You can sign up at superagent.com or you can email me with any questions at arjun@superagent.com. Um, I will be out in the back as well for any questions. Thanks so much.

最后给大家几条建议。第一，把你的代码库和 agent 放进沙箱里跑起来，它能解锁一大堆玩法和工作流，我今天讲的这些，还有更多。第二，把 agent 接进人本来就在用的那些界面里，让团队和 agent 能一起干活，不用来回切上下文、把上下文复制来复制去。我们当然觉得 Superconductor 是最好的做法，但也有很多人是自己攒的、拿胶带糊出来的。总之想办法把这件事做成，不然摩擦成本真的太高了。最后一点，找到一种做基准测试的办法，让自己变成模型中立的，这样你就不会被任何一家绑死，可以一直待在成本、速度、质量这条前沿曲线上最合适的位置。非常感谢大家。我们在展区有个展台，欢迎过来聊。你们可以在 superconductor.com 注册，有任何问题也可以发邮件给我：arjun@superconductor.com。我一会儿也会在后面，随时来问我。谢谢！

[18:12]

[applause]

（掌声）

[18:16]

One one one last thing, um, if, uh, you know, at the booth we're mentioning we're giving away a MacBook Neo. If you were here cuz you signed up through for that, just meet us outside and we will announce the winner. Thank you.

最后再补一件事：我们在展台那边说过要送一台 MacBook Neo。如果你是冲着这个报名来的，到外面找我们，我们会公布获奖者。谢谢！