How VS Code Went from Monthly to Weekly Releases with AI — Harald Kirschner
频道: AI Engineer
视频: https://www.youtube.com/watch?v=I2LL_wd89-A
原文语言: en
统计: 共 14 轮 · Harald 13
[0:12] Harald
Hey uh everyone. This is a hard tone to follow. Why did everybody leave? No. Um this is the rerun of a talk that she gave yesterday upstairs. So this will be better than the one yesterday. So uh kudos to you for catching this one. I'm working on VS Code and we move from a monthly release to a weekly release cycle thanks to AI. And this is about the things we learned, the systems that made this work, not just to ship faster, but actually to ship more effectively. So, it's not about how we build agents into VS Code, which we can try out anytime. You can come by the booth and see us showing of agents, but this is about how we use agents to build the product itself. And that's VS Code, that's the copci and other areas, but also how as you use agents and more AI, things are breaking and you have to rethink actually how you work. And that's a very small team shipping to over 50 million users. What happened is over the past year is we started tracking already early on this code survival metrics in VS code. So that means the percent of agent written code that actually gets committed. how much kind of garbage has the agent produced that you threw out the window again or didn't trust and that you that you read it u and and scrapped. So GBD 4.1 started at 55%.
大家好。前面那场珠玉在前，真不好接。怎么大家都走了？没有没有，开个玩笑。嗯，这一场其实是我昨天在楼上讲过的那个演讲的重讲，所以这次会比昨天那场讲得更好。能赶上这一场，得给你们点个赞。我在做 VS Code，靠着 AI，我们从每月发一版变成了每周发一版。今天要讲的，就是我们一路学到的东西，还有让这件事跑得起来的那套体系——不只是为了发得更快，而是真正发得更有成效。所以这场不讲我们怎么把 agent 做进 VS Code，那个你们随时都能上手试，也可以来展位看我们秀 agent；这场讲的是我们怎么用 agent 来做产品本身。产品包括 VS Code，包括 Copilot CLI 和其他几块；同时也会讲到，随着你用的 agent 和 AI 越来越多，很多东西会开始出问题，你得回过头来重新想清楚自己到底该怎么干活。而做这些事的是一支非常小的团队，面对的是超过 5000 万用户。过去这一年发生的事情是这样的：我们很早就开始在 VS Code 里追踪 code survival（代码存活率）这个指标，也就是 agent 写的代码里，最后真正被提交进去的比例。换句话说，就是 agent 到底产出了多少垃圾——被你转手又扔出窗外的、你信不过的、被你撤回废掉的。GPT-4.1 起步的时候是 55%。
[1:37] Harald
But then over time and over improving the harness and new models cloud opus 4.6 is now today at 86%. And that increase clearly shows developer trust and confidence in shipping AI code faster and more AI code generated. And this success created problems on the VS code side. We're one of the biggest or the biggest open-source project on GitHub. So we saw a massive increase in issues because AI helps to file issues. So we actually see an increase in quality issues but also the counterpart of more automated lowquality issues. We also see more open PRs by the team by our own velocity and by our community. And interesting part we actually see everybody would assume we see a lot of garbage PRs but actually the number of community merged PRs is going up as well which is really amazing that we can unlock more community contributions as well. And as part of this new workflow, we actually moved several months ago, we moved from monthly releases, which happened over 10 years. Since the inception of VS Code 1.0, we have been shipping a new version of VS Code on a monthly basis. Including fun facts like the January release actually ships in February. So when you read the release notes in VS Code, you're wondering why it's February and you're reading the January release notes if you're a month late. No, that's just how we tracked our iterations.
但后来，随着时间推移，harness 不断改进、新模型不断出来，Claude Opus 4.6 今天已经到了 86%。这个涨幅很清楚地说明，开发者对于更快地发布 AI 写的代码、让 AI 生成更多代码，是越来越信任、越来越有底气的。而这份成功，也给 VS Code 这边带来了麻烦。我们是 GitHub 上最大的开源项目之一，或者说就是最大的那个。所以我们看到 issue 数量暴涨，因为 AI 能帮人提 issue。实际上，高质量的 issue 确实变多了，但与之相对的另一面是，自动生成的低质量 issue 也变多了。我们还看到挂着没合的 PR 更多了，有团队自己提的——我们自己的速度上来了——也有社区提的。有意思的是，大家都会以为我们收到的是一大堆垃圾 PR，但实际上，社区 PR 被合并的数量也在往上走。这一点真的很棒，说明我们也能释放出更多的社区贡献。作为这套新工作流的一部分，我们在几个月前告别了月度发布——这个节奏已经持续了 10 年。从 VS Code 1.0 诞生起，我们就一直是每个月发一个新版本。这里面还有些冷知识，比如一月版其实是二月才发的。所以你在 VS Code 里读发布说明的时候会纳闷：明明已经二月了，怎么看的是一月的发布说明，是不是晚了一个月？不是的，那只是我们给迭代记账的方式。
[3:08] Harald
So that cadence was to match to velocity, but it also showed us that the it's not about using more AI as you work dayto-day, which a lot of companies are pushing. Like you're not token maxing. You're not spending every day using AI. It's about evolving the whole system, how you're shipping software to evolve to actually make better use of AI along the whole process. And that what makes this velocity actually work and doesn't just generate more code. I'm breaking this down into this first part which a lot of people have been experiencing. I see shipping code faster and that's the easy part. You have like this 100x engineers that has built this beautiful skill MCP plug-in system and they're just shipping code like crazy and slowly maybe more in the more people in the team uh adopt the same patterns and you see a lot more code shipping faster. But the problem is once you ship faster, you get into code review. You get into how do you actually ship high quality code that doesn't constantly break. And that's important for us as VS Code. We're shipping binaries to user machines. So if something breaks, that recovery is even more costly. But once you hit that velocity with shipping high quality code faster with confidence, how do you actually know you're shipping the right thing or if we should ship this thing? And that's where AI kicks in to actually learn faster in the end and shipping better products which a lot of people are missing out as as they write more code is applying that product taste and applying that learning.
所以调整节奏是为了跟上这个速度，但它同时也让我们看清了一件事：重点不在于你日常干活时多用 AI——很多公司正在推的就是这个。不是让你去 tokenmaxxing（刷 token 用量），不是让你每天从早到晚都在用 AI。重点是让整套体系进化，让你发布软件的方式进化，这样才能在整个流程里把 AI 真正用好。真正让这个速度跑得起来、而不只是产出更多代码的，是这个。我把它拆成几部分。先说第一部分，很多人都已经体会过了，就是更快地发布代码，这是容易的部分。你身边会有那种 100 倍工程师，搭了一套漂亮的 skill、MCP、插件体系，然后疯了一样往外发代码；慢慢地，团队里也许有更多人跟着用起同样的套路，你就会看到多得多的代码以更快的速度发出去。但问题是，一旦发得更快，你就撞上了代码评审。你就要面对：到底怎么才能发出高质量、不会三天两头出问题的代码？这对我们 VS Code 来说很重要。我们是把二进制文件发到用户的机器上，所以一旦出了问题，恢复的代价更高。可是，等你达到了那种又快、又有把握地发布高质量代码的速度，你又怎么知道自己发的是对的东西？或者说，这个东西到底该不该发？这就是 AI 派上用场的地方：最终让你学得更快，做出更好的产品。很多人代码越写越多，却恰恰漏掉了这一块——把产品品味用上去，把学到的东西用上去。
[4:49] Harald
So let's start shipping faster. Now that everybody can vibe, we can really all ship faster. And it's a big part is getting your codebasation ready. So that's kind of the no-brainer slide that everybody should already been doing is thinking about how to get your code bases agent ready is creating agents.mmd that give it that are lightweight enough and give the agents a good kind of map of the codebase where to look at mapping out that is every agent now at your uh at your fingertips has a slash in it that gives you a good draft and then you keep itering those are living documents that should evolve as the agent makes mistakes and works. And that's where I see a lot of teams that have invested in developer experience actually gain a lot because once they have good documentation and onboarding for a codebase that is read by the agent as well and that can help the agent as much. Next up you have skills. VS Code has always messed heavily in accessibility. So one skill we added early on was infusing all the accessibility best practices and how we think about accessibility into a skill that everybody will use and that previously was one person that had to then got get pulled in or give feedback and now that skill was reviewed and is maintained by the kind of area owner of accessibility.
那就从发得更快说起。现在人人都能 vibe 了，我们确实都能发得更快。其中很大的一块，是让你的代码库 agent-ready（对 agent 就绪）。这算是一页不用动脑子的幻灯片，大家本来就都该在做了：想清楚怎么让你的代码库对 agent 就绪，也就是创建 AGENTS.md，它要足够轻量，同时给 agent 一张像样的代码库地图，告诉它该去哪儿找东西。至于怎么把这张地图画出来，现在你手边的每个 agent 都有一个 /init 命令，能给你生成一份不错的初稿，然后你再不断迭代。这些是活的文档，应该随着 agent 犯错、随着 agent 干活而不断演进。我看到很多在开发者体验上下过功夫的团队，在这里其实收获很大，因为一旦他们的代码库有了好的文档和上手指引，这些东西 agent 也会去读，对 agent 的帮助一样大。接下来是 skill。VS Code 一直在无障碍（accessibility）上投入很大。所以我们很早就加的一个 skill，就是把所有无障碍方面的最佳实践、还有我们思考无障碍的方式，都灌进一个人人都会用的 skill 里。以前这件事靠的是一个人，得把他拉进来，或者等他给反馈；而现在，这个 skill 是由无障碍这个领域的负责人评审过、并由他来维护的。
[6:10] Harald
So to bring that that accessibility that short work everywhere into everybody's hands. So think how you can encode your expert knowledge to unblock this like few users that might be always asked for this input into a skill. And lastly once you do it right for us in VS Code the litmus test is can I as a PM effectively VIP code in the VS code repo which I do um and how is that accepted by engineering? How much work does it cost? Is it out of the box working? Is it for me working locally? Or is it eventually breaking when I ship it? So, a lot of these quality improvements will show in the end that you can allow everybody to be more effective in the repo. Another improvement, and that's a massive one, is we switched to TypeScript Go. And if you've been in any developer codebase that takes some time to build or run linting or run CI/CD, you've noticed it's already annoying. But developers find other ways. They can review a PR. They can context switch into something else while runs and then do something else. But once you hit that with agents, you have 10 agents, 20 agents running and hitting the same bottleneck, these any slow part of your CI/CD will compound. And for us, Typescript Go was a 10 times improvements in our builds, which at the scale of automatic PRs running with agents and them fixing getting needing that CI/CD loop for feedback to improve the code is a massive improvement.
这样就能把无障碍这块的专业功夫带到各处，交到每个人手里。所以想一想，你可以怎么把专家知识编码成一个 skill，好给那少数几个总被人追着要意见的人松绑。最后一点，一旦你把这些做对了——对我们 VS Code 来说，试金石就是：我作为一个 PM，能不能在 VS Code 的仓库里有效地 vibe code？我确实在这么干。然后，工程团队接不接受？要花掉多少功夫？是不是开箱即用？是只在我本地能跑，还是等我发出去的时候迟早要出问题？所以这些质量上的改进，很多最终会体现在：你能让每个人在仓库里都更有成效。另一项改进，而且是非常大的一项，是我们切换到了 TypeScript Go。如果你待过任何一个构建、跑 lint 或者跑 CI/CD 要花点时间的开发代码库，你就会发现这本来就够烦人的了。不过开发者总能找到别的办法：可以去评审一个 PR，可以趁它跑着的时候切换上下文去忙别的，再干点别的事。但一旦是 agent 撞上这个问题——你有 10 个、20 个 agent 在跑，撞的是同一个瓶颈——你的 CI/CD 里任何一处慢的地方都会层层叠加。对我们来说，TypeScript Go 让构建有了 10 倍的提升。在我们这种规模下，自动 PR 是 agent 在跑，agent 要修东西、需要靠 CI/CD 这个循环拿到反馈来改进代码，这个提升是巨大的。
[7:55] Harald
And that leads to me uh filing big PRs. That was beginning of the year to get the ask questions tool in which I believe we really needed and then bringing it to the team to show to let them try it out in the product. Within a few weeks we had a perfectly polished and designed experience based on my initially very scrappy PR. But it unblocked that this these discussions the participation of everybody else because we had the ground set laid and the idea and the experience was clear. Next up, when agents actually work on UI, and we all had this, that the agent gives you a UI and it says it looks perfect, then you open it and it's all misaligned. Even with SVG, even with the latest models, you still get it. And it's like the number one thing I keep telling everybody now these days. If your agent cannot use your application, your product directly to get this feedback loop of is it all working, then that's a really big investment that pays off every time you work on UI. And there's two areas we invested in. One is the component browser, which is an automated build that runs every time VS Code changes something. Let me open this in this issue as an example. In this case, we added a back button to our customization screen. And because it's changing wide, we actually have this automated process in the back that takes screenshots of every component in VS Code and points out the differences.
这也让我自己开始提大 PR 了。那是今年年初，为了把 ask questions 工具加进去——我认为我们确实需要它——然后把它带给团队看，让他们在产品里试用。几周之内，在我最初那个非常粗糙的 PR 的基础上，我们就有了一个打磨到位、设计完整的体验。不过它真正起到的作用是打开了局面：这些讨论得以展开，其他所有人都能参与进来，因为地基已经铺好，想法和体验也都清楚了。接下来说说 agent 真正去做 UI 的时候。我们都遇到过：agent 给你做了个 UI，还说看起来很完美，结果你一打开，全是错位的。哪怕用 SVG，哪怕用最新的模型，照样会这样。这差不多是我这阵子逢人就讲的头号大事：如果你的 agent 还没法直接使用你的应用、你的产品，来拿到「是不是都能正常工作」这个反馈循环，那么把这件事做起来，就是一笔非常大、而且每次做 UI 都能回本的投入。我们在两块上做了投入。一块是组件浏览器（component browser），它是一个自动化构建，每次 VS Code 有改动就会跑一遍。我打开这个 issue 当例子。在这个例子里，我们给自定义设置界面加了一个返回按钮。因为这个改动影响面比较广，我们后台其实有一套自动化流程，会给 VS Code 里的每个组件截图，并把差异标出来。
[9:33] Harald
It allows us to detect unexpected changes like we're changing one component as the ripple effect of something else moving or some other icon disappearing. and allows you to quickly review a PR as well because you also get directly a view and what the PR is actually doing and that's what we ask every developer before we ask them to at least attach a video or screenshot on what they have been implementing so we don't have to run everything from scratch and we can quickly assess and and give feedback and this is now automated with the component explorer next up we have the self-correcting loop because in VS Code we run uh a web browser and open a website. Basically, that's what VS Code is. It's all HTML. We can actually use Playright to automate a lot in the browser. So, we have a /launch skill that would just open up VS Code and give it a specific scenario to click through in VS Code. So, it's a really powerful way to diagnose issues because you can also get the locks along the way, but also validate any fixes you did before and after without you having to click through it. So you can just hit launch move on to the next task and then come back once the agent verified it's worked because it can access your application and to call out this is nice because we have playright and we are a web app uh running in electron but there's many other solutions out there one of my favorites is Xcode MCP because on my site I work on iOS apps and once you see Xcode MCP just creating in VS code without Xcode involvement just making an iOS app opening it up clicking through it, taking screenshots. These moments are magical and give you a really tight feedback loop as well. So there's other tools available for Android as well to make this work.
这样我们就能发现意料之外的变化——比如我们只改了一个组件，结果连锁反应让别的东西挪了位置，或者另一个图标不见了。它也能让你很快地评审一个 PR，因为你直接就能看到画面，知道这个 PR 到底干了什么。这其实就是我们以前对每个开发者的要求：至少附上一段视频或一张截图，展示自己实现了什么，这样我们就不用什么都从头跑一遍，能很快做出判断、给出反馈。现在这件事靠 component explorer 自动完成了。接下来是自纠正循环（self-correcting loop）。因为 VS Code 里面跑的其实就是一个浏览器、打开一个网页——说到底 VS Code 就是这么个东西，全是 HTML——所以我们可以用 Playwright 在浏览器里自动化很多操作。于是我们做了一个 /launch skill，它会直接把 VS Code 打开，再给它一个具体的场景，让它在 VS Code 里一路点过去。这是个非常强大的诊断问题的办法，因为你一路上还能拿到日志，也能在修复前后各验证一遍，全程不用你自己去点。所以你可以直接敲 launch，转头去做下一个任务，等 agent 验证完确实跑通了再回来——因为它能直接操作你的应用。这里要特别说一句：我们这么做很顺，是因为我们有 Playwright，而且我们本身就是一个跑在 Electron 里的 web 应用，但外面还有很多别的方案。我最喜欢的之一是 Xcode MCP，因为我业余会做 iOS 应用。当你亲眼看到 Xcode MCP 就在 VS Code 里、完全不用碰 Xcode，直接把一个 iOS 应用做出来、打开它、一路点过去、截图——这种时刻真的很神奇，也能给你一个非常紧的反馈循环。Android 那边也有别的工具能做到这一点。
[11:21] Harald
Let's talk about code review, which I think is the next big investment. We know in the past when we enabled code we on GitHub the automated GitHub copiloted one we're not convinced yet but over the next months and weeks they have massively improved and now it's a mandatory uh review every time somebody opens a PR and there's more improvements coming actually dial in the uh the effort that a code review does depending on how much risk you have in this repo so you have a low medium high effort for code review so you can dial in kind of the cost benefit as well but for us is if a review after review is done, humans will not even look at PR until all the comments are resolved and and addressed. Okay, now we're holding we're shipping faster. We got to hold quality. And a big signal for us because we're on GitHub is of course issues. Uh who has filed a GitHub issue? Who has had problems in VS Code before? Yeah, cool. Kind of similar. Awesome. This the last crowd had less hands up. So it's good. Um, yeah. So it's a massive, it's a really good signal. And as I said, we get sometimes even better issues now because of AI and more people with different language backgrounds being able to file very good issues. But we also get a lot of issues. And previously engineers actually triage them manually. And now we have AI doing the filtering to get out spam enriching and translating in the end assigning area owners for all the issues to raise the signal to noise quality of this.
下面说说代码评审（code review），我认为这是下一个重点投入的方向。说实话，以前我们在 GitHub 上启用代码评审——就是 GitHub Copilot 那个自动评审——的时候，我们还没被说服。但在之后的几个月、几周里，它有了巨大的改进，现在它已经是强制环节了：只要有人开 PR，就必须过一遍。后面还有更多改进要来——你可以根据这个仓库的风险有多高，去调一次代码评审要花多大力气，分低、中、高三档，这样成本和收益之间也能自己拿捏。而对我们来说，规矩是这样的：评审跑完之后，在所有评论都被处理、解决掉之前，人根本不会去看这个 PR。好，现在我们发布得更快了，质量也得守住。因为我们在 GitHub 上，一个很重要的信号当然就是 issue。呃，谁提过 GitHub issue？谁以前在 VS Code 里遇到过问题？嗯，不错，差不多。很好，上一场的观众举手的还少一些，所以这是好事。嗯，对。所以这是一个量很大、质量也非常好的信号。而且就像我前面说的，因为有了 AI，我们现在有时候收到的 issue 反而更好了，更多不同语言背景的人也能提出非常好的 issue。但我们收到的 issue 数量也确实很多。以前是工程师手动去做分诊（triage）的，现在我们让 AI 来做过滤——把垃圾信息筛掉，补充信息、做翻译，最后给所有 issue 分配领域负责人（area owner），把这里面的信噪比提上去。
[12:54] Harald
We also add a human layer to that once you are in issue because the agent will make mistakes and we need that feedback loop when it makes mistakes. We create a Chrome extension that allows you to fix duplicated issues as well. So we have the agent doing the work and we have a feedback loop where humans can still fix it that we can bring back into the into the agent work up front. That's a really important thing to think about as you have agents doing the work that you get the human feedback as well. The other input we have is a lot of error stacks. Every time there's an exception and most apps will have this you get an error stack reported and we actually don't use any external tool for that. It's all in our own data and we built all the tooling around it and it's mostly we already had it. We already had an ML doing sorting on top and this is now actually done by a mix of criteria. So one is we collect the raw telemetry. We filter it down and that's like 51 billion per day. We filter it down to just error stacks that actually have the full stack. Then we group it and bucket it. That's more of the fingerprinting exercise. And then in the end after we all of that we have 10 issues filed assigned to specific area owners that a PR is autocreated to actually already try to fix the problem.
在这之上我们还加了一层人工——你进到一个 issue 里之后就能介入，因为 agent 会犯错，而它犯错的时候我们需要那个反馈循环。我们做了一个 Chrome 扩展，让你也能修正重复 issue 的判定。所以是 agent 在干活，同时有一个反馈循环，人还能去纠正它，而这些纠正我们又能带回到前面 agent 的工作里去。当你让 agent 干活的时候，这一点非常值得想清楚：你也得拿到人的反馈。我们的另一路输入是大量的错误栈（error stack）。每次出现异常——大多数应用都会有这个——你都会收到一条上报的错误栈。这一块我们其实没用任何外部工具，全都在我们自己的数据里，周边的工具也都是我们自己搭的，而且大部分是早就有的。我们原本上面就有一层 ML 在做排序，而现在这件事实际上是靠一组混合的标准来完成的。首先，我们采集原始遥测数据，然后往下筛——那大概是每天 510 亿条。我们把它筛到只剩那些真正带完整调用栈的错误栈，然后做分组、归桶，这更像是一个打指纹（fingerprinting）的活儿。最后，这一切都做完之后，我们会有 10 个 issue 被提出来、分配给具体的领域负责人，并且会自动创建一个 PR，直接先尝试把问题修掉。
[14:15] Harald
This is our error page where we see all the stacks including how many hits and how many users affected so we can quickly react to anything that's coming. And once we have this working we get a issue filed and a PR automatically opened. This is the issue. is the error with some initial diagnosis and then the PR got opened with and the agent actually goes in and finds out why does this happen who was responsible in this case um added a change that was connected to this to find f find symbols and then it figured that the on cancellation request wasn't in the RPC protocol so the agent already opened the PR we could just merge it and that's a really great automated way you don't care about errors you just want to get a more stable code base. So all of this automated the triage initially mix of agents and deterministic systems then the issue is filed handed off to an agent in the PR multi- aent system and in the end we have fixing fixes landing mostly automatically with some human oversight to still approve the PR. There you go. So previously VS Code did yolo releases in a way that it was release day everything was stable. We did our testing and we shipped the release to 100%. We just opened the floodgates but because of agents we want to derisk that process. We should have probably done it earlier. So now actually we do stage rollouts and as we do the roll out we monitor error locks and issues and everything else. So just like good citizens in a in an web application, we now apply the same because rollbacks on an install app are so expensive.
这是我们的错误页面，在这里能看到所有的错误栈，包括命中了多少次、影响了多少用户，这样不管来的是什么我们都能迅速反应。这套东西跑起来之后，就会有一个 issue 被提出来，同时自动开出一个 PR。这就是那个 issue：里面是这个错误，外加一些初步诊断。然后 PR 就开出来了，agent 会真的钻进去，查清楚这为什么会发生、是谁引入的——在这个例子里，是有人加了一个跟它相关的改动，用来 find symbols（查找符号）——然后它发现 on cancellation request 没有在 RPC 协议里。所以 agent 已经把 PR 开好了，我们直接合并就行。这是一种非常棒的自动化方式——你并不想操心这些错误，你只想要一个更稳定的代码库。所以这一整套都自动化了：最开始的分诊是 agent 和确定性系统混着做，然后 issue 被提出来，交给 PR 里的 agent——一个多 agent 系统——最后修复基本上是自动落地的，只保留一点人工监督，由人来批准这个 PR。就是这样。以前 VS Code 的发布某种程度上是 YOLO 式的：到了发布日，一切都稳定了，我们做完测试，就把这个版本直接推给 100% 的用户，相当于把闸门一下子全打开。但因为有了 agent，我们想给这个流程降降风险——其实我们大概早就该这么做了。所以现在我们实际上做的是分阶段发布（staged rollout），一边放量，一边盯着错误日志、issue 以及其他所有东西。就像 web 应用里守规矩的好公民那样，我们现在也照着同样的做法来，因为一个装在用户机器上的应用，回滚的代价实在太高了。
[15:58] Harald
Okay, now we have this shipping faster, shipping at quality. How do we actually do this in a in a speed that we actually learn faster, which is the which is a bigger challenge with AI? You actually want to build a better product faster. So one of the examples is our VSC bench. We're shipping an agentic product. So it's important that we have our own evals of our product always at hand and being really easy to extend. It's all in the GitHub issue. It's all in GitHub. When we file an issue to add a new scenario, it's actually an agent picking up from a template and bootstrapping the scenario for you. So we make it as easy as possible to bring in all the developer scenarios that are coming up in issues through customer conversations through all these channels to bring them back into our emails. So we then start hill climbing actually against them and then com also confirm that it's all improving offline and then an online experimentation. So classic product life cycle for um for us and we see these interesting things in our stack. Understanding your evals is critical. One experiment we did that you can read about on our blog is that we gave a very simple eval scenario of just write a file with hello world just testing out our eval harness end to end and how different models react.
好，现在我们发得更快了，质量也跟得上。那我们怎么才能以足够快的速度来做这件事，让我们真正学得更快？——这才是 AI 时代更大的挑战。你真正想要的是更快地做出一个更好的产品。其中一个例子就是我们的 VSC-Bench。我们发布的是一个 agentic 的产品，所以很重要的一点是：我们手边要随时有针对自己产品的 eval（评测），而且要非常容易扩展。它全都在 GitHub issue 里，全都在 GitHub 上。当我们提一个 issue 要加一个新场景的时候，其实是一个 agent 接手，从模板出发，帮你把这个场景搭起来。所以我们把这件事做得尽可能简单，好把开发者遇到的各种场景——从 issue 里冒出来的、从客户交流里来的、从所有这些渠道来的——都带回到我们的 eval 里。然后我们就开始对着它们做 hill climbing（爬坡优化），接着先在线下确认整体确实在变好，再去做线上实验。对我们来说，这就是很经典的产品生命周期。在我们这套技术栈里，也会看到一些很有意思的事情。理解你自己的 eval 至关重要。我们做过一个实验，你可以在我们的博客上读到：我们给了一个非常简单的 eval 场景，就是写一个内容为 hello world 的文件，只是为了把我们的 eval harness 从头到尾测一遍，看看不同的模型会怎么反应。
[17:18] Harald
The same five character file took the most expensive model 70x more tokens and that was not the high reasoning model. You can read more about unblock. We actually don't talk about the models, but just that insight in how your models react with your harness is extremely is a very important exercise. Next up, we have prototyping, which I do a lot of because I noticed that I don't actually want to land most of my work into VS Code PRs itself. It's a great way to start a conversation, but most often I want to have a quick conversation because now we're meeting on a daily basis where we just want to have very quick feedback loops on here's the idea, here's how it could look like, what if we do this. Next day you come back with updated uh prototypes and you just keep having this conversation. So much quicker feedback from a month release cycle to weekly to daily sprints where you just keep working on problems and prototypes unlock those deeper discussions and how the experience should look like. It also means that we have the durable ownership in internal work as I mentioned daily sprints um smaller squads, smaller work streams that really prioritize and drive smaller sets um of areas to move faster. So try these things, find out where your bottlenecks are to actually ship higher quality, ship faster and learn faster and find fix that next bottleneck. And don't just tune how you work with agents, but tune where how how do they get feedback? How do you build these loops? And then figure out after you have these these success moments, how can you improve that feedback loop for the next moment of what's blocking you to move even faster and it's a constant iterative cycle across all these cycle ones.
同样一个只有五个字符的文件，最贵的那个模型多花了 70 倍的 token，而且那还不是高推理档的模型。更多细节可以去博客上看。我们其实没有去讲具体是哪些模型，但单单是这个洞察——你的模型在你的 harness 里会怎么反应——就是一个非常重要的功课。接下来是做原型（prototyping），这件事我做得很多，因为我发现，我其实并不想把自己的大部分工作都落成 VS Code 本身的 PR。PR 是开启一场讨论的好办法，但更多时候我想要的是一次很快的讨论，因为我们现在是每天碰头，要的就是非常快的反馈循环：想法是这样的，它大概可以长成这样，要是我们这么做呢。第二天你带着更新过的原型回来，就这么把这场对话一直聊下去。反馈快了太多——从按月的发布周期，到每周，再到每日冲刺，你就一直在问题上持续推进，而原型能打开那些更深的讨论，把体验到底应该是什么样子聊清楚。这也意味着我们在内部工作上要有持久的归属（durable ownership），就像我提到的，每日冲刺，嗯，更小的小队、更小的工作流，真正去排优先级、推动更小的一组领域，这样才能跑得更快。所以，去试试这些做法，找出你的瓶颈在哪里，真正做到发布质量更高、发布更快、学得更快，然后去找到并解决下一个瓶颈。而且，不要只调你跟 agent 协作的方式，还要去调：它们是在哪里、怎么拿到反馈的？你怎么搭起这些循环？等你有了这些成功的时刻之后，再去想：下一个卡住你、让你没法更快的地方是什么，针对它，这个反馈循环还能怎么改进？这是一个不断迭代的循环，贯穿所有这些环节。
[19:09] Harald
That's it. Come by code booth. Happy to chat now. Otherwise, happy coding.
就这些。欢迎来 VS Code 展位找我，现在就可以聊。不然的话，祝大家写代码愉快。
[19:16]
[applause] [music]
（掌声）（音乐）