Benchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, Wisedocs
频道: AI Engineer
视频: https://www.youtube.com/watch?v=7vn4WpqNpck
原文语言: en
统计: 共 18 轮
[0:01]
[music]
[音乐]
[0:12]
So, it's not just my AI pipeline that's on fire, but also my PowerPoint. So, it's 2025. Uh we're scaling as a business and things are going poorly. We're adding too many customers, we're not getting the throughput we need, and we need to improve our underlying technology. And there's three main issues that we're facing. The first one is that we're too slow to meet customer demand. The second one is that this AI pipeline that we've built is too complicated to update. And the third one is because it's a legacy code base, or actually more than 10 repos, nobody actually wants to touch the code. It's not a fun experience. So, we made this decision to refactor over the course of 6 months. And the real question for this talk today was, "Is this the right move to do?" So, I'll spend this time answering this question, but let's start off with the use case. So, the company I work at, WiseDocs, processes complex medical claims, which are PDFs that are more than 10,000 pages in size. Some of these files are bigger than video files. So, it's a pretty complex application, and because of this, it's actually non-trivial to scale the different parts. So, we're going to talk about the pipeline today, which has a number of ML models.
所以，着火的不只是我的 AI pipeline，还有我的 PPT。时间来到 2025 年，我们这门生意在扩张，可情况并不乐观：客户加得太多，吞吐量却跟不上，底层技术必须升级。当时摆在面前的主要有三个问题。第一，我们的速度跟不上客户需求。第二，我们搭起来的这套 AI pipeline 太复杂，改都改不动。第三，因为它是一套 legacy codebase——准确说是散在十多个 repo 里——根本没人愿意去碰那些代码，那不是什么愉快的体验。于是我们做了个决定：用六个月做一次 refactor。而今天这场分享真正想回答的问题是：这一步走对了吗？我会用这段时间来回答它，但先从我们的业务场景说起。我所在的公司 Wisedocs 处理的是复杂的医疗理赔材料，那些 PDF 动辄上万页，有些文件比视频文件还大。所以这是个相当复杂的应用，也正因为如此，把它各个部分扩容起来一点都不简单。今天我们要聊的就是这条 pipeline，里面跑着好几个 ML 模型。
[1:17]
So, I'll divide this talk into a number of chapters. We'll start off with the first one, which is the concept of tech debt. So, I think we all have this feeling universally, if we've been developers for a while, that we all write bad code. The question is, do we do this intentionally or not? If I look back to some of the earliest code I used to write, uh it was bad. This was more than 15 years ago. I tried to print an image of this character from a video game, and I didn't understand that you can't uh system.out.println in Java to render something on the screen. So, hopefully I've come uh further from that point in time, but there's these moments where we all know that we've written bad code before. Now, if we think about technical debt as financial debt, it compounds in mysterious and sometimes unexpected ways. But you should think about it in a rigorous format as well. For us to achieve some kind of ROI by taking on technical debt, such as building a feature or getting new customers, we want to make sure that the ROI makes sense. If we introduce additional complexity into our code base, we can very quickly outrun the ROI we've generated. Now with AI engineering, you've probably seen a number of different stories that have come out to showcase the progress that's been made. These are two case studies from Anthropic, one from Spotify and the other from Stripe, talking about the immense progress that they've made both in shipping velocity and also the ability to refactor code.
我把这场分享分成几个章节。先从第一章开始：技术债。我想只要写代码写过一阵子的人，大概都有个共同的体感——我们都写过烂代码。区别只在于，是不是明知故犯。回头看我最早写的一些代码，那是十五年前了，真的挺烂：我想把某个电子游戏角色的图片打印出来，却不明白在 Java 里 System.out.println 是没法在屏幕上渲染图像的。希望我从那时到现在有点长进，但每个人都有过那种「我知道我写了烂代码」的时刻。如果把技术债类比成金融债务，它会以各种隐蔽甚至意想不到的方式利滚利。不过你也该用一套严谨的方式去看它：我们之所以愿意背技术债，是为了换取某种 ROI，比如上一个新功能、拿下新客户，那就得确认这笔 ROI 算得过来。一旦往 codebase 里塞进额外的复杂度，你赚到的那点 ROI 很快就会被吃光。而在 AI 工程这一侧，你大概已经看过不少展示进展的案例。这里是 Anthropic 的两个 case study，一个是 Spotify，一个是 Stripe，讲的都是他们在交付速度和重构代码能力上取得的巨大进展。
[2:37]
So at this point in time, writing code or making changes is something that teams are doing faster and faster. Now I'll pause here. Who here thinks that products have gotten better in the past 20 years? Technical products. Also raise your hand. I hope everybody, right? Phones are pretty cool. How about 5 years? How about the past year? Okay. So the challenge is that we're going faster and faster through the technology life cycle, but we've lost something. The the product focused on customers in some way has degraded, the maintainability of the code and the reliability has degraded. You can see some of the uptimes here from two leading companies. I blurred out their names for for it it doesn't actually matter who they are, but we are below a three-nine or even four-nine reliability. So even though we're shipping faster and faster, the code quality and the product quality has not necessarily gone up. So let's talk about the refactor that we did. So we started this refactor with actual code implementation in April and did some pre-work earlier. So I'll go through five different tasks that we did and share some of the the findings that we had before and after, especially with as new models have come out.
所以到今天这个时点，写代码、改代码这件事，各个团队都做得越来越快了。这里我停一下，问个问题：在座有谁认为过去二十年里产品变得更好了？技术产品。请举手。我希望是所有人吧？手机就挺酷的。那过去五年呢？过去一年呢？好。所以问题就在这儿：我们跑技术生命周期跑得越来越快，但有东西丢了。产品对客户的关注度在某种意义上退化了，代码的可维护性和可靠性也退化了。你能看到这里是两家头部公司的可用率数据，我把名字打码了——具体是谁其实不重要——我们连三个九都没达到，更别说四个九。所以哪怕交付越来越快，代码质量和产品质量并没有跟着涨上去。接下来说说我们做的这次 refactor。真正动手写代码是在四月开始的，前期还做了一些准备工作。我会讲五个具体任务，分享一下我们前后的发现，尤其是新模型出来之后的变化。
[3:46]
So we spent around 2 months evaluating orchestrators for our AI pipeline. We looked at five open-source projects and we wanted to benchmark and see how effective they were for our use case. And we started this off before deep research came out as part of Google and OpenAI, so that web search capability to do a comprehensive analysis was still not there. Now, after we actually gathered these requirements, we built out proof of concepts with a team of three to make sure that we actually got the right results. Now, I'm pretty confident we could do this 90% faster now with the tooling that we have. Before we would manually go through, use a little bit of AI, but put everything into a Confluence doc, and we'd evaluate across 17 different criteria that we came up with. Nowadays, we could build a much more agentic workflow to do that, starting off with deep research, uh making sure that we match that against the problem statements that we have, creating sub-agents for each of these criteria and uh products, and then finally building POCs and evaluating. So, things have changed in the past year and a half where we could actually go much, much faster. But, we still have to maintain that same set of quality because it's very easy to undergo AI psychosis, where you look at a deep research report that's 20 pages long and you say, "Wow, this looks good." And then those features don't actually exist in the product, and you've set yourself back.
我们花了大概两个月给这条 AI pipeline 选编排器（orchestrator）。我们看了五个开源项目，想做个 benchmark，看看它们对我们的场景到底有多好用。而我们启动这件事的时候，Google 和 OpenAI 的 deep research 还没出来，那种靠联网搜索做全面调研的能力当时还不存在。等把需求梳理清楚之后，我们用三个人的小组做了几个 POC，确保结论靠谱。而现在，我很有信心用手上的工具能快 90%。以前的做法是人工一条条过，稍微用一点 AI，然后把所有东西堆进一份 Confluence 文档，再按我们自己定的 17 个维度来评估。放到今天，我们完全可以搭一套更 agentic 的工作流来干这事：先用 deep research 起头，把结果对照我们的问题陈述做校验，为每个评估维度、每个候选产品各开一个 sub-agent，最后再做 POC 和评估。所以这一年半里事情变了，我们现在能跑得快得多。但质量标准还得守住，因为人太容易掉进「AI 精神错乱」（AI psychosis）——你看着一份二十页的 deep research 报告说「哇，这看着不错」，结果那些功能在产品里压根不存在，你反而把自己拖后腿了。
[5:01]
Now, after we've done the initial uh orchestration research and model serving research, we wanted to actually commit in code. This is just a example of what happened when we were experimenting. I was doing some initial research with Temporal and committed some activities and workflow code to make sure that we can actually replicate what we have in the legacy code base. So then, I did what we wanted to do over a number of iterations and at the time gave it to O3 to actually try to implement this code. And it did it much faster than I would be. This refactor took uh 3 hours of back and forth chatting with Incursr, but it made 10 major mistakes. So, at the time when we were going through this refactor, agentic coding was getting better and better, but it still hadn't reached the point of where it is now. And it was still a very manual process where you had to intervene and actually guide the model and manually edit or delete code. Now, I reran these benchmarks on some some modern models. So, we have Sonnet 4.6 and Opus 4.8. And things were much faster. Sonnet 4.6 with one additional iteration was able to solve the task. And with Opus, it was basically able to one-shot this problem. So, models are getting significantly better along with harnesses.
做完最初的编排调研和模型服务调研之后，我们要真正落到代码上了。这只是我们做实验时的一个例子：我当时在用 Temporal 做初步调研，提交了一些 activity 和 workflow 的代码，想确认我们能把 legacy codebase 里的东西复刻出来。然后我把想做的事迭代拆解好，交给 o3 去真正实现这段代码。它确实比我快多了。这次 refactor 在 Cursor 里来回聊了三个小时，但它犯了十个重大错误。所以在我们做这次 refactor 的那个时间点，agentic coding 是在越来越好，但还没到今天这个程度，整个过程仍然非常依赖人工——你得随时介入、给模型指路，还要手动改代码、删代码。而现在，我拿几个新模型重跑了这套 benchmark，用的是 Sonnet 4.6 和 Opus 4.8，快了非常多。Sonnet 4.6 只多迭代了一轮就把任务解决了；Opus 基本上是 one-shot 搞定的。所以模型在飞速变强，harness 也是。
[6:10]
And the interesting part here as well is that the way that the models interacted has changed substantially as well. Before with O3, there weren't substantial tool calls on certain categories. And then as we moved into Sonnet 4.6 and Opus, we see now that in modern harnesses, we get sub-agents, we get some of those plan calls, we get different shell commands, and we get different verifications. And overall, this process, even though the model execution was a little bit more expensive, it was a lot less manual. So, we could actually accomplish a lot more. So, if I was rebuilding the same task that I had for for this refactor, it would take around 1/5 of the time to accomplish, which is pretty good progress. So, I think all of us realize the scenario that models are substantially better now than they were before. Now, this is really important because it shapes the way we think about the software development life cycle. We think about 2025 and the types of work that we were doing. We were making some small changes. We would give specific code snippets to models. We were just starting to get into this agentic framework of the type of work we can do. And now if we provide a well-constructed spectrum model, it could generally execute it at a very very high capability level.
这里还有个很有意思的点：模型交互的方式本身也变了很多。以前用 o3，在某些类别上几乎没有像样的工具调用；到了 Sonnet 4.6 和 Opus，在现代 harness 里我们能看到 sub-agent、看到 plan 类的调用、各种 shell 命令，还有各种验证动作。总体来说，虽然模型这一侧跑起来贵了一点，但人工介入少了很多，我们能完成的事情反而多得多。所以如果让我现在重做当初那个 refactor 任务，大概只需要原来五分之一的时间——这进步相当可以了。我想在座各位都感受到了：模型现在比以前强太多。这一点非常关键，因为它改变了我们对软件开发生命周期的思考方式。回想 2025 年我们干的那类活：改动都很小，得把具体的代码片段喂给模型，我们那时才刚开始摸到 agentic 的门槛。而现在，只要你给出一份写得足够扎实的 spec，模型基本上能以很高的水准把它执行下来。
[7:19]
And we can see this both in anecdotal experiences as well as some of the thought leadership that has been coming out of the big labs. The this image is one from Anthropic. Let me ask the group a question. Who here has kicked off an agent and realized that either the prompt, the plan, or the requirements were incomplete or missing? A lot of people, yeah? It's it's very frustrating, right? You you're like, "Okay, I'm ready to go. It's 11:00 p.m. or 5:00 p.m. I'm going to set off an agent and then come back." And then you realize there is a there is a critical flaw. Now, the reason I bring this up is it's very important to have a good mental model and understanding how accurate models can be in accomplishing tasks. Who here has seen this meter graph before? I think a decent number of people. So, this is pretty common on actually mapping how much time models can complete tasks of certain categories for. So, the idea being that as models get better and better, they can do longer-running tasks. Now, typically this graph is shared with the 50% accuracy rate, but I think it's much better to actually look at the 80% accuracy rate or higher. And you can see there you can still see a similar exponential trend, but we're no longer claiming that models can accomplish tasks that would take a human 18+ hours.
这一点在个人经验里能看到，在几家大厂实验室放出来的观点文章里也能看到。这张图来自 Anthropic。我问大家一个问题：在座有谁启动了一个 agent 之后，才发现 prompt、计划或者需求本身是残缺的、漏东西的？很多人是吧？这非常让人抓狂，对吧？你想着「好，可以开跑了，现在是晚上十一点或者傍晚五点，我先放个 agent 出去，回头再来看」，结果回来发现里面有个致命缺陷。我提这个是因为：你必须对模型完成任务的准确率有一个良好的心智模型和判断。在座有谁见过 METR 这张图？看起来不少人见过。这张图很常见，讲的是模型能完成多长时长量级的某类任务，逻辑是模型越强，能扛的长任务就越长。通常大家分享的是 50% 成功率那条线，但我觉得看 80% 甚至更高的成功率要有意义得多。你会发现指数趋势依然在，但我们就不能再宣称模型能完成人类需要 18 小时以上才能做完的任务了。
[8:27]
Now, I actually think it's much better to measure the accuracy at 90% or 99% because this is where the mental model is most efficient. You construct a plan, you create a spec, [snorts] you hand it off to an agent, and you're pretty sure that it'll get things done, right? You don't want to be creating a plan or a spec and then have a 50/50 chance of coming back and knowing that you wasted compute and your attention span. Now, if you're kicking off a process that is going to take an hour and it has a 50% chance of completing, there's a very high chance you just wasted that hour and you could have been doing something different. Now, if you think about uh broader evaluation, so Meter does have some more information about their frontier models. So, this is one for for Metis' preview that they did roughly a month ago, and you can see here that generally the success uh rate starts to decline significantly at that 4-hour mark, but even before then at the 15-second mark or even before the 15-minute mark, there are certain tasks that Metis, uh in in all its glory, cannot complete uh effectively and consistently. So, we're making rapid progress in in the AI model space, but we're still not there where you can just kick off an agent and have something be completed reliably.
实际上我认为按 90% 甚至 99% 的准确率来衡量更好，因为那才是这套心智模型最划算的位置：你做好计划、写好 spec，交给 agent，然后你相当有把握它能把活干完，对吧？你不会希望辛辛苦苦写完计划和 spec，回来却只有一半概率成功，白白浪费了算力和你的注意力。如果你启动的是一个要跑一小时的流程，而它只有 50% 的完成概率，那你很可能就是白扔了这一小时，这段时间本来能干点别的。再说更全面的评测，METR 对前沿模型还有更多数据。这一份是他们大概一个月前对 Metis preview 做的测试，可以看到成功率在四小时那个刻度上开始明显下滑；但即便在更早的位置——15 秒量级，甚至还没到 15 分钟量级——就已经有一些任务是 Metis 这么强的模型也没法稳定、有效完成的。所以 AI 模型领域进步很快，但我们还没到「随手丢个 agent 出去就能可靠地把事做完」的阶段。
[9:33]
So, again, this is really important for your software engineering teams and for you as an IC to understand what is your mental model and how are you going to contribute to that. I think what's really important is I think you've been hearing this throughout this conference is that there are a number of different frameworks and primitives that you need to have implemented in order to have good agentic development. And this is no different from from what we found. As we were continuing to mature as an organization and going through our refactor, these are the things that made sure that we can implement the solutions effectively and not waste our time just running in loops with models. So let's go to chapter three. Let's talk about the refactor itself and some of the productivity gains that we saw. So the core idea is that we had these 10 repositories, we put them into a mono repo, and we wanted to build additional features on top of it. So this is this is the result. The previous repos had been around for more than six years, and you can see the progress that was being made it is pretty slow. Part of it was because of the tech debt that was taken on. Other parts were because we didn't have AI coding tools.
所以再强调一遍，对你的软件工程团队、对作为一线工程师的你来说，搞清楚自己的心智模型、以及自己该怎么发挥作用，非常重要。我觉得最关键的一点——这场大会上你应该已经反复听到了——是要想把 agentic 开发做好，你得先把一批框架和基础件（primitives）落地。我们的经验也一样。随着组织不断成熟、refactor 一路推进，正是这些东西保证了我们能有效地把方案落地，而不是陪着模型在原地打转、空耗时间。接下来进第三章，聊 refactor 本身，以及我们看到的生产力提升。核心思路很简单：我们把原来那十个 repo 合并成一个 monorepo，然后在上面继续加新功能。这就是结果。之前那些 repo 已经存在六年多了，你能看到那段时间的推进速度相当慢。一部分原因是背上的技术债，另一部分原因是我们当时还没有 AI 编码工具。
[10:35]
And you can see that within the first six months of the rebuild when we got to parity that we had before, that steepness of that curve is immense. And it didn't slow down after we we kept shipping. So after that that dotted line in the middle there, we kept adding new and new features into the repository. And we shipped a lot faster both in terms of the amount of code even though that's not a great metric, but also the commit rate that we had among developers. And we actually saw that a lot more developers actually joined in into the contributions. So this is a log graph on the commits that we had from the repository initially, and then we slowly on-boarded more and more people. And we had fewer commits because it's much easier to commit code when you're just refactoring and replicating something, but we still kept up that velocity as we were adding product features towards the end. And now almost every developer within the company is committing to this new mono repo even though it might not be their area of expertise, but they might need to make changes to schemas, API calls, and other parts of of the stack. So, let's go into chapter four. Can a modern LLM zero-shot this problem?
你可以看到，在重建的头六个月里，等我们做到和以前功能对齐的时候，那条曲线陡得惊人。而且之后也没有减速，我们一直在往外交付。中间那条虚线之后，我们不断往这个 repo 里加新功能，交付速度快了非常多——既体现在代码量上（虽然代码量并不是个好指标），也体现在开发者的提交频率上。而且我们还看到，参与贡献的开发者明显变多了。这是一张关于提交数的对数图：一开始是这个 repo 最早的提交，然后我们慢慢把越来越多的人接进来。中间有一段提交数偏少，因为在纯 refactor、纯复刻的阶段提交代码要容易得多；但到后期开始加产品功能时，我们依然保住了那个速度。现在公司里几乎每个开发者都在往这个新的 monorepo 提交代码，哪怕那不是他们的专长领域——他们可能只是需要改改 schema、API 调用，或者技术栈里的其他部分。好，进第四章：现代 LLM 能不能零样本（zero-shot）搞定这个问题？
[11:39]
Can I say, "Hey, amazing LLM, go refactor this code base." So, I ran this experiment with GPT 5.5 extra high, and I gave it this goal, giving some of the names of the repositories with the underlying models and other components, and it completed its goal in in 10 minutes and 22 seconds. And it only wrote 2,000 lines of code, which was a little bit fishy. So, I I dug deeper. Um and it actually just implemented a bunch of scaffolding and didn't implement the models. So, you can see here, I did not add a ray serve deployment or bootstrap command yet, right? So, we're still not there where models can self-validate and just one-shot these kinds of problems, but we're getting close. I think in in 6 months, we'll get to the point that we can complete pretty substantial refactors, as we saw in the in the Stripe example, consistently across the board. So, get to the core question. Was this refactor worthwhile? Should we have waited a year to do this refactor as models and harnesses continue to get better, or did it make sense to do it at the time? Now, I'll say the the other side of the argument, right? Things are getting substantially better. Models are getting better. They can call tools better. We have a lot more infrastructure like sandboxes and monitoring frameworks in order for us to actually understand what's happening under the hood with these models. So, taking on technical debt and refactoring later is getting exponentially easier as the days go by.
我能不能直接说一句「嘿，牛逼的 LLM，去把这个 codebase refactor 了」？我用 GPT 5.5 的 extra high 档做了这个实验，给它设定了目标，把几个 repo 的名字连同底层模型和其他组件都告诉它，它在 10 分 22 秒里完成了任务。但它只写了两千行代码，这就有点可疑了。于是我往深里扒了扒——它其实只是搭了一堆脚手架，模型部分压根没实现。你在这儿能看到它自己写着：我还没有加 Ray Serve 的 deployment，也没加 bootstrap 命令，对吧。所以我们还没到模型能自我校验、一把梭解决这类问题的阶段，但已经在逼近了。我觉得再过六个月，我们就能像 Stripe 那个案例里一样，稳定地完成相当有分量的 refactor。回到那个核心问题：这次 refactor 值不值？我们是不是该等一年再动手，让模型和 harness 继续变强？还是说当时做就是对的？这里我先说反方的理由：一切都在快速变好，模型在变强，工具调用能力在变强，我们还有了 sandbox、监控框架这些基础设施，能真正看清模型引擎盖底下发生了什么。所以「先背着技术债、以后再 refactor」这个选项，正在以指数级的速度变得越来越容易。
[13:03]
Now, the problem is that a lot of times when when you build a lot of code and you do this kind of development in an AI AI-native world, it starts looking like some of the legacy code we've we've seen in the past. There's a lot of code written. It's written with low performance or quality, and the broader problem is people don't actually understand what's happening there. So, if you have some issues within the code base or you want to adjust based on customer requirements, it's actually much harder to do so. So, you do have to make sure that there are appropriate guardrails, whether or not you do a full refactor or only a partial one. So, if you ask me, was it worthwhile? I'd say yes. Uh we had built out the patterns that we had earlier with the number of different repos in in order to match customer requirements and demands. Uh it took an amount of time, but we ultimately achieved the goals of the business. Then we came back and refactored and we were able to accelerate. Uh we were able to actually reduce the amount of time the pipeline took. We were able to reduce the costs. We could support larger files, and now we can ship features that would take multiple months in under a week. So, the monorepo refactor, the cleanup was worthwhile, and we have some of the productivity metrics we saw there.
但问题在于，很多时候你在这种 AI 原生的模式下写出大量代码，写着写着它就开始长得像我们过去见过的那些 legacy code：代码量很大，性能和质量都不高，而更麻烦的是，没人真正搞得清里面到底发生了什么。所以一旦代码库出了问题，或者你想根据客户需求做调整，改起来就难得多。因此你必须确保有合适的护栏机制，不管你是做一次彻底的 refactor 还是只重构一部分。那如果你问我，这么做值不值？我会说值。我们之前为了满足客户的需求，用好几个 repo 搭出了那一套模式，花了不少时间，但最终确实达成了业务目标。然后我们回过头来做 refactor，速度就起来了：pipeline 的耗时降下来了，成本降下来了，能支持更大的文件，而且现在原本要几个月才能做完的功能，一周之内就能上线。所以这次 monorepo 的 refactor、这次清理是值得的，前面那些生产力指标就是证据。
[14:09]
The other part is that beyond just shipping velocity, developers actually want to work in this codebase. So, everybody comes along and says, "Hey, can I work in this codebase? It's much cleaner compared to the other ones. Uh can we actually contribute in a way that makes sense?" And a lot of the patterns we have adopted here have spread to other repos within the company. Now, whether or not you refactor, uh the AI delivery system is is a layered approach. Uh you can isolate different parts of your codebase to avoid a full refactor, but there's so many components that you need to keep in mind. And hopefully throughout this conference, you've heard more details about this. But I I really encourage everybody to think about uh the the business value of delivering a a big refactor and the tradeoffs of doing it now versus in the future. So, models will continue to get better, uh but sometimes it's good to pause, build a monorepo, and forge ahead. So, thank you everybody. Uh happy to take any questions.
另一方面，除了交付速度之外，开发者是真的愿意在这个代码库里干活。大家都跑过来问：「嘿，我能不能到这个代码库里做点事？它比其他几个干净太多了。我们能不能用一种说得通的方式往里贡献代码？」而且我们在这里沉淀下来的很多做法，已经扩散到公司里的其他 repo 了。不管你最后决不决定做 refactor，AI 交付体系本身是分层的——你可以把代码库的不同部分隔离开，从而避开一次全量 refactor，但需要顾及的组件实在太多了。希望这次大会上你们已经听到了更多这方面的细节。但我真心建议每个人都想清楚：做一次大 refactor 的业务价值是什么，现在做和以后再做各自的取舍是什么。模型会持续变强，但有时候停下来、把 monorepo 搭好、再往前推，反而是更好的选择。谢谢大家，欢迎提问。
[15:01]
[applause]
[掌声]
[15:15]
Yeah, so the question was before we had multiple repos and did we move into monorepo? Yes, uh we did that. One of the things we found now was that models are much better at navigating multiple repos. So, if you put it into a higher-level folder, right, they could navigate the the file directory. But, for doing that end-to-end testing and verification and deployment, it's still much harder to do with multiple repos. Uh and if you're building a sandbox environment to run sort of a full AI factory, it's it also takes more time to clone repos and get everything set up. So. Yep. Yeah, so the question was uh when we define certain features and requirements that we go back and check them and make changes uh as well as sort of the guardrails framework. Uh we did. I think we got uh 15 out of 17 requirements right when we were going ahead with the refactor. Um and some of the processes that we added for the actual AI engineering verification, that evolved over time. So, for example, when we started, plan mode was just barely coming into Cloud Code and didn't exist in Cursor, but we adopted it as part of our development life cycle. Yep. So, our PR reviews were all all human PR reviews during that refactor. Uh we did some local checks where we ran skills to say, "Hey, review this code. Make sure that it's good." And they're continuing to get more autonomous as time goes on.
对，刚才的问题是：我们之前是多个 repo，后来是不是迁到了 monorepo？是的，我们迁了。我们现在发现的一点是，模型在多个 repo 之间导航的能力已经强很多了——只要你把它们放到一个上层文件夹里，它是能在文件目录里找路的。但要做端到端的测试、验证和部署，多 repo 依然难得多。而且如果你要搭一个沙箱环境，跑一整套 AI 工厂式的流程，克隆多个 repo、把环境都装好也更费时间。就这些。对，这个问题是说：我们定下某些功能和需求之后，会不会回头去核对、去做调整，以及那套护栏框架是怎么回事。我们确实会。我记得我们推进这次 refactor 的时候，17 条需求里做对了 15 条。另外，我们为 AI 工程验证加的一些流程，是随时间演进出来的。举个例子，我们刚开始做的时候，plan mode 才刚进 Claude Code，Cursor 里还没有这个东西，但我们把它纳入了自己的开发流程。对。至于 PR review，那次 refactor 期间全部是人工 review 的。我们也在本地跑了一些检查，用 skills 让它「审一下这段代码，确认没问题」。而且这类工具的自主性还在不断变强。
[16:49]
But, at that point, PRs were really good way for us to build context uh for that repo as we only had a few developers working on it, and we wanted to make sure people understood what had gone into the refactor. Yep. Um in terms of factors, I think that the complexity of the task you can give to a model uh is going to be different, and many more companies will have more scaffolding in terms of actually doing a refactor. So, for example, when I showed the the life cycle of doing the research, the POC work, um validating the code quality, checking hidden assumptions, like you you thought an open-source library had this feature, but it was actually in uh a beta, for example. I think that is going to be much much faster on top of sort of the standard refactoring of hey, here's a file, rewrite it to to match this uh set of requirements. All right. Great. Thank everybody. Have a great rest of the conference.
但在当时那个阶段，PR 对我们来说是建立这个 repo 上下文的绝佳方式——因为参与的开发者只有几个人，我们希望大家都清楚这次 refactor 里到底改了什么。对。至于影响因素，我觉得往后你能交给模型的任务，复杂度会不一样，而且会有更多公司为「做 refactor」这件事搭起配套的脚手架。比如我前面展示的那条链路：做调研、做 POC、验证代码质量、检查那些隐藏假设——你以为某个开源库有这个功能，结果它其实还在 beta 阶段。我认为这部分会快非常多，而不只是那种标准的重构：「这是一个文件，按这套需求重写一遍」。好，非常好。谢谢大家，祝各位在剩下的会程里玩得开心。
[18:04]
[music]
[音乐]