How to Kill the Code Review — Ankit Jain, Aviator

频道: AI Engineer
视频: https://www.youtube.com/watch?v=YgEv7IQzGdM
原文语言: en
统计: 共 14 轮 · Ankit 12

[0:01]

[music]

[音乐]

[0:13] Ankit

Okay. Ooh, hello. Hey everyone. Thanks for joining in. Today, uh we will be talking about how to kill the code reviews. Everyone's favorite topic. I'm Ankit, co-founder of Aviator. At Aviator, we are building AI code verification platform. So, we'll bring in some of the ideas and concepts that we talked about in the uh that we build in our product. Uh but first, let's dive into a little bit. So, a few months ago, I wrote a post on LinkedIn space about uh how to kill code review, creating a framework, a five-layer trust model. So, this model was focused around how do we actually layer by layer build trust into the code that can then be merged without needing line-by-line review. And I got some things right, and I got some things wrong. So, this talk will be about just really diving a bit more more into it. I'm not going to talk about specific layers, but we will talk about some of the concepts that emerged from this session. So, let's just talk about the problem. We are looking today at the volume of code is increasing every day, and we are struggling to keep up. So, when we think about like how long will it take us to actually stop reading code line by line? And the reality is we've already stopped reviewing it.

好，哦，大家好。嗨各位，感谢来听。今天我们要聊的是怎么干掉 code review——大家最爱的话题。我是 Ankit，Aviator 的联合创始人。我们在 Aviator 做的是 AI code verification 平台，所以今天会把我们在产品里做的一些想法和概念带进来讲。不过先稍微铺垫一下：几个月前我在 LinkedIn 上写过一篇帖子，讲怎么干掉 code review，提了一个框架，叫五层信任模型。这个模型的核心是——我们怎么一层一层地给代码建立信任，让它最终不用逐行 review 就能 merge。里面有些我说对了，也有些说错了。所以这次演讲就是把它再往深里挖一挖。我不会去讲某一层具体是什么，而是聊聊从那次讨论里冒出来的几个概念。先说问题。今天代码的产出量每天都在涨，而我们已经跟不上了。所以当我们问「还要多久我们才会不再逐行读代码」的时候，现实是——我们其实早就已经不 review 了。

[1:39] Ankit

There is 861% code churn. That means we are like producing more code. The incidents to PR ratio is increasing. That means even if you're doing reviews today, they're not effective. So, the medium time of review is increasing. We have just increased the bottleneck to from the coding now kind of like coding is solved to now reviewing, where everything just gets stuck there. You're spending four x the time that you were spending before just waiting for the reviews. And today, over 30% of changes are actually getting merged without a review at all. So, let's just think about it a little bit. Think about AI reviews. So, everyone is probably using some form of AI reviews today. When AI write the code and AI review the code, why are we doing it in a UI? Right? Like we open a GitHub, there's kind of like, you know, maybe two or three AI coding agents who are like doing the reviews. You know, you the review goes back passed to the user and or the agent and it gets resolved and they kind of like you're doing back and forth with the agent. Where is human here in the loop? Like you're eventually just looking at, okay, you know, if AI has reviewed it, most of the things probably found, let's just kind of like, you know, do a skimming of it and merge it.

code churn 涨了 861%，意思是我们在产出更多代码。事故数与 PR 数的比值在上升，意思是就算你今天还在做 review，它也已经不管用了。review 的中位时长在变长。我们只是把瓶颈从写代码挪到了 review——写代码这一环现在算是解决了，结果所有东西全卡在 review 这儿。你现在光是等 review 花掉的时间，就是过去的四倍。而今天有超过 30% 的改动，是完全没经过 review 就直接 merge 进去的。我们再想想 AI review 这件事。今天大家多多少少都在用某种形式的 AI review。当代码是 AI 写的、review 也是 AI 做的，我们为什么还要在一个 UI 界面里做这件事？你打开 GitHub，可能有两三个 AI coding agent 在那儿做 review，review 意见回传给用户或者 agent，然后被解决掉，你就在跟 agent 来回拉扯。人在这个环里的位置到底在哪儿？你最后无非就是想：行吧，AI 都 review 过了，该找的问题大概都找出来了，那我扫一眼就 merge 吧。

[2:53] Ankit

So, when AI reviews and nobody reads, we have configured the wrong thing. So, let's just take a step back. Code reviews are not very old. They are maybe 15 to 20 years ago. In 2006 was when Google launched Mondrian internally and they made formal code review as a thing. If you think about it, Windows back in the days, the first versions were actually built without reviews. But if we look carefully, code review is not just about code reviews. You know, obviously we are looking at it like catching bugs, understanding kind of conventions, identifying security issues. But, code review is also about alignment. And this was one piece which was missing from my five layers model that I talked about a few months ago. So, a big part of code reviews is knowledge sharing, mentorship, architectural feedback, onboarding, being able to collaborate. This is Again, if you're doing uh white coding, you're working as a solo project, this is not a talk for you. If you are working in teams, which I believe most of you folks are, collaborating in teams, you're not likely using completely dark factories, orchestrators, where nobody looks at the code. You're actually collaborating in teams, you need to do knowledge sharing.

所以当 AI 在 review、而没有人真的去读的时候，说明我们配置错了东西。往回退一步看，code review 其实没多老，也就十五到二十年。2006 年 Google 内部上线了 Mondrian，把正式的 code review 变成了一件标配的事。你想想，当年的 Windows，最早那几个版本其实是没有 review 就做出来的。但仔细看，code review 从来不只是「review 代码」。我们当然会盯着它抓 bug、统一规范、找安全问题。但 code review 同时也是在做对齐（alignment）。这恰恰是我几个月前那个五层模型里漏掉的一块。code review 里很大一部分是知识共享、带人、架构层面的反馈、新人上手、协作。当然，如果你是在 vibe coding、一个人干单人项目，那这场演讲不是给你听的。如果你在团队里干活——我相信在座大部分都是——你是在团队里协作，你多半也不是那种完全无人值守的黑灯工厂式 orchestrator、没人看代码。你是真的在团队里协作，那你就需要做知识共享。

[4:08] Ankit

Which is the alignment part. And that is the part which is the most important aspect of release. So, for semantic accuracy, we can build better tooling, but alignment must survive. So, let's just kind of like dive into alignment. What does it mean? Like, in today's world, can we actually think of better model than aligning just based on reading line by line code? So, most folks have probably heard about spectrum development by now. So, spectrum development is like, okay, we write a spec, you know, it covers all the details, we pass it to an agent, it generates a code, and then we verify. So, what's wrong here? If you look back in 1970, this is what waterfall model was. You know, you have requirements, you have specification, you implement, and then you verify. But, there's no feedback loop. You know, the spec is written before we identified everything else. Right? That's why today everyone still wants to use your coding sessions, whether it's cloud code, codex, cursor, whatever you're using, you want to interact with the agents. And the reason you're interacting with agents is because there were certain things which were not clear in the spec, and we still need to capture that. And second is, as you implement, you identify more issues, and you never go back and update the spec, because like, if you're doing the spectrum development, it's already done. Once the spec is done, you expect, like, you know, the code will come deterministically. But guess what? LLM is not deterministic. It's going to make decisions itself. So, that's why spectrum development is a great uh methodology, but it falls short in day-to-day software development. But there are some interesting aspects of this, which we should carry forward. The most important part is the intent.

这就是对齐这一块。而这恰恰是发布环节里最重要的一面。语义正确性（semantic accuracy）我们可以靠更好的工具来解决，但对齐这件事必须保住。那我们展开讲讲对齐。它到底指什么？在今天这个环境下，除了逐行读代码，我们能不能想出一个更好的对齐模式？现在大部分人应该都听过 spec-driven development 了。它的逻辑是：我们先写一份 spec，把细节都覆盖到，然后丢给 agent，它生成代码，我们再去验证。问题出在哪？回头看 1970 年，这不就是瀑布模型吗——先有需求，再有规格说明，然后实现，最后验证。但它没有反馈回路。spec 是在你还没搞清楚其他所有事情之前就写好的。所以到今天，大家还是想用交互式的 coding session，不管你用的是 Claude Code、Codex、Cursor 还是别的什么，你都想跟 agent 来回聊。而你之所以要跟 agent 交互，就是因为 spec 里有些东西没讲清楚，那些东西还是得被捕捉下来。第二点是，你在实现过程中会发现更多问题，但你从来不会回头去更新 spec——因为按 spec-driven 的玩法，spec 已经定稿了。spec 一写完，你就默认代码会确定性地长出来。可问题是，LLM 不是确定性的，它会自己做决策。所以 spec-driven development 是个很好的方法论，但放到日常软件开发里就不够用。不过它里面有些有意思的东西我们该继承下来，最重要的一点就是「意图」（intent）。

[5:53] Ankit

An intent doesn't only live in the spec. Intent live in your Jira ticket. That's the goal, right? Like it's kind of like where you express what we want to do. It lives in your PRDs. It's kind of like details, like it's a plan. But most importantly, it lives in your prompts today. This is where the real decisions are being made. You start with like, okay, this is a Jira ticket I'm going to look at, but you're going back and forth with the agent, and this is where all the user decisions are being made. But what we do today is we create a change, we create a pull request, and then we throw away the prompts. And this is one of the things that we need to change. Let us first talk also about semantic accuracy because like, you know, you're saying, "Hey, Ankit, okay, I understand the alignment part, but like there are still bugs in the code. Who's going to look at that?" LLMs are also not great at this, and we already talked about how AI agents may reviewers may not always be perfect. So, this is where I introduce you to the concept of AI slop registry. So, think about this. We are reviewing If you're reviewing code today manually, and I expect everyone should be doing some degree of this.

意图不只活在 spec 里。意图活在你的 Jira ticket 里——那是目标，是你表达「我们想做什么」的地方。它活在你的 PRD 里，那是细节，是计划。但最重要的是，今天它活在你的 prompt 里。真正的决策就是在那儿发生的。你确实是从「我要处理这个 Jira ticket」开始，但接下来你跟 agent 一来一回，所有用户侧的决策都是在这个过程里做出来的。可我们今天的做法是：做完改动、开一个 pull request，然后把 prompt 全扔了。这是必须要改的一件事。我们也顺带聊聊语义正确性，因为你可能会说：「Ankit，对齐这块我懂了，可代码里还是有 bug 啊，谁来看？」LLM 在这方面也不算强，我们前面也说了 AI agent 当 reviewer 未必总是靠谱。所以这里我要引入一个概念，叫 AI slop registry（AI 垃圾模式登记册）。你想想看：如果你今天还在人工 review 代码——我觉得每个人多少都该做一点——

[7:02] Ankit

We are essentially possibly identifying the same issues over and over again. Can we actually capture these concepts and codify them so that we don't have to always create those review feedback one by one. You actually also have all of those things automatically identified. The beauty of this is if you do it a few times, you now build sort of like a system which actually learns over time. So, think of this as sort of like you know, you know, you're doing more training on top of the standard LLM that you have actually extracted like built on top of. So, AI slop registers now can create better results because it's trained it's learning from the review experience that you as humans are providing. Every recurring comment is now a guardrail that you don't have to review again. Okay, so let's kind of like try to put both of these alignment and the semantic accuracy together. It is two halves of the same problem. We are trying to understand what are the core mechanics of review. How do we actually break it down into two parts which is the alignment and the semantic accuracy and bring them together into a single loop. So, first thing is you could take your session and you capture the user responses and that essentially forms your acceptance criteria.

——那你其实是在一遍又一遍地发现同样的问题。那我们能不能把这些模式捕捉下来、把它编码成规则，这样就不用每次都一条条手写那些 review 意见？这些东西完全可以被自动识别出来。妙就妙在，你这么做几次之后，就建起了一个会随时间学习的系统。你可以把它理解成：在你手上那个标准 LLM 之上，又叠了一层训练、一层提炼和沉淀。于是 AI slop registry 能给出越来越好的结果，因为它是在从你们人类提供的 review 经验里学习。每一条反复出现的评论，现在都变成了一道护栏，你不用再 review 第二遍。好，我们试着把对齐和语义正确性这两件事合起来看。它们是同一个问题的两半。我们要搞清楚 review 的核心机制是什么，怎么把它拆成对齐和语义正确性两部分，再把它们收进同一个闭环。第一步是：拿你的 session，把里面用户的回复抓出来，这些本质上就构成了你的验收标准（acceptance criteria）。

[8:30] Ankit

The acceptance criteria then tied with your AI slot register that you are now constantly maintaining finally creates a test plan. And this is the test plan which then gets verified. This is part of the system that we are building is the verification system where it spins up a preview, takes your test plan and make sure it actually works end-to-end. Even if the code looks right, does it actually work? So, this verification part has become interesting. And now the kicker is this is now your review surface. You're not reviewing code line by line, but rather you're looking at the evidence of what was the intent, did the user actually implement the capability that was defined in the intent, and did the behavior actually meet the requirements that we had in the as acceptance criteria. So, you're still having the architectural decisions, you're still having these arguments, but the review surface changes. So, just giving a walk through of like how we have built our system. The session becomes a criteria, so all these decisions that you are doing here with the agent, you're asking uh you know, you're providing this feedback. Like even if there is a simple task, many times you're going back and forth. Agent will stop to ask questions.

验收标准再和你持续维护的那个 AI slop registry 结合起来，最终生成一份测试计划（test plan）。这份测试计划就是拿去被验证的东西。我们在做的系统里有一块正是这个验证系统：它会起一个 preview 环境，拿着你的测试计划，确认它端到端真的跑得通。代码看起来对是一回事，它到底能不能用是另一回事。所以验证这一环变得很有意思。而关键在于——到这一步，它就成了你新的 review 界面。你不再逐行 review 代码，而是去看证据：意图是什么？开发者有没有真的实现意图里定义的那个能力？行为有没有真的满足我们在验收标准里写的要求？架构层面的决策你照样在做，照样在争论，但 review 的落点变了。我大概讲一下我们的系统是怎么搭的。session 变成验收标准——你跟 agent 之间做的这些决策、你给的这些反馈，哪怕是个很简单的任务，你也常常要来回好几轮，agent 会停下来问你问题。

[9:49] Ankit

These are the decisions that we need to capture. This is the intent. This is what makes your review, makes your collaboration more valuable. This is how you teach your junior engineers on how to improve over time. These are the decisions which make a software engineer valuable today. We convert those into an acceptance criteria. This is where you can also leverage LLM to do so. So, you capture these user uh decisions and make sure you can actually create a test plan based on this. I know test plan creation is always painful, which is where I would always recommend people to use LLM for this purpose. And finally, the criteria plus invariant is what makes the test plan. Uh and we build the verification systems to actually capture the test plan, run your previews, and be able to test based on this test plan. So, even imagine if you're building a new feature, you don't have to maintain tests at all. This is creating tests in real time, and this is where you can leverage the power of LLM because the test plan maintenance and creation can be really painful, but the value of human in the loop here is the governance and the review part, and the part where you're reviewing the test plan and not the code.

这些就是我们要捕捉的决策。这就是意图。这也是让你的 review、你的协作变得有价值的东西。这是你带初级工程师、让他们持续成长的方式。这些决策才是今天一个软件工程师真正值钱的地方。我们把它们转换成验收标准——这一步你完全可以借 LLM 来做。你把这些用户决策抓出来，确保能据此生成一份测试计划。我知道写测试计划一向很痛苦，所以这块我特别推荐大家交给 LLM。最后，验收标准加上不变量（invariant），就构成了测试计划。而我们做的验证系统，就是去接住这份测试计划，起你的 preview 环境，按测试计划把测试跑起来。所以你想象一下，哪怕你在做一个全新的功能，你根本不需要维护测试——它是在实时生成测试。这正是能借上 LLM 力气的地方，因为测试计划的编写和维护真的很折磨人。而人留在这个环里的价值，在于治理和 review——你 review 的是测试计划，不是代码。

[11:13] Ankit

Right? I know it's been like over 20 years we came up with test-driven development. This in some ways is closer to behavior-driven development, where the test plan is now something which even you can share with your product managers, your designers. Everyone can participate because these things are now in English. At the same time, we have now deterministic verification, which actually verifies whether the particular test criteria has been met or not. Let's move on. So, this is where I would say the system is not supposed to be perfect. It's deterministic where it can be, but LLM where you must. Not everything can be deterministic. Not every system can be built in a way which is 100% built on deterministic systems. This is where you use LLM as a fallback. Let me give an example. Like if you're making a change in your web application, what you can do is it creates a test plan of what the behavior changes. Let's say you introduce a new payment form. So, it creates a new payment form. The verification here is does the payment system changes? Uh an AI agent can go and browse through your application to fill out a form and capture screenshots as evidence. And then take those screenshots as well as your database snapshots to identify whether the criteria was met.

对吧？测试驱动开发（TDD）提出来到现在都二十多年了。我们这套东西某种意义上更接近行为驱动开发（BDD）——测试计划现在是你可以拿给产品经理、拿给设计师看的东西。所有人都能参与，因为这些内容现在是用大白话写的。同时我们又有了确定性的验证，能真正判断某一条测试标准到底满足没满足。往下讲。所以我想说的是，这个系统不必做到完美。能确定性的地方就用确定性，非得用 LLM 不可的地方才用 LLM。不是所有东西都能确定性，也不是每个系统都能 100% 建在确定性机制之上。LLM 在这里是兜底方案。举个例子，你在 web 应用里改了个东西，系统会针对行为变化生成一份测试计划。比如说你新加了一个支付表单，那验证的问题就是：支付流程有没有按预期变化？一个 AI agent 可以去你的应用里点一遍、把表单填掉、截图留证。然后把这些截图连同数据库快照一起，用来判断验收标准有没有被满足。

[12:52] Ankit

So, the screenshot testing or like sand flying can then still be done by agents. But at the same time, you're creating more solid evidence, which now a reviewer can look at and build more confidence that this actually works. So, now reviewers are reviewing the intent, not the diff. You're reviewing the intent decisions. Uh what we said to build out. What we tried and rejected, and capture all of these things from the sessions. Remember, capturing it from the sessions is is key. If we try to build it from the code, you'll end up in the same situation that we were talking about I think Dex was talking about yesterday, which is if your code is built by the same agent which is actually building a test plan, it's not going to build a test plan which will actually catch issues. So, that's why it's important to actually use the session information to build out a test plan. You can discuss architectural decisions. So, how you're creating the data models, how these services interact with each other. So, you've moved one level above. So, you're essentially instead of like reviewing line by line, you're actually having discussions on architecture, which are very critical for any kind of collaboration.

所以截图测试、在页面上点一遍这类检查，照样可以交给 agent 去做。但同时你也在积累更扎实的证据，reviewer 看到这些证据，对「这玩意儿是真的能跑」这件事就更有信心。于是现在 reviewer review 的是意图，不是 diff。你 review 的是意图层面的决策——我们说了要做什么、我们试过又放弃了什么，把这些全都从 session 里捞出来。记住，从 session 里捞是关键。如果你想从代码反推出来，你就会掉进我们前面说的那个坑——我记得 Dex 昨天也讲过这个——如果生成测试计划的，就是那个写代码的同一个 agent，那它生成的测试计划根本抓不出问题。所以一定要用 session 里的信息来构建测试计划。你可以讨论架构决策：数据模型怎么设计、这些服务之间怎么交互。你等于是往上抬了一层——不再逐行 review，而是在讨论架构，而架构讨论对任何形式的协作都至关重要。

[14:05] Ankit

And then you look at the evidence, everything that was collected from the verification. So, here's a homework for everyone. Go home and mine your last 1,000 review comments and build out a AI slot register for the things which are repeatable. So, a vast majority of the comments that you're providing in your code review are something that we repeat over and over again. This compounds with every merge PR. Every time you capture something as a register, you don't have to capture that comment again. And this is where you can actually codify some of the best practices of doing code, maintain semantic accuracy, at the same time, do not lose the collaboration part of the review. It does follow a J curve. So, pain is real. Uh you will have to spend some time to actually make it pay off because initially creating a registry can take some time. And this is where I would recommend you folks can come and try out our product. So, code review is not just about code review. It is about uh really getting the alignment. And where we can build better tools is creating the semantic accuracy and defining your AI slot register. So, if you remember one thing from today, remember code review is not just about code review, it is about getting the alignment.

然后你再去看证据，看验证环节收集到的一切。所以给大家留个作业：回去把你们最近的 1000 条 review 评论挖一遍，把那些会重复出现的东西做成一个 AI slop registry。你在 code review 里给出的绝大多数意见，其实都是在重复、重复、再重复。而这东西会随着每一个 merge 的 PR 复利累积。每次你把一条东西沉淀进 registry，你就不用再写那条评论了。这也是你把写代码的最佳实践真正固化下来的地方——保住语义正确性，同时又不丢掉 review 里协作的那一面。它确实会走一条 J 曲线，前期的痛是真的，你得先花点时间它才开始回本，因为一开始建 registry 是要下工夫的。这块也欢迎大家来试试我们的产品。所以说，code review 不只是 code review，它真正的意义是拿到对齐。而我们能用更好的工具去解决的，是语义正确性，以及把你的 AI slop registry 定义出来。所以如果今天只带走一句话，那就记住：code review 不只是 code review，它是为了拿到对齐。

[15:38] Ankit

And yes, we are piloting our new product called Verify. Please join and be our early design partners. We are working with a few companies to uh pilot out a new verification system. This combines both the alignment side of things as well as building tools and capabilities for detecting semantic accuracy using the AI slot register. Thank you, everyone. Thanks for joining.

另外，我们正在试点一款新产品，叫 Verify。欢迎大家来做我们的早期设计合作伙伴。我们正在和几家公司一起试跑这套新的验证系统。它把对齐这一侧，和用 AI slop registry 检测语义正确性的工具与能力，合到了一起。谢谢大家，感谢各位来听。

[16:06]

[applause]

[掌声]