Building the Automated AGI Lab: Core Automation's Jerry Tworek and Rohan Anil
频道: Sequoia Capital
视频: https://www.youtube.com/watch?v=2RJiaf0SY8s
原文语言: en
统计: 共 88 轮 · 主持人 19 · Jerry Tworek 22 · Rohan Anil 25 · 提问者 2
[0:00]
If I play football, for example, it looks very very closely to reinforcement learning. I get all a lot of times and every time I adjusted a little bit and I see if it roughly matches what I what I wanted and there are there are there are some self-reinforcement happening. When I learn mathematics, it's a very different type of thing. It's it's like reading about hard concepts and thinking about them very deeply inside my head until things click and I until until I have them connected. And both of those in some way are learning from experience. They are just very different. We probably are spending the most compute than ever on learning from experience, but there reinforcement learning is is not the end of learning from experience and there will be better approaches that researchers will be coming up in the coming [music] years on how to how to use that data.
比如我踢足球，那个过程跟 reinforcement learning（强化学习）非常非常像：我可以试很多次，每次微调一点点，再看结果跟我想要的大致对不对得上，中间会有某种自我强化在发生。但我学数学的时候，完全是另一回事——那是读那些很难的概念，在脑子里反复深想，直到某一刻突然通了，直到我把它们串起来。这两种在某种意义上都是「从经验中学习」，只是形态差别极大。我们今天投在「从经验中学习」上的算力大概是史上最多的，但 reinforcement learning 并不是「从经验中学习」的终点，未来几年研究者一定会拿出更好的办法来使用这些数据。
[0:55]
[music]
[音乐]
[1:00] 主持人
Jerry Rohan, thank you so much for joining us today. The two of you are the founders of Core Animation, one of the hottest new labs in San Francisco right now. And before starting Core Animation, you led some of the most important research projects of the AI era. Jerry, you were VP at OpenAI, where you worked among amongst other things on running the strawberry and reasoning teams. And Rohan, you were one of two of the four pre-training leads at Gemini. And before that, led a lot of the fundamental AI research at Google Brain and were the fix-it guy across Google and then at Anthropic. And so, between the two of you, you've seen more than a fair your fair share of what the world looks like in terms of doing frontier research. And so, I'm very very excited to dig in. Um, let's start with you, Jerry. You tweeted a very spicy take recently. The first step to replacing transformers is appreciating deeply how far they were able to carry us. Is that a eulogy for the transformer? What does that mean?
Jerry、Rohan，非常感谢两位今天来。你们俩是 Core Automation 的创始人，这是眼下旧金山最受关注的新实验室之一。在创办 Core Automation 之前，你们各自主导过 AI 时代最重要的一批研究项目。Jerry，你在 OpenAI 做 VP，做过很多事，其中包括带 strawberry 和 reasoning 团队。Rohan，你是 Gemini 预训练的四位负责人之一，在那之前你在 Google Brain 主导了大量基础 AI 研究，还是整个 Google 内部的「救火队员」，后来又去了 Anthropic。所以你们两位加起来，见过的前沿研究世界的样子比绝大多数人都多，我特别期待今天能聊深一点。Jerry，从你开始吧——你最近发了一条挺辣的推：「取代 transformer 的第一步，是深刻地体会它究竟把我们带出了多远。」这是在给 transformer 写悼词吗？这话什么意思？
[1:57] Jerry Tworek
Thank you very much for inviting us here, Sonia. I feel like a lot of my interviews these days is explaining my tweets. And what did I
非常感谢你邀请我们来，Sonya。我感觉我最近很多访谈都在解释自己发的推。我到底是什么意思——
[2:06]
[laughter]
[笑声]
[2:06] Jerry Tworek
What did I mean? But appreciating Transformer means like understanding what it does well. So, if you're not solving the problems that it is solving well, you have to focus on its on its weaknesses. You have to You have to understand good parts and bad parts. And it's very easy in a lot of the work what people are doing in architectures is trying to make Transformers cheaper. They are trying to make Transformer more efficient. I I very rarely see people thinking about how do we make Transformers more powerful, trying to do more express. But But like seeing someone's weak parts and seeing someone's strong parts are almost almost almost the same thing. It's just just understanding the shape of Transformer a little bit more.
我到底想说什么？「体会 transformer」的意思是，你得真正搞懂它擅长什么。如果你要解决的不是它已经解决得很好的那些问题，你就必须盯着它的弱点。你得同时理解它好的部分和不好的部分。而现在架构方向上的很多工作很容易滑向一个方向：大家都在想怎么把 transformer 做得更便宜、更高效。我很少看到有人在想，怎么让 transformer 更强大、表达力更强。但看清一个东西的短板和看清它的长处，几乎是同一件事，无非就是把 transformer 的形状再看清楚一点。
[2:44] Jerry Tworek
What I think right now we are in this stage. We got to really really good at training really really big models. We mastered two algorithms. We mastered pre-training at a large scale. And we mastered reinforcement learning at a large scale. And I'm asking myself a lot what is next in machine learning. I think at this moment what the bottleneck is to better models and to smarter systems is the architecture itself. It is this moment to revisit uh the train we've been riding for the last 6 years of trying to add more and more parameters to essentially two of the same operations, which is MoE and and attention. And when I am when I'm thinking about it, like what's what like where we are today and what we are doing, I am thinking a lot about about like what Codex and what Cloud Code are doing for us. And I'm really really appreciative of those systems and of the coding and of the workflow automation and of the systems and of the products that we have today that we essentially have built over those those 6 years of scaling. And I think I think this is this is the first step of thinking. Like, what is the If we want to work on the replacement, we need to like see where we are, what problems we have solved to like start seeing what the next stage is, what problems we haven't solved yet, what what kind of are we missing. And this is this is kind of whenever whenever I use Codex and I am successful at a task, I also start thinking, why did I why didn't I try to push that thing harder?
我觉得我们现在处在这么一个阶段：我们已经把「训练超大模型」这件事练得非常非常好了。我们掌握了两个算法——掌握了大规模的 pre-training（预训练），也掌握了大规模的 reinforcement learning。我一直在问自己，机器学习的下一步是什么。我认为此刻通往更好的模型、更聪明的系统，瓶颈就是架构本身。现在是时候回头审视我们过去六年一直坐着的这趟车了——不断往同样的两种运算（也就是 MoE 和 attention）上堆越来越多的参数。当我在想「我们今天到了哪儿、我们在做什么」的时候，我会想很多 Codex 和 Claude Code 到底在替我们做什么。我对这些系统、对编码、对工作流自动化、对我们今天拥有的这些产品是真心心怀感激的——它们本质上就是这六年 scaling 堆出来的。我觉得这是思考的第一步：如果我们想做出替代品，就得先看清我们现在在哪儿、已经解决了哪些问题，才能开始看清下一阶段是什么、还有哪些问题没解决、我们还缺什么。所以每次我用 Codex 顺利完成一个任务，我同时也会开始想：我刚才为什么没有把它再往前推一把？
[4:15] Jerry Tworek
Whenever whenever I come to work, there are a lot of things I do with Codex, but I still come to work. I still ask it to do certain things for me. And I'm always asking myself, why am why am I even needed there? Why is uh core automation is name and its concept is we want to we want to be automating tasks. And why why why why are those things not yet automated? Why why why is not Codex doing everything for me? And this is this is the question of like the research, where we want to go. And with that research, I'm trying to think, what kind of models, what kind of what kind of systems do we need, what kind of qualities do we need what that that that we don't we don't have today. And that's that's what I'm thinking a lot these days.
我每天还是得来上班。有很多事我交给 Codex 做，但我还是得来上班，还是得让它替我做某些事。我一直在问自己：为什么这里还需要我？Core Automation 这个名字和它背后的理念，就是我们想把任务自动化掉。那为什么这些事到今天还没被自动化？为什么 Codex 不能把所有事都替我做了？这就是研究要往哪儿去的问题。围绕这个研究我在想：我们需要什么样的模型、什么样的系统、什么样的能力——那些我们今天还不具备的能力。这就是我这段时间想得最多的事。
[4:54] 主持人
And you have the starting premise of the architecture is the issue, which I think is a contrarian point of view. So, what's what led you to that point of view? What did you see that made you think the architecture was the issue?
你的出发前提是「架构才是问题所在」，我觉得这在今天算是个反共识的看法。是什么把你推到这个结论上的？你看到了什么，让你认定问题出在架构？
[5:05] Jerry Tworek
It's fundamentally what is the issue. It's it's it's like comes back from that from from the previous implication. What I I think is the issue is that the models are being trained in the lab and are being deployed in the in the real world. That is that that is the fundamental fundamental tension that is that is there. And mm like a bit of a bit of my disappointment came comes from my my my personal story. Whenever we were starting uh the research and progress on scaling up reinforcement learning at OpenAI, I basically believed that scaling up reinforcement learning is a necessary stepping stone on a path to AGI uh since since since I started like uh working working at Open AI. And I was always reinforcement learning maximalist. I was always believe this is what we need to focus on. This is what we need to do. I've seen LLMs being scaled up to to to higher and higher levels through GPT-3 to GPT-4. And we're still doing very little RL. And I had this this internal belief that the moment we start scaling up RL we'll we'll we'll solve everything. We'll we'll we'll be able to solve solve all the problems. And we eventually started scaling up RL. I was I was just in there. I was I was in the center of it.
这从根本上就是问题所在。它其实接着刚才那层意思：我认为问题在于，模型是在实验室里训练出来的，却被部署到真实世界里。这就是最根本的那个张力。我的一部分失望来自我自己的经历。当年我们在 OpenAI 刚开始做「把 reinforcement learning 扩上去」这件事的时候，我基本上从进 OpenAI 那天起就相信，扩展 RL 是通往 AGI 路上一块必经的踏脚石。我一直是个 reinforcement learning 的最大化主义者，一直相信这就是我们该聚焦的方向、该做的事。我看着 LLM 一路从 GPT-3 扩到 GPT-4，越扩越大，但我们做的 RL 还是非常少。我心里一直有个信念：等我们真的开始把 RL 扩上去，一切就都解决了，所有问题我们都能解决。后来我们真的开始扩 RL 了，我人就在里面，就在最中心的位置。
[6:20] Jerry Tworek
I was thinking here we are. If you ask Jerry in 2024, when do we get AGI? I would say 2025 will be will be that year. This is this is where we solve everything. And I saw us training model after model. This model was getting better and better. All the benchmark scores were going up. And did we also solve all the real-world tasks at that moment? Unfortunately, unfortunately not. We we we we still have work. And I realized there was this bit of distinction as all the benchmarks that we are evaluating our models, they were essentially the same thing as we were training the models on. Like all the evals and training side tasks are the same sides of the coin. But the real-world distribution and real-world task is much messier, much murkier, much more much more different. Our our training data didn't really replicate the real-world use cases. And despite us basically maximizing all the task, if you see ask anyone training models, "Hey, what is one of your main issues?" "I don't have hard enough tasks. I don't I don't have what to what to train our model on." Yet we are still not covering the the entirety of the of the real-world distribution. From that, my conclusion is we need to have models that learn at test time. We need to have models that learn with users on their data, on their real-world task, on the real world distribution. And there when you ask why why why why why why don't we have that today? Why why why why are transformers not not learning any anywhere? And there are essentially two types of learning that we could be doing at test time. We could be doing in-context learning essentially of transformers, which is it it it doesn't have fundamental problems of catastrophic forgetting. It doesn't have that issue.
我当时想，来了，就是现在。如果你在 2024 年问 Jerry「AGI 什么时候到」，我会说 2025 年就是那一年，就是我们把一切都解决掉的那一年。然后我看着我们一个模型接一个模型地训，模型确实越来越好，所有 benchmark 分数都在涨。那我们同时也把真实世界的任务都解决了吗？很遗憾，没有。我们还有活要干。我意识到这里面有个区别：我们用来评估模型的所有 benchmark，本质上跟我们训练模型用的东西是同一批东西。所有 eval 和训练侧的任务其实是同一枚硬币的两面。但真实世界的分布、真实世界的任务要乱得多、浑得多，也不一样得多。我们的训练数据并没有真正复刻真实世界的使用场景。而且尽管我们几乎把所有任务都刷满了，你去问任何一个训模型的人「你最头疼的问题之一是什么」，答案是「我没有足够难的任务，我不知道该拿什么去训模型」——可我们依然没有覆盖真实世界分布的全貌。由此我的结论是：我们需要能在 test time（测试/推理时）学习的模型。我们需要模型能跟着用户学、在用户自己的数据上学、在他们真实的任务上学、在真实世界的分布上学。那接下来的问题就是：为什么我们今天没有这种东西？为什么 transformer 在任何地方都不学习？在 test time 能做的学习本质上有两类。一类是 in-context learning（上下文内学习），也就是 transformer 天然会的那种——它没有 catastrophic forgetting（灾难性遗忘）这种根本性毛病，它不存在那个问题。
[8:08] Jerry Tworek
It is pretty data efficient. So that that is great, but it's not very scalable. We only can have so much of it. It is limited and it has some more even more of mechanical limitations of what actually are you doing when you when you build context? But maybe maybe we can we can come back to it later. But we have we have in-context learning, which is very limited and very very small amount of data. Whenever I'm using Codex roughly after around 20 minutes of usage, I need to I need to compact it and move and move it afterwards, which is not that much not that much data. If we have all all we can learn is for for 20 minutes, it's not that much. And the second thing is fine-tuning. We could try to continuously fine-tune our models, but then those have the issues of catastrophic forgetting. We have issues of very low data efficiency. And neither those are very solvable. Neither of those are very easy to uh to find ways. People have been trying. If they was were easy to solve, someone already solved it. So my personal belief is we need to find an algorithm that we can we can meta learn, that we can express on the architectural layer, that can represent how does how does learning look like. How does learning look like that can work on much much longer horizons.
它的数据效率也不错。这些都很好，但它不可扩展，我们能塞进去的量就那么多。它是有上限的，而且还有一些更偏机械层面的限制——你在「构建 context」的时候到底在干什么？这个也许我们待会儿可以回头再聊。总之我们有 in-context learning，但它很受限，能装的数据非常少。我用 Codex 的时候，大概用上 20 分钟左右就得做一次 compaction（上下文压缩）再往下走，那点数据量真不算多。如果我们能学的只有 20 分钟的量，那太少了。第二类是 fine-tuning。我们可以试着持续地 fine-tune 模型，但那就会撞上 catastrophic forgetting，还会撞上数据效率极低的问题。这两条路都不好解，都不容易找到出路。大家都试过。要是好解，早就被人解掉了。所以我个人的信念是：我们得找到一种算法，一种可以 meta learn（元学习）出来、能在架构层面表达出来的算法，它能刻画「学习到底长什么样」——一种能在长得多的时间尺度上起作用的学习方式。
[9:24] 主持人
Mhm. Do you expect the architecture will look transformer-like? Cuz my my my understanding from the from the chief seats is that you know, OpenAI have been trying to scale up reinforcement learning for a long time, and it wasn't until the transformer came about that it seemed like there was an even kind of scalable prior on the world upon which to even scale RL. And so, how do you even go about trying to think about scaling up this this new resume?
嗯。你觉得这个新架构会长得像 transformer 吗？因为我在看台上的理解是，OpenAI 其实很早就在试着把 reinforcement learning 扩上去，一直到 transformer 出现，才算有了一个足够可扩展的「世界先验」，RL 才有东西可以往上扩。那你要怎么去想「把这套新东西扩上去」这件事？
[9:50] Jerry Tworek
Yeah, it's a it's a it's a great question. I think those two things happened at the same time, but if anything that that happened there was was mostly about about economics, because technically you can scale up LSTMs. Just no one no no one really really dared to go in that direction. And they did scale much much more poorly. They are they are they are scaling in a scaling loss paper there is presented a comparison of LSTMs and Transformers. And fundamentally scaling law of Transformers was better. There is a world where we never invented Transformers and we would be scaling LSTMs and we would be having some models. But because they would be much more expensive to train and much less impressive as a product, we would have we would have just this worse experience and maybe no one would be able to convince people to spend as many dollars training that those gigantic LSTMs because we wouldn't get a market return. The the the majestic thing about Transformer, which goes back to like why why why why do we have to appreciate Transformers so deeply, is that Transformers are economically valuable. The training them the cost of training them is lower than the revenue that they that they generate, which is which is magic of machine learning and it's not guaranteed by itself. But for for LSTMs it probably wouldn't be that way, which which which may have happened. But you can in many ways you can scale most of the architectures.
这问题很好。我觉得那两件事是同时发生的，但真要说的话，那里发生的主要是经济学上的事。因为技术上你是可以把 LSTM 扩上去的，只是没人真敢往那个方向走。而且它们 scaling 的效果确实差很多。那篇 scaling law 论文里就有 LSTM 和 transformer 的对比，从根本上讲 transformer 的 scaling law 更好。存在一个平行世界：我们从没发明 transformer，我们就一路扩 LSTM，也能得到一些模型。但因为训练它们贵得多、作为产品又没那么惊艳，我们只会得到一个更差的体验，而且可能没人能说服别人掏那么多钱去训那些巨型 LSTM，因为拿不回市场回报。transformer 真正神奇的地方——这也回到「为什么我们必须如此深刻地体会 transformer」——在于 transformer 在经济上是成立的：训练它的成本低于它带来的收入。这是机器学习的魔法，而这件事本身并不是天经地义的。换成 LSTM，很可能就不是这样了，那种情形本来是可能发生的。但从很多角度看，大部分架构其实都是可以扩上去的。
[11:11] Jerry Tworek
I think I think a lot of reasons why people didn't scale things before was because researchers before OpenAI had a lot of reluctance to scaling. It was often seen as unscientific and research in algorithm was was providing how do we become more and more efficient? How do we for the same compute budget get better better results? And it was it was a bit of a bet by OpenAI at that moment. Try to say, "Hey, we don't care about better and better algorithms. We care about smaller and more scalable algorithms." And how do we how do we pour more and more compute and get better and get better results, which OpenAI was criticized repeatedly by many people in the community for for a long time. But thanks to that, we have we have the models that we that we have today. Uh and I think there are tons of architectures that can be scaled up. And I am part of the core automations mission, and our our belief is that a lot of architectural research happened at too small scale for too long time. A lot of people are trying to say, "Hey, let's let's try to first try our architecture on a on a on a small data set on a small in a small computer regime, and then and then and then see where where where where where where it scales only after after you prove itself." But for example, when you do work on reinforcement learning, you know that to get to any interesting results, you only you need certain level of compute to even see the capabilities in a model. Reinforcement learning needs a baseline of of of ability to only to only start working. So so where I am coming from, probably there are many architectures that need a baseline of compute to even start doing anything anything interesting and anything useful.
我觉得以前大家不去 scale 的一大原因是，OpenAI 出现之前，研究者对 scaling 有很强的抵触，那常被视为不够科学；算法研究的价值取向是「怎么做得越来越高效」「同样的算力预算怎么拿到更好的结果」。当年 OpenAI 算是下了一个赌注，说：我们不在乎算法越来越好，我们在乎的是算法更小、更可扩展。我们要的是怎么把越来越多的算力倒进去、拿到越来越好的结果。就为这个，OpenAI 被社区里很多人反复批评了很长时间。但正因为如此，我们今天才有了现在这些模型。我觉得能被扩上去的架构多得是。这也是 Core Automation 使命的一部分，我们的信念是：架构研究在太小的规模上做了太长时间。很多人的做法是：先在小数据集上、小算力条件下试自己的架构，等它证明了自己，再看它往上扩会怎么样。但举个例子，你要是做过 reinforcement learning 就知道，想得到任何有意思的结果，你需要一定量级的算力，才能在模型里看到那些能力冒出来。RL 需要一个能力基线才可能开始起作用。所以我的看法是：很可能有很多架构，也需要一个算力基线，才可能开始做出任何有意思、有用的事情。
[12:47] 主持人
Can I ask you then maybe a touchy question?
那我能问一个可能有点冒犯的问题吗？
[12:49] Jerry Tworek
Please do.
请问。
[12:49] 主持人
If you need a baseline of compute, that sounds like a job that would be well served inside of a big research lab. Why start a company to go do this?
如果你需要一个算力基线，那这听起来像是大研究实验室里更适合干的活儿。为什么要为这件事出来创业？
[12:57]
[laughter]
[笑声]
[12:58] Jerry Tworek
It's a it's a it's a great question, and it's I think in in in many ways it's likely a timing thing, timing issue. Market is right now in a in a very specific place where the biggest and the most successful labs, by coincidence or by fate, are probably in the most competitive market fight ever right now, which makes them not very keen on trying different paths, trying alternatives. If Transformer is profitable and if you can spend more efforts and more resources scaling Transformer to win in the next quarter, it's very hard to put at least a lot of attention and a lot of energy to work on something that will that will maybe better or maybe or maybe will will redefine the field in a year or two. So so I think the big biggest labs and I talked to basically all of them don't don't have that much interest in trying the alternatives to to Transformer. And the labs that are not the biggest are doing whatever they can to do what they what the what the what the most successful labs are doing. And everyone is trying to trying the same the same coding agent. If you look at the last week's releases, everyone everyone is is trying to release a coding agent right now. And and I think we need different paths and different and different approaches here. So so that's what that's the niche in the ecosystem we are trying to trying to fill in.
这问题问得很好。我觉得从很多角度看，这大概是个时机问题。市场现在处在一个很特殊的位置：最大、最成功的那几家实验室，不知是巧合还是命运，正处在史上最激烈的市场厮杀里，这让它们不太有兴趣去试别的路径、试别的方案。如果 transformer 是赚钱的，如果你多投一分力气、多投一分资源去扩 transformer 就能赢下一个季度，那你就很难把大量的注意力和精力放到一件也许一两年后才更好、才可能重新定义这个领域的事情上。所以我觉得最大的那几家实验室——我基本上跟它们全都聊过——对尝试 transformer 的替代方案兴趣不大。而不是最大的那些实验室，则在拼尽全力做最成功那几家在做的事。所有人都在试同一个 coding agent。你看上周的发布，所有人都在发 coding agent。我认为这里需要不同的路径、不同的做法。这就是我们想去填的那个生态位。
[14:24] 主持人
Mhm. And you were you were at you were at Brain when the Transformer was invented. Do you Do you agree with Jerry's eulogy for the Transformer?
嗯。你当时正好在 Brain，transformer 就是在那会儿被发明出来的。你同意 Jerry 给 transformer 写的这份悼词吗？
[14:30] Rohan Anil
Yes. Um in some sense like um once the first when the Transformers Ashish Noam and others came up with it I I had like work I worked on my work on online distillation around the same time. We presented it at the same internal research conference. It wasn't a big deal internally. There was only few people who actually got it.
同意。某种意义上说，当年 Ashish、Noam 他们几个人做出 transformer 的时候，我差不多同一时间在做我自己那份 online distillation（在线蒸馏）的工作。我们是在同一场内部研究会议上做的报告。那件事在内部当时并不算什么大新闻，真正看懂它的只有很少几个人。
[14:49] 主持人
Mhm.
嗯。
[14:49] Rohan Anil
A lot of few people were like, oh, it's it's like, yeah, it's another work. And people were finding ways to And it was also very focused on at least the original work was very focused on a real problem, which was translation. So they solved like they beat LSTM on translation. And it took opening up I mean, internally at Google there was definitely like Noam and a few others were definitely interested in scaling language models. I think it is until GPT-2 and GPT-3 that we saw the benefit of Transformers working quite well. At least the way I think about architecture is how do we spend computation and transformer is one way very efficient way to spend computation. But now that I look at the industry, it's it's a lot of our computation is inference time and spending it on tokens. Let me ask the question like if I want to optimize for a better architecture, I want to look at both pre-training and RL together and I would like to find architectures that spend computation much better than current chain of thought token generation. Uh in at a like to give a much better overview, it's I think of like pre-training is built the transformer with certain context length and RL comes in and it's like, well, that's not sufficient. I need more computation. Let me do it via adding one token at a time.
当时不少人的反应就是「哦，又一篇工作而已」，大家只是在想办法拿它做点什么。而且它——至少最早那篇工作——非常聚焦在一个真实问题上，就是机器翻译，他们在翻译任务上打赢了 LSTM。真正把它推起来的是 OpenAI——我是说，Google 内部当然也有人，Noam Shazeer 还有另外几位，明确对 scaling 语言模型很感兴趣。但我觉得一直要到 GPT-2、GPT-3，我们才真正看到 transformer 跑得多好。至少在我的理解里，架构问题本质是「我们怎么花掉算力」，transformer 就是花算力的一种方式，而且是很高效的一种。但现在回头看整个行业，我们大量算力是花在 inference 阶段、花在吐 token 上。那我就要问：如果我想优化出一个更好的架构，我应该把 pre-training 和 RL 放在一起看，我想找到那种把算力花得比现在这种 chain of thought 逐 token 生成好得多的架构。说得更直观一点：pre-training 相当于先按某个 context length 把 transformer 建好，然后 RL 进来说「这不够啊，我需要更多计算量」，那就一次加一个 token 地往上堆。
[16:03] Rohan Anil
This is quite inefficient from uh like inference perspective. You're doing one token at a time, so most of the solutions have been finding to do better ways of speculative decoding. So it's like a band-aid to a problem that we've picked something that's can only generate one token at a time. So autoregressive decoding. Uh there is problems with the transformer in terms of how do we spend the computation? Uh for all the longest time, I think uh most of the world was training very large dense models and it took uh like the industry like roughly two to three years to get to refine the architecture to what we now take for granted was not obvious to a lot of people. Sparsity and the mixtures of experts and getting good training efficiencies with them, right? So then you can ask like what's wrong with the transformer? Well, it's it the computational depth is poor. You how do we increase computational depth? And just posing that question opens up like 20 new directions on how we can modify the mechanism to incorporate it. So um I see like to do work like this, it takes time and usually like fundamental research in the past have taken like five, six years to land into industry and it's largely from organizational knowing that it is important. This is the bet. Like just like Jerry had the inner belief that RL is needed.
从 inference 的角度看这非常低效——一次只出一个 token，所以大部分解法都在往 speculative decoding 上使劲。这其实是在给「我们选了一个一次只能生成一个 token 的方案」这个问题打补丁，也就是 autoregressive decoding。transformer 在「算力怎么花」这件事上是有问题的。很长一段时间里，全世界都在训非常大的 dense 模型，整个行业大概花了两到三年，才把架构打磨成我们今天视为理所当然的样子——但当时对很多人来说一点都不显然：sparsity、mixture of experts，以及怎么把它们训得足够高效，对吧？那接着就可以问：transformer 到底哪里不好？它的 computational depth（计算深度）太差。那我们怎么把计算深度提上去？光是把这个问题抛出来，就一下子打开了二十来个可以改机制的方向。所以我的感受是，做这类工作是需要时间的，历史上基础研究通常要五六年才能落进工业界，而且很大程度上靠的是组织层面认定「这件事重要，这就是我们要押的注」——就像 Jerry 当年内心笃信 RL 是必需的一样。
[17:26] Rohan Anil
Absolutely do not have that belief at Google. I was a pre-training maximalist. Pre-train a bigger smaller
我在 Google 的时候是完全没有这个信念的。我是个彻头彻尾的 pre-training 最大化主义者——先预训练一个更大的模型，再蒸馏成小的……
[17:31]
are a good fit.
……这套组合是很搭的。
[17:32] Rohan Anil
Right? Like all right. And then so that inner belief and second is you need your architecture to run efficiently on hardware. A theoretically optimal architecture is not useful to anyone. It is something when it comes into practice. So you need the research inception to getting it productionized and getting kernels and everything written. The end-to-end loop. And there's only few places right now which have integrated teams doing that and I think we have built a team in a way that puts every like the experts together not in different silos that like we are accelerating on having everybody look at the problem holistically from end to end. So that's like where I'm quite bullish. That's why I'm here. Um the current mechanisms are quite poor and it will if if you leave it to the world, I am afraid that it'll take us like a much longer time horizon before we replace the transformer and I think a lot of folks are already complaining on a lot on token costs. And that seems like as someone
对吧？好。所以第一是这种内心的信念，第二是你的架构必须能在硬件上高效跑起来。一个理论上最优的架构对谁都没用，它得真的落到实践里才有意义。所以你需要打通从研究萌芽到产品化、到把 kernel 之类全都写出来的整条链路，端到端的闭环。现在全世界能把团队整合成这样的地方没几个。我觉得我们把团队搭成了这样一种形态：把各路专家放在一起、而不是分散在各自的筒仓里，让所有人都端到端、整体地看同一个问题，这一点我们是在加速的。这就是我特别看好的地方，也是我为什么在这儿。现在的机制其实相当差，如果就这么交给世界自己演化，我担心我们要等非常久才能把 transformer 换掉。而且很多人已经在抱怨 token 成本了，这看起来就像是有人……
[18:33]
complaining.
在抱怨。
[18:34] Rohan Anil
Uh I mean in terms of like yeah, exactly. I come from the Google mindset where we had to like serve billions of people. So like finding more efficient architectures that fit have a like a deadline on latency and the number of tokens that you can serve. So like when I look at that like the amount of like the world that can use like frontier tech is very little and we someone or some group has to like accelerate and make this better and we are taking that shot at doing that. The current technology just scaled up is still only relevant to like a subset of humans.
嗯，我是说……对，就是这样。我是从 Google 那套思维里出来的，我们得给几十亿人提供服务。所以要去找那种更高效的架构，既要满足延迟上的硬指标，也要扛得住你能服务的 token 量。从这个角度看，全世界真正用得上前沿技术的人其实非常少，总得有人、有某个团队去加速、把这件事做得更好，我们就是要打这一枪。现在的技术光是往上堆规模，覆盖到的仍然只是人类中的一小撮。
[19:09]
Yeah.
嗯。
[19:09] Rohan Anil
And this is the bet we're making.
这就是我们押的注。
[19:11] 主持人
So one of the things I heard you say was the problem with transformers is the computational depth is poor.
你刚才说的其中一句我印象很深：transformer 的问题在于 computational depth 太差。
[19:17] Rohan Anil
Yeah.
对。
[19:17] 主持人
If that's the crux of the issue, tell us what does that mean? Why is that the case? How do you fix it?
如果这是问题的症结，能不能给我们讲讲：它到底是什么意思？为什么会这样？又该怎么解决？
[19:23] Rohan Anil
I can give you like one insight like most transformers that we train are quite shallow. It's at most like 100 layers deep. Depth is like it's called deep learning cuz you want a deeper representations. There has been experiments on going with no one has actually shown us learning extremely deep representations. Chain of thought reasoning and RL to do chain of thought on by model itself is one way to increase computational depth because every token you add you add like one more pathway. And there has been So, then you can get out of like this bottleneck that the pre-trained architecture has set you up on. You can only do layer number of layers times sequence length. Now, you can increase the sequence length and you get much stronger results. You can do inference time scaling. Now, the issue with inference time scaling is that models now have to produce more tokens to get better results and that's very one token at a time. And from this you can see like you can directly address many of these things and this is like a subset of work that we're looking at, right? Making this much more efficient.
我给你一个直观的点：我们训的绝大多数 transformer 其实都很「浅」，最多也就 100 层左右。深度这件事——之所以叫 deep learning，就是因为你想要更深的表征。这方面有人做过实验，但还没有人真的展示出能学到极深的表征。chain of thought 推理、以及用 RL 让模型自己做 chain of thought，是提升 computational depth 的一种办法：你每多生成一个 token，就多出一条计算路径。这样你就能绕开 pre-train 好的架构给你设下的瓶颈——原本你的计算量上限就是层数乘以序列长度；现在你可以把序列长度拉长，结果就明显更强，也就是 inference time scaling。但 inference time scaling 的问题在于，模型必须吐出更多 token 才能拿到更好的结果，而且是一次一个 token 地吐。顺着这个思路你就能看出，其中很多问题是可以被直接解决的，这也正是我们在看的一部分工作——把这件事做得高效得多。
[20:32] 主持人
What's your forecast for the transformer-based architecture? If it's not the end state, how far can it get us? When do we start to see it topping out?
你对基于 transformer 的架构怎么预判？如果它不是终局，它还能带我们走多远？我们什么时候会开始看到它见顶？
[20:41] Jerry Tworek
I I I think it all it all comes back to what we are what we are training transformers for and what we can what we can do with them. We're doing pre-training which is very good at distilling all the knowledge from the internet into transformers and then we can RL them through which is we basically can bake all the workflows that we want into a transformer. So, what what what where transformer has like capped out is we have all the knowledge of of of humanity in the model together with the relationships and how do they how do they work together, how we how they can be combined. And basically any task that we have training data for, we can we can put into the model and we this can be gigantic model trained with a lot of compute on all the on all the all the data in the world. And then if we ever stop training that model, what would what would happen? Like the the question worth asking often and thinking about transformer, what would happen if OpenAI and Anthropic stopped training new models and we got we got the transformer we have today and say this is this is it. This is this is this is the best the best model we have. Uh months pass, uh year year year years pass and the model is getting less and less useful.
我觉得这一切都要回到：我们到底在拿 transformer 训什么、又能用它做什么。我们做 pre-training，它非常擅长把互联网上的全部知识蒸馏进 transformer；然后我们再对它做 RL，本质上就是把我们想要的各种工作流烤进 transformer 里。所以 transformer 的封顶之处在于：我们已经把人类的全部知识、连同知识之间的关系、它们怎么协同、怎么组合，都装进了模型；基本上任何我们有训练数据的任务，都能塞进模型里；而且这可以是个巨大的模型，用海量算力在全世界所有数据上训出来。那接下来——如果我们哪天停止训练这个模型，会发生什么？这是个很值得常问、也很值得围绕 transformer 去想的问题：如果 OpenAI 和 Anthropic 都不再训新模型了，我们就守着今天这个 transformer，说「就它了，这是我们最好的模型」。几个月过去，一年、几年过去，这个模型会变得越来越没用。
[21:50] Jerry Tworek
Maybe the lab really like recorded of every human on earth what they were doing and what their what their tasks were and their environments and put them put them in the model, put them in the in the in the learning environment. But then that that then then then what happens if the if anything of that stuff changes? If there are new events in the world, if those new events have new relationships between them, if there are there are new types of tasks, if there are new code bases, new tools to use, transformers are getting a lot of their usefulness and all value through the things that are that are valuable have to be present in in in in in in training. And when when when when when they are not, they they they they suffer. There's there's some ability to adapt, but it's not very not very big and not very not very flexible. So in my mind this is this is kind of the level where the where the transformers uh top, which in many ways what I think is a is a tool to use for us. If we if if we kind of know if there ever there's human who knows the limitations of a transformer, they can they can schedule that model, they can they can write a prompt of what is what is it the task that you want. And by doing the training we are doing, you can you can like get very successful in that and any task the model fails, you get added to training data and the model and the model can can succeed. But that that loop is has to go for the lab training training the model for you. And if the model that that fundamentally needs to be trained in the lab, like how much do you think of it that this is this is the goal or or or you would want to be able to update the model somehow not having to to go back there.
也许某个实验室真的把地球上每个人在做什么、他们的任务是什么、他们的环境是什么全都录了下来，塞进模型、塞进学习环境里。但那之后呢——只要这些东西里有任何变化会怎样？世界上出现了新事件，新事件之间又有了新的关系，出现了新类型的任务、新的代码库、新的工具要用。transformer 的绝大部分用处和价值来自一件事：那些有价值的东西必须出现在训练里。一旦没出现，它就抓瞎。它是有一点适应能力，但不大、也不灵活。所以在我看来，这差不多就是 transformer 的天花板所在。从很多角度说，我认为它是一个供我们使用的工具：只要有一个了解 transformer 局限的人，他就可以去调度这个模型、写一个 prompt 讲清楚任务是什么；而靠我们现在做的这种训练，你可以在那个任务上做得非常成功，模型失败的任何任务都会被加进训练数据，然后模型就能做对了。但这个循环必须由训练模型的那个实验室来跑。如果模型从根本上就只能在实验室里训——你们觉得这就是目标本身吗？还是说你会希望能有办法直接更新模型，而不必再回到实验室去？
[23:22] 提问者
Have you read the Rich Sutton and David Silver have this paper the the age of experience? Have you read it? I'm curious how much you agree or if you have any different opinions where your opinions diverge.
你读过 Rich Sutton 和 David Silver 那篇论文《The Era of Experience》（经验的时代）吗？我很好奇你有多认同它，或者你在哪些地方看法不同。
[23:32] Jerry Tworek
Reinforcement learning is not a particularly new approach particularly particularly new thing to do. So, uh and some some way age of experience I think I think always has been there and people have been criticizing a bit pre-training because pre-training work clearly is this is the other way of of looking at the models which is like we have static data that data is mostly generated by others. Although I I have this personal view that pre-training today is largely distilling other models other models into the new model because most of the tokens in the internet are are coming from from AI. But there there's clearly pre-training which is which is behavioral cloning which is mimicry which is which is compression of of of of of internet data. But reinforcement learning is not something that that people haven't been thinking and people haven't been haven't been doing. Reinforcement learning was used to solve backgammon back in the day. They used to solve go, Starcraft, Dota and to solving programming right now and every time it comes down to model writing its own experience and learning learning from that experience. But this is very clear and what I think is interesting and what I think is still is still perplexing to people that reinforcement learning is not really the only way to learn from experience and there there will be more and there will be there will be a little bit more of I think you can call it algorithmic, but essentially innovation of how we learn from experience. Just because just because reinforcement learning is only one way to do it. It's a mathematical formulation. And especially right now how we are using it. It really likes those parallel rollouts for variance reduction and for for comparing how the model does in in in parallel versions of the world, which is not how not how we do not how we learn from experience. We learn from our experience much more efficiently and much more
reinforcement learning 并不是什么特别新的路子。某种意义上，「经验的时代」一直都在，而且一直有人在批评 pre-training，因为 pre-training 显然代表了看待模型的另一条路：我们有静态数据，而且这些数据大多是别人生成的。虽然我个人的看法是，今天的 pre-training 在很大程度上其实是把别的模型蒸馏进新模型，因为互联网上大部分 token 现在都来自 AI。但 pre-training 确实就是 behavioral cloning、是模仿、是对互联网数据的压缩。而 reinforcement learning 也不是大家没想过、没做过的东西：当年 RL 被用来解 backgammon（西洋双陆棋），后来解围棋、星际争霸、Dota，现在在解编程。每一次都归结为：模型写出自己的经验，再从这些经验里学习。这一点非常清楚。我觉得有意思、而且很多人到现在还没转过弯的一点是：reinforcement learning 其实并不是从经验中学习的唯一方式。以后还会有更多——你可以称之为算法层面的创新——关于「我们怎么从经验中学习」的创新。因为 reinforcement learning 只是其中一种做法，它是一种数学表述。尤其看我们现在用它的方式：它非常依赖并行 rollout 来做方差缩减、来比较模型在「平行世界」里各自表现如何，而这根本不是我们人类从经验中学习的方式。我们从自己的经验里学习要高效得多，也……
[25:27] Jerry Tworek
Um we use we use those in many in many many ways. At some moment I've been trying to explain to people that what what brain does how how we learn. So uh there's one learning algorithm in a brain. I think I think there are there are multiple actually and they and they they work together. But if I play football for example, it looks very very closely to reinforcement learning. I kick ball a lot of times and every time I adjusted a little bit and I see if it roughly matches what I what I wanted and I learn or or some self-reinforcement had money. When I when I when I learn mathematics, it's very different type of thing. It's it's like reading about hard concepts and thinking about them very deeply inside my head until until things click and I until until I have them connected. And both of those in some way are learning from experience. They are just very different. So so summarizing my thinking of the learning from experience is that we've been doing it for a while. We probably are spending the most compute than ever on learning from experience, but the reinforcement learning is is not the end of learning from experience and there will be there will there will be better approaches that will researchers will be coming up in the coming years on how to how to use that data in in a richer new chart settings.
嗯，我们用它们的方式有很多种。我有阵子一直试着跟人解释：大脑到底是怎么学习的？大脑里有一套学习算法——其实我觉得是好几套，而且它们协同工作。比如我踢足球，那个过程就非常像 reinforcement learning：我把球踢出去很多次，每一次都微调一点，看结果跟我想要的差多少，然后我就学会了，或者说产生了某种自我强化。但当我学数学的时候，那完全是另一回事：那是去读那些很难的概念，然后在脑子里往深里想，直到某个瞬间「咔」一下想通，直到我把它们连起来。这两种在某种意义上都是从经验中学习，只是差别非常大。所以总结一下我对「从经验中学习」的想法：我们已经做了挺久了，而且我们现在花在「从经验中学习」上的算力大概是史上最多的，但 reinforcement learning 绝不是从经验中学习的终点，未来几年研究者们一定会拿出更好的方法，在更丰富的新场景里去用这些数据。
[26:39]
Thank you. Same.
谢谢。同上。
[26:41] 提问者
I'm Rohan. I'm curious since a lot of your work has been around optimization and and efficiency, how do we get to a orders of magnitude more efficient, I guess more compute efficient and more data efficient learning algorithm.
我叫 Rohan。我很好奇，既然你的很多工作都围绕 optimization 和效率：我们要怎么才能做出计算效率和数据效率都高出几个数量级的学习算法？
[26:55] Rohan Anil
You know, I would start with measurement. I think pre-training as we define it right now is about compression. We look at perplexity and then measure how do we decrease the perplexity and then we find that scaling and increasing parameter count and putting more compute is the way and every time we increase compute in log scale, we get this epsilon more improvement in these metrics. This is I think this is fine for building the prior. But I think this is the wrong way to look at the problem. We should be looking at the end-to-end. What are we training these models for? Look at the outcome. Like for example, I train this model and give it to Jerry. Jerry will do RL and destroy all the perplexity metrics that I have created, right? Like so then it's sort of like it was the best way we had so far to attempt to solve the problem and I think the labs and everyone else have done a great job in producing intelligence that's super valuable and makes my work so much fun. But it was the bootstrap process to get there. We have to combine pre-training and RL together and that's like where one order of magnitude improvement would come from and that's like a training procedure. You can say it's a learning algorithm. Um in terms of optimization, my story is I started optimization at Google for logistic regression back in 2016. Got nerds like me and I worked on some solvers for what we used to call Sybil, which was the large-scale linear solver that was used at Google before neural network took off and then replaced this. Then I asked myself like what do I want to work on with neural network and it was quite clear like I want to understand the training algorithm and make it better. And then someone Vinit Gupta just showed up one day at my desk. It's like I heard you're really good at writing optimization methods. We have this idea that you know, uh like we worked out on a whiteboard uh and what turned out to be the Shampoo
我会从「怎么度量」开始。我们现在定义的 pre-training 本质上是压缩：我们看 perplexity，然后想办法把 perplexity 降下去，接着发现 scaling、把参数量堆大、把算力堆上去就是办法——每次算力按对数尺度往上加，这些指标就再改进一个 epsilon。我觉得作为建立先验，这没问题，但这么看问题是错的。我们应该端到端地看：我们训这些模型到底是为了什么？要看最终结果。举个例子，我训好一个模型交给 Jerry，Jerry 拿去做 RL，会把我做出来的那些 perplexity 指标全毁掉，对吧？所以某种意义上，这只是我们迄今为止手上最好的一种尝试解法。我觉得各家实验室和所有人都干得非常出色，造出了超级有价值的智能，也让我的工作变得特别有意思——但那只是把我们带到今天的 bootstrap 过程。我们必须把 pre-training 和 RL 结合到一起，那一个数量级的提升会从这里来，它是一种训练流程，你也可以说它是一种学习算法。说到 optimization，我的故事是这样的：我 2016 年在 Google 从 logistic regression 的优化做起，都是我这种书呆子在干的活。我做的是一些求解器，用在我们当时叫 Sibyl 的系统上——那是神经网络起飞之前 Google 用的大规模线性求解器，后来被神经网络取代了。然后我问自己：转到神经网络上我想做什么？答案很清楚——我想搞懂训练算法，并且把它做得更好。后来有一天，Vineet Gupta 直接跑到我工位上说：听说你写优化方法很在行，我们有个想法……我们在白板上推了一通，后来那就成了 Shampoo——
[28:52] Rohan Anil
algorithm. Can you help us implement scale, make sure it works at large scale for neural network training? So, as working on this, I I thank my manager, Yanghui Wu, who has like supported it throughout then to like till my end of my tenure 2024. But largely like the community and most of the other people were not as excited by this idea. And for me, this is the most exciting thing cuz I was like I'm putting in computation and making training better. This is the thing I have to figure out. I will spend as much time I would would take to do it. And then people were um making this assumption, "Oh, why like what's the upper bound? You could still use Adam. That's fine. Like why are we You could spend all the time on everything else, not optimization." But in some sense, optimization is like like you have a model, you're optimizing it, you want to optimize it better. Now, I'll connect it to some of the stuff that we talked about architecture. What has happened is that a lot of the work that we've done in architecture is to make these networks train. And in some sense, it's like two sides of the coin and optimization and architecture go together. You could have a stronger optimizer train a much more harder harder to optimize model and get better performance. Or you can use a weaker optimizer on easier to optimize models and get decent performance. So, there are these tradeoffs that appear all over. And for me, it's a lot of time working on it. I think we used it for Gemini 1.5 Flash. And then like the community started getting like more interested in it. There was the Soap paper published. We have like an entire literature of like shampoo, soap, all like bath time the things that you would use. And then it was quite clear like so that was like maybe a 2x improvement over what was happening. But even then, if you look at Shampoo, it's quite weak in what it's doing. It's not using all the information that's available to you
——那个算法。他问：你能帮我们把它实现出来、把规模做上去，确保它在大规模神经网络训练里真的管用吗？我就这么开始做了。我要谢谢我的老板 Yonghui Wu，他一路支持这件事，一直支持到我 2024 年离职。但总体上，社区里和大多数人对这个想法并没那么兴奋。而对我来说这是最激动人心的事，因为我等于是在投入算力、把训练变得更好，这就是我必须搞明白的事，要花多少时间我都愿意。当时很多人的默认假设是：「这上限能有多高？你用 Adam 不也挺好的吗？干嘛折腾这个？你可以把时间花在其他所有事情上，就是别花在 optimization 上。」但某种意义上，优化就是——你有一个模型，你在优化它，你当然想优化得更好。这里我把它跟刚才聊的架构连起来：过去我们在架构上做的很多工作，其实都是为了让这些网络「训得动」。某种意义上这是一枚硬币的两面，optimization 和架构是绑在一起的：你可以用更强的 optimizer 去训一个更难优化的模型，拿到更好的效果；也可以用较弱的 optimizer 去训容易优化的模型，拿到还不错的效果。这类权衡到处都是。我在这上面花了很多时间，我记得 Gemini 1.5 Flash 我们就用上了。之后社区才开始对它更感兴趣，SOAP 那篇论文也发了出来。现在我们有一整套文献：Shampoo（洗发水）、SOAP（肥皂），全是洗澡时会用的东西。当时已经很清楚了，那大概是相对此前做法 2 倍的提升。但即便如此，你看 Shampoo，它做的事情其实还是很弱的——它并没有用上你手里所有可用的信息。
[30:43] Rohan Anil
when you train. And as you use more and more information as part of training, you can get better improvement. And in some sense, like your optimization algorithm defines what architectures you will discover. Like I have colleagues, it's not very popular in the literature. It's only like maybe sub like four people in the world care about it. Kind of ideas that are extremely interesting like residual connections have been extremely useful for training neural networks. There are folks who have like gotten rid of them and learned deeper representations. But they needed a better optimization method. So like for me, optimization methods and the question you asked is just like how do we get there? It's It's combined with architecture and thinking about the problem end to end is where a lot of the computational efficiency is in it.
……训练的时候。你在训练里用上的信息越多，能拿到的提升就越大。某种意义上，你的优化算法决定了你会发现什么样的架构。我有些同事——这方向在文献里不算热门，全世界可能只有不到四个人真的关心——他们的一些想法特别有意思。比如 residual connections 对训练神经网络一直非常有用，但有人把它整个去掉，照样学出了更深的表征，只不过他们需要一个更好的优化方法。所以对我来说，优化方法和你刚才问的那个问题其实是同一件事：我们要怎么走到那一步？它和架构是绑在一起的，把问题端到端地想清楚，计算效率的空间大部分就藏在这里。
[31:29]
Yeah.
嗯。
[31:30] Rohan Anil
And I also see RL as spending a lot of compute not as efficiently. And so like if you could spend it because you don't get much feedback and you're spending a lot more compute because you have to decode all this long chain of thought to get this one bit of information into the network. Seems quite inefficient and an easy target to get orders of magnitude on top of. I can go on talking about optimization all day, but I
而且我还觉得 RL 花算力花得很不划算。你拿到的反馈很少，却要解码出一整条很长的 chain of thought，才能把这么一点点信息喂回网络里，所以要烧掉多得多的算力。这看上去效率很低，也是个很容易拿下数量级提升的靶子。优化这个话题我能聊一整天，不过我……
[31:54]
it.
……这个。
[31:55]
Rohan really has
Rohan 是真的特别……
[31:56] 主持人
Do you think we'll ever approach or surpass biological learning efficiency?
你觉得我们有一天能接近、甚至超过生物学习的效率吗？
[32:01] Rohan Anil
I do not think so because I think we would need to change I maybe like that was a strong statement. At least with the hardware we have, it seems pretty unlikely. Um it our biological like we have something as Jeff Hinton says, model computation. So we build our own circuit as we grow up and we build our own learning algorithm with the hardware. And then we die and then we're gone. Uh neural networks are quite different. Our hard The hardware stays, the neural network stays, but it's learning very inefficiently and you need a lot more of them and a lot of parallelism to get small amounts of information through. So until I think we design hardware to be much more like how humans operate, maybe more analog, figure out how to deal with analog circuits, and to figure out how to do with error correction, figure out how to get information through it will be much harder. Uh we I think we're safe.
我觉得不会，因为那需要我们改变……这话说得可能有点绝对。至少以我们现在的硬件来看，这不太可能。人这套生物系统，用 Geoff Hinton 的说法叫 mortal computation（可朽计算）：我们在长大的过程中长出自己的电路，也连带长出自己的学习算法，它和硬件是长在一起的；然后我们死了，这一切就都没了。神经网络很不一样：硬件留在那儿，网络也留在那儿，但它学得非常低效，你需要很多份副本、很大的并行度，才能把很少一点信息灌进去。所以除非我们把硬件设计得更接近人的运作方式——也许更偏模拟电路，搞清楚怎么处理模拟电路、怎么做纠错、怎么把信息传进去——否则会难得多。所以我觉得，我们暂时还算安全。
[32:55] 主持人
Safe. That's an interesting way to put it. Um the idea that pre-training and RL should be optimized end-to-end seems like such a, you know, clear maybe obvious statement. Do do you think Do you think the labs realize this? And then is it just hard for them to, you know, get rid of org charts and process to be able to make that come together? Or what stops the labs from from being able to, you know, unify the the two?
安全，这个说法挺有意思。嗯……pre-training 和 RL 应该端到端一起优化，这个论点听上去很清楚，甚至有点显而易见。你们觉得那些大 lab 意识到这一点了吗？还是说他们只是很难摆脱组织架构和流程，没法把这两件事真正捏到一起？到底是什么拦住了这些 lab 去统一这两者？
[33:20] Rohan Anil
Uh I don't think it's that obvious cuz uh it's a completely, again, different optimization problem. You have a prior, you're doing rollouts, you have higher variance, you're and then pre-training is much large batch, like more parallelism or compute for the unit of time that you can spend. So, it is not an obvious thing for folks to combine these two training procedures until uh you think a bit more like, "Why is it that the naive combination doesn't work?" So, that's one. Uh the second one is if I pull like some of the best researchers in these labs, they would say, "Oh, this makes sense. We should probably explore it, but it would be probably not in the top bucket because they have to train a model for the next cycle before It's like as as Jerry said, like there are companies now competing for release cycles because tokens are not sticky. So, it's much harder to do long sight, like even long-term research of 6 months in many of these labs in the environment they are in.
我倒不觉得这有那么显而易见，因为它们本质上是完全不同的优化问题。RL 这边你有一个 prior，你在做 rollout，方差更高；而 pre-training 是大 batch 的，单位时间里能吃下的并行度和算力都大得多。所以对大多数人来说，把这两套训练流程合起来并不顺理成章——除非你再往下多想一层：为什么最朴素的那种合并方式行不通？这是第一点。第二点，如果我去问一圈这些 lab 里最顶尖的研究员，他们大概会说：有道理，我们应该探索一下，但多半进不了最优先的那一档，因为他们得先把下一轮的模型训出来。就像 Jerry 说的，现在这些公司是在抢发布周期，因为 token 是没有黏性的。所以在他们所处的环境里，要做长线的事——哪怕只是六个月周期的长期研究——都难得多。
[34:24] 主持人
So, it seems like one of the core premises for Coral Lemonade is, you know, you're starting you're starting a lab at a time when you know, Sam's been talking about the AI scientists. I think Dario's been talking about the AI scientist. It seems like your job as researchers has actually fundamentally changed. Um and you're get you get to start the company native to that era. And as a result, you're maybe able to run a lot more experiments than otherwise might be possible. How automatable do you think the research job is? Um and how are you guys approaching building your lab to be as I believe your mission one of your missions is to be the most autonomous lab there is?
所以 Core Automation 的一个核心前提好像是：你们创办这家 lab 的时间点，正好是 Sam 在讲 AI scientist、Dario 也在讲 AI scientist 的时候。研究员这份工作本身好像已经发生了根本性的变化。而你们是在这个时代里从零开始建公司，所以能跑的实验数量可能比过去多得多。你们觉得研究这份工作能被自动化到什么程度？又打算怎么把这家 lab 建成——我记得你们的使命之一，就是要做最自主的 lab？
[34:59] Jerry Tworek
Most most automated lab in the world and to to start with I think that automation the version of automation by core automation is about giving each human maximum level of agency in some way. It is we we're not trying to really get humans out of the loop which is like one version to automate, but it is about give humans ability to do the most with with their their amount of time. Whenever whenever you are walking you can get some distance. Whenever you get a bike you can go go work a larger distance. Whenever you are a car you can you can go much much much much larger. When whenever whenever humans started farming they had to farm by hand and will work on a small plot of land. When you have a machine you work on a much much larger plot of land. Uh personally I am I'm both really great fan of the of the current coding agents and and and very happy it's in some way it is what I've been working for many years both doing coding research and working on various versions of of AI scientist inside inside of OpenAI and uh in the end I realized starting a company to realize that vision is is is is one of the best one of the best ways to realize it because the way you can do research today is very very different because a single researcher can do much more. In the end the speed of iteration the speed of research the speed of how quickly you can you can move through ideas and how quickly you can get data on your ideas is something something very very different and you can try to move the old structures around it and the teams workflows how data is gathered or you can try to build like like like you said you can you can try to build build natively for it for processes that that maximally empower each researcher and allow them to just to just iterate on their on their idea much much quicker. We are we are here and we are trying to to to rebuild deep learning stack and try to think how we can do almost each
是世界上最自动化的 lab。首先我想说，Core Automation 所理解的自动化，是在某种意义上给每个人最大程度的 agency。我们并不是真想把人从回路里拿掉——那是自动化的另一个版本——而是让人在同样的时间里能做成最多的事。走路能走一段距离；有了自行车能走远一点；有了车就能走远得多得多。人类刚开始务农的时候只能用手种，只能种一小块地；有了机器就能种大得多的地。我个人既是现在这些 coding agent 的忠实粉丝，也挺高兴——某种程度上这就是我这么多年一直在做的事：在 OpenAI 做代码方向的研究，也做过各种版本的 AI scientist。最后我意识到，要把那个愿景真正落地，创办一家公司是最好的路径之一。因为今天做研究的方式已经非常不一样了：单个研究员能做的事多得多。归根到底，迭代的速度、研究的速度、你穿过一个个想法的速度、你为自己的想法拿到数据的速度，都完全不同了。你可以试着在旧结构上修修补补——团队、工作流、数据怎么收集；也可以像你说的那样，为这件事原生地去建一套流程，最大化地赋能每一个研究员，让他们能以快得多的节奏迭代自己的想法。我们现在就在做这件事：重建整个深度学习栈，去想其中几乎每一个环节
[37:01] Jerry Tworek
operation differently. What are what are various options and if we can execute at least even one of those experiments a day, that's already a pretty good iteration speed versus anything that was that was done before and there isn't really like any any fundamental like laws of physics reason why not and maybe maybe one day we get to to 10 of those a day. Maybe one day we get we get 200 those a day and fundamentally for that like search process optimization process, we should be able to just just find things that work in a better deep learning setting and what what we are what we are trying to do is like we've been like almost all of us we are we are we are a team that is very agent built and automation built. We are we are we are we are we are trying to do an experiment like how far how far we can we can push those thing and how far how how much an organization that tries to do as much as we can with a small team, how far how far we can get with that.
能不能换个做法。都有哪些不同的选项？如果我们哪怕一天只跑完其中一个这样的实验，相比过去的任何做法，这个迭代速度就已经相当不错了；而且从物理定律的角度讲，并没有什么根本原因说不行。也许有一天我们能一天跑十个，也许有一天能一天跑两百个。本质上，对这种搜索、优化的过程来说，我们应该能在更好的深度学习设置里找到真正有效的东西。我们想做的是：我们几乎所有人，都是围绕 agent 和自动化来搭建的团队。我们在做一个实验——这类东西到底能推到多远，一个尽量用小团队做尽量多事情的组织，到底能走多远。
[37:53] 主持人
When will we know that we've reached AGI?
我们怎么才能知道自己已经到 AGI 了？
[37:56] Jerry Tworek
At some moment I used to say it's it's very much in everyone's heart, whatever whatever they consider AGI. Open AI says the system that can outperform all humans in economically valuable work, but it goes to my my my my previous statement. What if what if Open AI stops training models? Would that would that would that would that still keep working and I still would would that still keep the automation level the same the same or would that would that drive? And for me AGI is a model that can improve itself without human in the loop in any in any in any way. That's That's I think I think the moment where we can meaningfully talk about about AGI because that is in some way it is a sub definition of the previous one because improving air models is actually a job that humans can do. It is economically valuable work. And it definitely definitely is the case, but removing humans from from from loops with models has been actually notoriously notoriously difficult so far. We haven't come anywhere close to it. It's very very hard for me to find any task where all of them were able to like get get humans out in the loop. We are We are the the the the human LLM hybrid is is really really successful right now, but LLMs without humans not so much, not at all. And what I have seen there in 2024 and in early 2025 is that the current like path doesn't doesn't get us there.
我以前常说，AGI 到底算什么，很大程度上取决于每个人心里怎么想。OpenAI 的定义是：在有经济价值的工作上能超越所有人类的系统。但这就回到我前面那句话：如果 OpenAI 停止训练模型呢？那套东西还能继续跑下去吗？自动化水平还能保持不变，还是会往下掉？对我来说，AGI 是一个能在完全没有人类参与的情况下自我改进的模型。我觉得那才是我们可以有意义地谈论 AGI 的时刻。某种意义上，这是前一个定义的子集——因为改进 AI 模型本来就是人类在做的工作，而且确实是有经济价值的工作。但事实是，把人从和模型的协作回路里拿掉，到目前为止出了名地难，我们离那一步还差得很远。我很难找出哪怕一个任务，是模型能真正把人从回路里挤出去的。现在真正非常成功的是「人 + LLM」的混合体，而没有人的 LLM 就不行，完全不行。我在 2024 年到 2025 年初看到的是：现在这条路走不到那儿。
[39:17] Jerry Tworek
And I think we need we need some pretty pretty serious research on my this company or some other to to try to unlock how do we how do we make our models learn and and adapt at a test on a deeper level than we've been so far.
所以我觉得我们需要一些相当扎实的研究——在我这家公司也好、在别处也好——去解锁一件事：怎么让模型在 test time 以比现在深得多的方式去学习和适应。
[39:32]
Mhm.
嗯。
[39:32] 主持人
I feel like we've alluded to this throughout the conversation, but it's kind of been one of these, you know, five blinded blindfolded people trying to find the elephant things. Uh, what is the grand master plan for for Core AI that you're willing to share?
感觉整场对话我们一直在旁敲侧击这件事，但有点像盲人摸象——五个人蒙着眼各摸各的。那么，Core Automation 的宏伟总规划是什么？当然，是在你们愿意分享的范围内。
[39:45] Rohan Anil
I can share our 6-month road map. In some sense, like building architectures as I said, like wouldn't it's not just how good the architectures does it run well and can we get users including ourselves as part of the lab to use it? Right? So, then that's directly like we can do many things now, but the thing that is going to be difficult and thing that we want to automate away is kernel generation. So, we would we have a set of hardware GPUs, black holes that we have to train and run inference on. Uh, we will build the best model that we can to basically reduce the time from having a very cool idea that can make these bottlenecks go away in the architecture to having them run at the highest TFLOPS on GPUs. And in some sense like current coding agents plus humans can go a long way, but like an example of this is our urgent QR kernel competition that we hosted with GPU mode. It's for running this fairly old linear algebra operation QR. Um, it's used for optimization like shampoo line of work uses it. Many other places uses it. And you want to run this efficiently on a B200 node. Uh, if you use cool solver for the shapes that we care about, you get some efficiency. And then a human plus some search loop can get you like something like 7x.
我可以分享我们六个月的路线图。某种意义上，做架构这件事就像我刚说的，不只看架构本身有多好，还要看它跑起来快不快、我们能不能让用户——包括 lab 里的我们自己——真的用上。对吧？所以很多事我们现在已经能做了，但真正难、也是我们最想自动化掉的，是 kernel 生成。我们手上有一批硬件、GPU、Blackwell，要在上面做训练和推理。我们会尽力做出最好的模型，把「有一个很酷的点子、能干掉架构里的某个瓶颈」到「让它在 GPU 上跑出最高 TFLOPS」之间的时间压到最短。某种程度上，今天的 coding agent 加上人已经能走很远。举个例子：我们和 GPU MODE 一起办的那场 QR kernel 竞赛，跑的是 QR 这个相当老的线性代数运算。它在优化里会用到，比如 Shampoo 这条线的工作就要用，其他很多地方也用。你要在 B200 节点上把它跑得足够高效。如果你直接拿 cuSOLVER 去跑我们关心的那些 shape，能拿到一定的效率；而一个人加上一个搜索循环，大概能做到 7 倍。
[41:10] Rohan Anil
But it requires the highest taste human like there's maybe three people on the world to and spend about $100,000 on these coding agents over span of 4 weeks to get to a solution that's 60x faster. So these models today are no way close to getting that 60x faster kernel. And there's there is a real bottleneck now. That was a single problem. It has like perhaps three different operators. Work on this panel, do this matrix multiply, fold it back in, and do this repeatedly. That's what the square factorization of a matrix would look like. And if you give this problem to entropic models of an AI model Gemini, it just wouldn't solve it. It just is not our models are not even close to solving this problem. So for us it's like something that we've talked about, something that we're getting close to as sort of like getting to that point because that's our inner loop to having more efficient architectures.
但那需要品味最顶尖的人——全世界大概只有三个——再在这些 coding agent 上砸进大约十万美元、跑四周，才能拿到一个快 60 倍的方案。所以今天这些模型离写出那种快 60 倍的 kernel 还差得远，这是一个实打实的瓶颈。而这还只是一个问题，里面大概也就三个不同的算子：处理这个 panel，做这次矩阵乘，再折回去，然后反复这么做——矩阵的 QR 分解大致就长这样。你把这个问题丢给 Anthropic 的模型、丢给 Gemini，它们就是解不出来，我们的模型离解开它还差得很远。所以对我们来说，这是我们一直在聊、也快要摸到的一个点，因为它就是我们做出更高效架构的内循环。
[42:06] 主持人
Why kernels? Is it just cuz like maximize intelligence per flop of compute you need to generate your own
为什么是 kernel？是不是因为要把每一 FLOP 算力换来的智能最大化，就必须自己写？
[42:12] Rohan Anil
In some sense, um I've had like three projects. Two of them have kind of landed in the industry. So, first is secondary methods. Kernels was a bottleneck cuz you have to run it well. If you were at a place like Google, you cannot spend 10x uh amount of compute and get a 2x win. So, I could only spend maybe a budget of 20% and get the 2x win. Everyone's happy. I don't like great. So, like I think that's the market, right? You spend uh less than
某种意义上……我大概做过三个项目，其中两个后来在业界落地了。第一个是 second-order methods（二阶方法）。kernel 在那儿就是瓶颈，因为你必须让它跑得足够好。如果你在 Google 这样的地方，你不可能多花 10 倍算力只换来 2 倍收益；我最多只能花 20% 的预算，换来 2 倍收益，那大家都开心。所以我觉得这就是那笔账：你花出去的要少于
[42:41]
Yeah.
嗯。
[42:41] Rohan Anil
you uh get. So, kernels ended up being a bottleneck there cuz most of the operations were novel that we haven't gotten a lot of people to look at it. There's only like two humans at Google who could write it, Rasmus and Peter Hawkins. Cuz it was deep XLA LLO code that you have to write to make this work and then that took them 2 years to do. The other idea I had with one of my coworkers at that time was like replacing some of the parameters in a transformer with extra memory and we called it N-gram and N-gram memory. We worked on it in 2020. It's a we had versions of it internally deployed, not the big version, the smaller version. But, there I needed like something that can accelerate sparse uh gathers and scatters as part of training.
你拿回来的。所以 kernel 在那儿成了瓶颈，因为大部分运算都是新的，没多少人研究过。在 Google 能写这种东西的大概只有两个人——Rasmus 和 Peter Hawkins——因为你得写很底层的 XLA、LLO 代码才能让它跑起来，而他们花了两年才做出来。我当时和一位同事还有另一个想法：把 transformer 里的一部分参数换成额外的记忆，我们管它叫 n-gram memory，2020 年做的。我们内部部署过它的一些版本，不是大的那版，是小版本。但在那个方向上，我需要能加速训练中稀疏 gather 和 scatter 的东西。
[43:29]
Yeah.
对。
[43:30] Rohan Anil
Uh it required hardware change and a hardware making use of the hardware. It never arrived. I had uh conferences set up with the TPU team, us and a bunch of others. We were talking about it and during COVID like oh, we're going to have it happen and it never arrived. I also was using uh TPUs at Anthropic. While I was leaving, just barely started the surface of being able to um do it. But, at the same time, 6 months before that, Deep Seek wrote uh their N-gram, which is a improved version of adding more memory. Short scaling loss that yeah, you don't need MOEs. You could actually replace it with these
这事需要硬件层面的改动，还得有东西真正把这套硬件用起来。但它一直没来。我当时跟 TPU 团队开过好几轮会，我们这边还有另外一堆人，大家一直在聊这件事；COVID 期间还说「马上就要成了」，结果始终没兑现。后来我在 Anthropic 也用 TPU，等我要离开的时候，才刚刚摸到能做这件事的边。但差不多在那之前半年，DeepSeek 就发了他们的 n-gram 那套东西——本质上是把「加更多记忆」这条路做得更好了。从 scaling 上看，你其实不需要 MoE，完全可以换成这些——
[44:06]
Mhm.
嗯。
[44:06] Rohan Anil
N-gram embeddings. for me that was like ah
——n-gram embedding。对我来说，那一下就是：啊……
[44:09]
Yeah.
是啊。
[44:10] Rohan Anil
I It was like a five-year thing and I was very happy for them.
我……这件事我惦记了五年，所以看到他们做出来，我真心替他们高兴。
[44:15]
isn't even possible if you're not writing kernels.
……而这事，你要是不自己写 kernel，根本就不可能做成。
[44:18] Rohan Anil
Kernels and you need to be assisted in writing kernels or solve that kernel to have the highest performance. Like and the the roof line is pretty high. So it's like the QR. If I use CuSolver's QR, I get some performance. If I we use our the competition winners QR, you get 60x faster. And that is a completely different playing field. Now it opens up an entire new set of algorithms you can apply and in terms of training transformers, training optimizers. QR is so fundamental in analyzing the eigen decompose for eigen decomposition and many other things. So it is a thing that I think also if you think about it, only few people have the skill set to and they're very much not at the same place. It's like one person here, one person there. And it would be ideal if models had those abilities.
是 kernel，而且你还得有人帮你写 kernel，或者说得先把 kernel 这道题解掉，才能榨出最高的性能。这里的 roof line 其实相当高。就拿 QR 分解来说：我用 cuSOLVER 自带的 QR，能拿到某个性能；但换成比赛冠军写的那版 QR，快 60 倍。这完全是另一个量级的战场，一下就打开了一整片你可以用的新算法空间。而回到训练 transformer、训练 optimizer 上，QR 是极其基础的一环——做特征分解（eigendecomposition）以及很多别的东西都要靠它。所以你想想，全世界有这套技能的人本来就没几个，而且还特别分散，这儿一个、那儿一个。要是模型本身就具备这种能力，那就太理想了。
[45:09] Jerry Tworek
Maybe I'll I'll summarize a little bit and talk from the high level of what we want. Kernel automation is a lab created to build models that continuously learn and then learn from deployment. We believe as I mentioned that transformers are incapable of continual learning. There there's no way how to put continual learning on transformers. So we know we have to find a different different architecture. Some of anyway, our quest is to find that that new architecture, find that transformer replacement and we want to build the most automated lab to do it. We want to be able to build experiments at scale the quickest we can, iterate on them, try a lot of new architectural ideas, have strong priors of what we want to do to search the space of architectures efficiently to find to go to that go to that place fastest than than anyone else. That's what we what we want to do and all the work we are doing on kernels, on large scale training, on trying new architectural ideas is is exploring that space.
我可能稍微总结一下，从更高的层面讲讲我们到底想做什么。Core Automation 这个 lab 成立的目的，就是造出能持续学习、并且能从部署中学习的模型。像我前面说的，我们认为 transformer 天生做不了 continual learning——没有任何办法把 continual learning 装到 transformer 上。所以我们很清楚，必须找一个不一样的架构。总之我们要做的，就是找到那个新架构、找到 transformer 的替代品；而为了做成这件事，我们想建一个自动化程度最高的 lab。我们希望能以最快的速度大规模地搭实验、快速迭代，试大量新的架构想法，同时对「该往哪走」有很强的先验，从而高效地搜索架构空间，比任何人都更快地走到那个位置。这就是我们想做的事。我们现在在 kernel、在大规模训练、在尝试新架构想法上做的所有工作，都是在探索这片空间。
[46:10] 主持人
So you can experiment your way into finding a superior architecture. How will you know when you've found it? What are you looking for to say, "Aha, this is the one."
所以你们是打算靠不断做实验，把更好的架构试出来。那你们怎么知道自己已经找到了？你们在等一个什么样的信号，能让你们说「啊哈，就是它」？
[46:23] Jerry Tworek
That's that's a great question. There are there always two angles. In in in my mind and my experience, every successful research had a plot that shows something that's that other other plots don't show. There there is a one line that is a little bit bending in a different different way and you're saying this is this is what you want. But at least it is my experience with research always has been that plot is already quite late in a journey where most of the time you already know what you want and already know what you are up to. I am a bit a bit joking, but it's actually true that all the best plots in my life I have done in a dream before before actually they were they were real. I kind of knew I was looking for. Just just the question is is like when when it actually clicks if you if you know what I mean because most of the time you kind of know what you are looking for, but you are you are not finding it. You you try one thing and it doesn't work. Try second thing and it doesn't it doesn't work. But eventually there all the all the right pieces fall into it and most of that deep learning systems are very intricate. So usually you have to get five things right in a row for the for the thing to start working and then eventually you get the plot that looks like like like like you want and then and then you know. So So I think what we are what we are looking for is systems that learn and test time and if we see meaningful long-term adaptability of our of our systems and like we are we are we are joking, but it's it's it's a it's a real we want to be evaluating our systems of our of our everyday work. Will they get better at doing the work of core automation scientists each day?
这问题很好。在我看来永远有两个角度。以我的经验，每一项成功的研究，最后都会有那么一张图，图里出现了别的图上没有的东西——有一条线，它拐弯的方式不太一样，你一看就知道「这就是我要的」。但至少以我做研究的经历，那张图出现的时候其实已经相当晚了：大多数时候你早就知道自己想要什么，也早就知道自己在干什么。我半开玩笑，但这其实是真的——我这辈子最好的那些图，在真正跑出来之前，我都先在梦里见过。我大概知道自己在找什么，问题只是「什么时候真的对上」，你懂我意思吧。因为大多数时候你心里有数在找什么，但就是找不到：试一个，不行；再试一个，还是不行。可最终，所有该到位的零件都会落到位。深度学习系统非常精细，通常你得连着做对五件事，整个东西才会开始跑起来，然后你终于拿到那张你想要的图，那一刻你就知道了。所以我们要找的，是能在 test time 学习的系统；如果我们看到系统表现出有意义的长期适应能力……我们虽然是在开玩笑，但这件事是认真的：我们想用我们自己每天的工作来评估我们的系统——它们会不会一天比一天更擅长干 Core Automation 研究员的活？
[47:57] Rohan Anil
Yeah, we like go on a vacation as a team and see if the lab produces something better. For the week, give the
对，比如我们全队一起去度个假，看看这一周 lab 自己能不能产出更好的东西。就把这一周交给它……
[48:04] 主持人
Then what you do when you get back?
那你们回来之后干什么？
[48:05] Rohan Anil
Um, we'll see what
呃，到时候再看吧……
[48:07] Jerry Tworek
Extend Extend the vacation two times, three or four times, until we are on permanent vacation.
那就把假期再延长——延两倍、三倍、四倍，一直延到我们进入永久休假状态。
[48:14] 主持人
Uh, that is a beautiful note to end on. Um, Rohan, Jerry, thank you so much for joining us. You've both worked on like really, really transformative work for where we are today. And I'm so excited to see you starting a lab on this is new journey. Uh, and very excited to see what you're able to come up with. Thank you for joining us.
这个收尾太漂亮了。Rohan、Jerry，非常感谢两位来上节目。你们各自做过的工作，对我们今天所处的这个局面来说，都是真正意义上的改变格局。看到你们开一个新的 lab、踏上一段新旅程，我特别兴奋，也非常期待你们最后能捣鼓出什么来。谢谢你们来。
[48:36]
It's an It's a great to be here and to chat with you.
能来这儿、能跟你们聊，真的很棒。
[48:39]
Thank you.
谢谢。
[48:43]
[music] [music] [music]
[音乐]