Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again
频道: Sequoia Capital
视频: https://www.youtube.com/watch?v=xH7U7w9Qzlo
原文语言: en
统计: 共 230 轮 · Rich Sutton 89 · 主持人 85 · Khurram Javed 35
[0:00] Rich Sutton
People think I'm have a radical point of view sometimes. They say they start questions saying how what I'm thinking is so different from everyone else. But I don't see it that way at all. I see it as like I'm thinking the ordinary way. It's just everyone else that's thinking a bit weird. And
有时候大家觉得我的观点很激进。他们提问一开口就说，我的想法怎么跟所有人都不一样。但我完全不这么看。在我看来，我才是在用最平常的方式思考，是其他人想得有点怪。而且——
[0:15]
[laughter]
[laughter]
[0:17] Rich Sutton
And I mean that like, you know, it's just the recent times people are thinking weird. Before there was all this AI craziness, uh you talk about you wouldn't have to say continual learning cuz it wouldn't make any sense to talk about learning that wasn't continual. All learning is continual. We always act and we learn. That's just the normal way of thinking. I'm not weird. The field is weird. The field they need to call it continual learning. It's just learning.
我这话的意思是，只不过是最近这些年大家想法变怪了。在 AI 这股疯劲儿出现之前，你根本不用说什么 continual learning（持续学习），因为「不持续的学习」这个说法压根就没意义。所有学习本来就是持续的。我们一边行动一边学习，这才是正常的思路。不是我怪，是这个领域怪。是这个领域非要给它起个名字叫 continual learning。它就是学习而已。
[0:50]
[music]
[music]
[0:59] 主持人
We are honored to have the great Rich Sutton with us here today. Rich, you invented reinforcement learning. You wrote the seminal textbook. You're the key students in the field, uh folks like Dave Silver. You wrote the essay The Bitter Lesson that I believe is the Bible of the field. And and you have just been one of the greats in propelling the field forward. So thank you for taking the time to join us today. Um Rich is joined by Quorum Javed, his co-founder uh and former students from the University of Alberta. Um the two of you have set off to found Oak Lab. I'm very excited to talk to you about that today. So for today's session, we're going to start talking about The Bitter Lesson, the state of the world as we know it today, whether LLMs will get us there or not. And then we're going to we're going to transition to start talking about your your research agenda and your plan for Oak. Um Rich, maybe take us back. I was going to start with The Bitter Lesson, but I actually want to start earlier than that. Decades ago, you decided to dedicate your career to reinforcement learning, to deep reinforcement learning in particular, and you established the University of Alberta as a bastion of that back when I think the field was very much in its infancy. What gave you the conviction to do that?
今天非常荣幸请到了伟大的 Rich Sutton。Rich，你发明了 reinforcement learning（强化学习），写了这个领域的奠基性教科书，还带出了领域里最关键的一批学生，比如 David Silver。你写的那篇《The Bitter Lesson》，我认为堪称这个领域的圣经。可以说你是把整个领域往前推的巨匠之一。所以真的感谢你今天抽时间过来。和 Rich 一起来的还有 Khurram Javed，他是 Rich 的联合创始人，也是他在 University of Alberta（阿尔伯塔大学）带过的学生。你们两位一起创办了 Oak Labs，今天我特别想聊聊这件事。今天这一场，我们先从《The Bitter Lesson》聊起，聊聊眼下这个世界的状态，聊聊 LLM 到底能不能把我们带到终点。然后再转到你们的研究议程，以及 Oak Labs 的规划。Rich，我们先往回倒一点——本来我想直接从《The Bitter Lesson》开讲，但我其实想从更早的地方说起。几十年前，你决定把整个职业生涯押在 reinforcement learning 上，尤其是 deep reinforcement learning，而且你把 University of Alberta 建成了这个方向的重镇——那时候我记得这个领域还非常初期。是什么给了你这份笃定？
[2:10] Rich Sutton
What else you going to do?
不然还能干什么呢？
[2:12]
[laughter]
[laughter]
[2:12] Rich Sutton
We were trying to figure out the mind and learning is a central part of the mind. And having a goal is a central part of the mind. Central part of intelligence. Yeah, so I was just doubling down on what I was always thinking.
我们当时是想搞明白「心智」到底是怎么回事，而学习是心智的核心部分。拥有一个目标也是心智的核心部分，是智能的核心部分。所以我不过是在自己一直以来的想法上继续加码而已。
[2:26] 主持人
Did people think you were crazy at the time?
当时大家觉得你疯了吗？
[2:29] Rich Sutton
Um It was a winter. It was an AI winter.
嗯……那会儿是寒冬，AI 寒冬。
[2:33] 主持人
Uh what year was this?
那是哪一年？
[2:35] Rich Sutton
It was in 2003.
2003 年。
[2:37] 主持人
Okay.
好。
[2:38] Rich Sutton
And it's kind of crazy actually the truth cuz I was like really sick. I was dying I was actually dying of cancer in 2003. And but I I wasn't quite dead, you know, I've been trying for a number of years. And I wasn't dead after another remission. And so so I said, well, I'm not dying I haven't succeeded in dying. So I might as well, you know, it's going on long enough might as well just try to get another job. And so I so I went to Alberta and and and started teaching there. And then in the end I didn't die. It's kind of amazing it's cuz you know, it's it's like that. I'm joking about it now but it was quite serious. And um it's an even more important question. Why why did I continue to work on this research stuff when I was, you know, I only had a few months. I would always keep reminded what I think it's Benjamin Franklin is supposed to have said that, you know, if you ever wonder why someone is doing something it's almost always one of two things. It's either habit or vanity. Okay? So I think I think it's probably true. Maybe it was my habit to just kept doing what I'd always been was doing or maybe it was vanity. I I know. I think it was more like habit cuz I was I was dying.
说实话，这事儿挺离奇的，因为我当时病得很重。2003 年我其实是在等死，癌症晚期。但我一直没死成——你知道，我都折腾好几年了。又一次病情缓解之后，我还是没死。所以我就想，行吧，我没死成，既然拖了这么久，那不如再去找份工作。于是我就去了 Alberta，在那儿开始教书。结果到最后我没死。现在回头看挺神奇的，事情就是这样。我现在能拿它开玩笑，但当时是很严重的。而且还有一个更重要的问题：为什么我以为自己只剩几个月的时候，还在继续做这些研究？我一直记着据说是 Benjamin Franklin 说过的一句话：如果你想不通一个人为什么要做某件事，答案几乎总是两种之一——要么是习惯，要么是虚荣。我觉得这话大概是对的。也许是习惯，让我继续做我一直在做的事；也许是虚荣。我说不好。我觉得更像是习惯，毕竟我当时是在等死。
[3:55] 主持人
Wow.
哇。
[3:56] 主持人
Um
嗯……
[3:56] 主持人
Wow. Divine intervention.
哇，这简直是天意。
[3:57] Rich Sutton
Yeah, it's always been easy for me to keep be very determined. Um and I'm I'm I'm going to go even longer on this answer.
是啊，保持那股一根筋的劲头，对我来说一直都不难。嗯，这个回答我还想再往下说一点。
[4:06] 主持人
Please go.
请讲。
[4:08] Rich Sutton
People think I am have a radical point of view sometimes. They they they start questions saying how how what I'm thinking is so different from everyone else. But I don't see it that way at all. I see it as like I'm thinking the ordinary way. It's everyone else that's thinking a bit weird.
有时候大家觉得我的观点很激进。他们提问一开口就说，我的想法怎么跟所有人都不一样。但我完全不这么看。在我看来，我才是在用最平常的方式思考，是其他所有人想得有点怪。
[4:24]
[laughter]
[laughter]
[4:26] Rich Sutton
And I mean that like, you know, it's just the recent times people are thinking weird. If you go look back what what what what people thought about the mind for, you know, even just a decade, you'll find the kind of thoughts that, you know, learning is important. You've got to have a goal. Um and you know, perception is important. We have a We are We are low-level We are low-level beings. We are generating actions and perceiving data at a fast speed and yet we have to think at higher levels. And you know, go back a few before there was all this AI craziness, uh you talk about you wouldn't have to say continual learning cuz it wouldn't make any sense to talk about learning that wasn't continual. All learning is continual. You know, you It's not a special phase. We always act and we learn. That's just the normal way of thinking. I'm not weird. The field is weird. The field they need to call it continual learning. It's just learning.
我这话的意思是——只不过是最近这些年大家的想法变怪了。你回头去看，哪怕只往回看十年，看看那时候人们是怎么理解「心智」的，你会发现那些想法都挺朴素：学习很重要，你得有目标，感知也很重要。我们本质上是低层级的存在——我们以很快的节奏产生动作、接收数据，可我们又必须在更高的层级上思考。再往前推一点，在 AI 这股疯劲儿出现之前，你根本不用说 continual learning（持续学习）这个词，因为「不持续的学习」这种说法压根没意义。所有学习都是持续的。它不是某个特殊阶段。我们一边行动一边学习，这就是最正常的思路。不是我怪，是这个领域怪。是这个领域非要管它叫 continual learning。它就是学习而已。
[5:22] 主持人
I'm not weird, everybody else is. That's a good good motto to live by.
「不是我怪，是别人怪。」这句座右铭挺值得拿来过日子的。
[5:26] 主持人
We're going to have to send out a an ex post about that.
这句我们得发条 X 帖子出去。
[5:29]
[laughter]
[laughter]
[5:30] 主持人
We're very happy that you you you lived on with the field is happy that you lived on and thank you for pushing the frontier of AI.
我们非常高兴你活了下来，整个领域也很高兴你活了下来，谢谢你把 AI 的边界往前推。
[5:36]
happy.
……高兴。
[5:37]
[laughter]
[laughter]
[5:38] 主持人
Thank you for pushing the frontier of AI. I'm sure really happy and thank you for all of that and push you've like been able to sort of educate a lot of students who pushed the frontier as well. How did you pick them? How did you over the last 20, 30 years?
谢谢你把 AI 的边界往前推。我想大家是真的很高兴，谢谢你做的这一切。而且你还教出了一大批学生，他们同样在往前推这个边界。你是怎么挑学生的？过去二三十年里，你是怎么挑的？
[5:51] Rich Sutton
Oh, well, you are giving me opportunities to be uh to be humble. Like I like to be humble and point out how all these great decisions are are just happen. And it's that that's the way I feel about students. I don't feel that I choose them very well. I've just Sometimes I'm lucky, sometimes I'm unlucky. I don't feel I'm particularly good at picking my students. I'm looking at Karam. I think sometimes you end up with the really great ones. David picked David Silver picked me.
哦，你这是在给我机会谦虚一下。我挺喜欢谦虚的，喜欢指出这些看起来很英明的决定其实都是碰巧撞上的。对学生我就是这种感觉。我不觉得自己挑学生挑得有多好，有时候走运，有时候不走运。我不认为自己在挑学生这件事上特别在行。我现在正看着 Khurram 呢——我想有时候你就是会碰上特别出色的那种。David——是 David Silver 挑的我。
[6:20] Rich Sutton
Yeah.
对。
[6:20] Rich Sutton
How is it that that I got you, Karam?
Khurram，说起来我当初是怎么把你收下的来着？
[6:23] Khurram Javed
Yeah, that was also so I finished my master's not with you uh and I was planning to join industry. And then we were collaborating on a project which also just had started organically. Like there was something I worked on that Rich was in a meeting, then they mentioned that I worked on it. So I got pulled into it. We started collaborating. It went really well. Like I felt so happy with that collaboration. Rich also felt really good about it. And then 6 months down the road we had made some progress and it just made sense to convert that into a thesis proposal. So at no point did I apply, at no point did I ask should you be my PhD advisor. We worked together, then we decided this would be a pretty good thesis. And then then after that I applied for the PhD.
对，这事也挺巧的。我硕士不是跟你读的，当时打算毕业就去业界。后来我们因为一个项目开始合作，也是自然而然发生的——我做过的一个东西，Rich 正好在某个会上听人提起是我做的，就把我拉了进去。我们就这么合作起来了，而且合作得特别顺，我自己觉得非常愉快，Rich 那边感受也很好。半年后我们做出了一些进展，把它变成一份博士论文选题就顺理成章了。所以我从来没正式申请过，也从来没开口问过「你能不能当我的博士导师」。我们先是一起做事，然后觉得这可以做成一篇很不错的论文，之后我才去申请了博士。
[7:08] 主持人
Life works in unexpected ways. Uh take us to 2019. You wrote the bitter lesson which has become the the mother in tome. 2019 was a funny time to be writing that piece because ImageNet was 2009. AlphaGo was 2015. What caused you in 2019 to reflect and and to write that? Because it was before the current kind of scaling paradigm around large language models had taken off, but it was after deep learning had really proven itself.
人生的展开方式总是出人意料。我们说回 2019 年。你写了《The Bitter Lesson》（苦涩的教训），这篇文章后来成了业内的圣经。2019 年写这篇东西其实时机挺微妙的——ImageNet 是 2009 年，AlphaGo 是 2015 年。是什么让你在 2019 年这个时间点回头做这番反思、把它写下来？毕竟那时候围绕大语言模型的 scaling 范式还没起飞，但深度学习已经充分证明了自己。
[7:36] Rich Sutton
Well, it was a long time coming. You know, as the bitter lesson expresses, it's something that you can for a long time, for many decades. And it's definitely at least as much due to the round of symbolic AI, which I lived through. It's all about not getting distracted by trying to put in your human knowledge and just paying attention to what the problem needs and how you can scale with computation. I know I I I wrote versions of it at least a year before and I I gave talks. I gave a talk a year before. And um it wasn't a particular response to the moment. It was a particular response to my my long experience, different people trying to think in different ways about how you can make smart systems.
其实这个想法酝酿了很久。就像文章里说的，这是一件几十年来你都能反复看到的事。而且它至少有一半是来自我亲身经历过的符号主义 AI 那个年代。核心就是：别被「往系统里塞人类知识」这件事带偏，要老老实实盯着问题本身的需求，看你怎么能靠算力扩展上去。我记得至少在那之前一年我就写过几个版本，也讲过好几次，前一年还专门做过一场讲座。所以它不是对某个具体时刻的回应，而是对我漫长经历的回应——这么多年来，我看着不同的人用各种不同的思路去琢磨怎么造出聪明的系统。
[8:28] 主持人
Mhm. What is the essence of the bitter lesson?
嗯。那 bitter lesson 的精髓到底是什么？
[8:32] Rich Sutton
The essence of the bitter lesson.
bitter lesson 的精髓啊。
[8:33] 主持人
You know, and maybe the phrase that I hear used the most in my meetings these days is is bitter lesson pill, is it not bitter lesson pill? I would imagine given the popularity of the phrase it's probably been tortured and misused in different ways that you didn't originally intend it. So, what do you think what is the essence of it and where do you think people go wrong in their in their attempt to understand it?
你知道吗，我最近开会时听得最多的一句话大概就是「这人 bitter lesson pilled 了没」（是不是被 bitter lesson 洗脑了）。我猜以这个词的流行程度，它多半已经被各种曲解和滥用过了，扭成了你当初根本没打算表达的意思。所以在你看来，它的精髓是什么？人们试图理解它的时候，通常错在哪里？
[8:53] Rich Sutton
Yeah, you're you're making me think about X now and my I recently made a post where I tried to do the the bitter lesson in in 26 words. It [laughter] goes something like don't be distracted by human knowledge as AI traditionally has been many times. Instead focus on learning methods that will scale with computation like search and like learning. So, it's really all about uh focusing on algorithms and improvements. It's not it's not saying you don't need fancy algorithms. You need fancy algorithms, but you want fancy algorithms that will scale with scale with computation.
对，你这么一说我想到我在 X 上发的东西了。我最近发过一条帖子，试着用 26 个词把 bitter lesson 讲清楚。[laughter] 大意是：别像 AI 历史上无数次那样，被人类知识带偏；相反，要专注于那些能随算力一起扩展的学习方法，比如 search（搜索）和 learning（学习）。所以它讲的其实全是算法和改进。它并不是说你不需要精妙的算法——你需要精妙的算法，但你要的是那种能随算力一起扩展的精妙算法。
[9:40] 主持人
Rather than scaling with data.
而不是随数据一起扩展。
[9:41] Rich Sutton
Rather than scaling with human input. Yeah, and then the the question if I can anticipate it um Yeah, what about large language models?
是不能随「人类输入」一起扩展。对。然后接下来那个问题，如果我猜得没错的话——那大语言模型（LLM）算怎么回事？
[9:52] 主持人
Yeah.
对。
[9:53] 主持人
Are they
它们究竟
[9:54] 主持人
consistent or inconsistent with your
是符合还是违背你的……
[9:56] Rich Sutton
Yeah.
对。
[9:57] Rich Sutton
And and I thought about this and I think there's an ex- there's another exposed model, but the conclusion is that it's both a a positive example and a negative example of the big lesson. First uh large language models enabled uh enormous scaling with computation. And you could just drink in the internet and scale so much. So it was a it was a way of getting uh much more capable system just by methods that scale. Then after that, as you go on further um it eventually gets limited by by that information. The the internet is finite and it's hard to get more examples. And uh the world is big and the world is massively bigger than everything we stored on the internet. And so in the end, it seems like it could be uh I guess that would be a positive example of when, you know, we relied too much on human knowledge and it eventually holds us back.
这个我想过。我觉得这里还有另一层要讲，但结论是：它既是 bitter lesson 的正面例子，也是反面例子。先说正面：大语言模型让算力的规模扩展变得空前巨大，你可以把整个互联网一口吞下去，规模拉到非常大。所以它确实是靠「能扩展的方法」拿到了强得多的系统。但再往后走，它最终会被那批信息卡住。互联网是有限的，再想拿到更多样本很难。而世界很大，世界比我们存进互联网的所有东西加起来还要大得多。所以到最后，它看起来又成了那个典型例证——我们过度依赖了人类知识，而它最终会把我们拖住。
[10:59] 主持人
Mhm. Can I just push on this a little bit?
嗯。我能就这点再追问一下吗？
[11:02] Rich Sutton
Yeah.
可以。
[11:03] 主持人
It seems like a lot of what the foundation model labs are working on right now is synthetic data generation in order to kind of get us beyond the fossil fuel that is the existing human internet. Um is synthetic data generation kind of as part of this LLM scaling paradigm, is that a general method that leverages computation?
现在各家基础模型实验室手头做的很大一块事情，是合成数据生成（synthetic data generation），目的就是突破「现有人类互联网」这块化石燃料的天花板。那么，作为 LLM scaling 范式一部分的合成数据生成，算不算一种能撬动算力的通用方法？
[11:24] Rich Sutton
No, that's that's just a big mistake.
不算，那纯粹是个大错误。
[11:27] 主持人
Why?
为什么？
[11:27]
[laughter]
[laughter]
[11:28] Rich Sutton
Well, it's such a big it's such a maybe it's the next the next big lesson. Um it's been floating around uh Alberta for 5 or 10 years.
因为这事太大了……也许这就是下一个 bitter lesson。这个想法在 Alberta 已经流传了五到十年。
[11:40] 主持人
Okay.
好。
[11:40] Rich Sutton
And uh we call it the big world perspective or big world hypothesis. Khuram, who eventually wrote it up as a paper. There's a little paper called the big world hypothesis.
我们管它叫「大世界视角」或者「大世界假说」（big world hypothesis）。后来 Khurram 把它写成了一篇论文，一篇不长的文章，就叫《The Big World Hypothesis》。
[11:50] Khurram Javed
So, the big world is that the world is infinitely big. There are infinitely many things to learn. And you can have people generating the synthetic data sets, but there will always be more things to learn. And because of that, if you could just learn from experience, if you could remove the humans from the loop, then you would have systems that can do everything. Because, you know, the world is big. There are many tasks that we want them to do. And they would be able to do anything by learning from their experience. Going back to the synthetic data question, too. Uh who decides what's a good synthetic data and what's a bad synthetic data? Because I can write a program that can output a lot of synthetic data, which would hurt programs. Right now, I would say humans decide. And that's the bottleneck where okay, you can have humans deciding how to generate these data sets, but you need human experts who know what's a good data set and what's a bad data set for that approach to scale. So, it is bottlenecked by humans.
大世界的意思是：世界是无限大的，要学的东西无穷无尽。你当然可以让人去造合成数据集，但永远还有更多东西要学。正因如此，如果你能直接从经验中学习、把人从循环里拿掉，你就能得到什么都能做的系统。因为世界很大，我们想让它们做的任务也很多，而它们靠自己的经验学习就能做成任何事。再回到合成数据那个问题：谁来判断哪些合成数据是好的、哪些是坏的？我完全可以写个程序吐出海量合成数据，但那些数据只会把模型训坏。就现状而言，做判断的是人。这就是瓶颈所在——好，你可以让人来决定怎么生成这些数据集，但这条路要想规模化，你就得有懂行的人类专家来判断什么是好数据集、什么是坏数据集。所以它被人卡住了。
[12:45] 主持人
Doesn't my loss curve decide like how much better did I get with this data set versus that better data set?
难道不是我的 loss 曲线在做判断吗？用这个数据集比用那个数据集我提升了多少，看曲线不就知道了？
[12:50] Khurram Javed
Right. But, if all the engineers open AI and tropic or all the big new labs and engineers went on vacation, who would generate the synthetic data? That's the question. It doesn't doesn't come from agent's experience. It's not something that the agent is generating itself. Some human has to decide what is the right synthetic data to generate. And that requires human expertise. So, for example, if you want a system to do something very challenging from a physics point of view. Maybe you want a drone that flies with echolocation, like a bat, for example. Um what's the right synthetic data for that? I think you would need to hire domain experts to go figure out what is the right data and generate it and then maybe you would be able to learn from that. But the domain expert has to exist first. So, we are bottlenecked by human expertise at that point.
没错。但如果 OpenAI、Anthropic 还有其他所有大实验室的工程师集体去度假了，谁来生成合成数据？这才是问题。它不来自 agent 自身的经验，不是 agent 自己产生出来的东西。总得有个人来决定该生成什么样的合成数据，而这需要人类的专业能力。举个例子，你想让系统做一件物理上很有挑战的事，比如你想要一架靠回声定位飞行的无人机，像蝙蝠那样。那这件事该配什么样的合成数据？我觉得你得先雇领域专家，让他们去搞明白什么才是对的数据、然后把数据造出来，之后模型也许才学得会。但前提是这个领域专家得先存在。所以到那一步，我们就被人类的专业能力卡住了。
[13:37] 主持人
But you can have infinite synthetic worlds. The existing world is finite.
可是合成世界可以有无限多个啊。现实世界才是有限的。
[13:42] Khurram Javed
But let's go back to the echolocation thing, right? That's what I want. I want a drone that can uh localize itself and move with echolocation. That's my goal. Um the robot that that's a robot that's generating its own experience. So, it could totally learn from its own experience, but it wouldn't be able to It doesn't matter how much synthetic data you generate. Doesn't matter if you generate synthetic data that captures 50 different universes, it will not be not allow you to do that task without humans figuring it out first.
那我们还是回到回声定位那个例子。我想要的就是一架能靠回声定位来定位自身、完成移动的无人机，这是我的目标。这个机器人是在自己生成经验的，所以它完全可以从自己的经验里学。但你生成多少合成数据都没用，哪怕你生成的合成数据涵盖了 50 个不同的宇宙，只要人类没有先把这件事想明白，它就还是做不成这个任务。
[14:12] Rich Sutton
I first just say it's it's the synthetic data is wrong. I mean, it won't be correct. It'll be a synthetic world. It won't be the real world. And it will matter. The world is incredibly complex. If you write a little program, cuz this is going to be a little program that will generate the synthetic data, it'll be a very It'll be a small world.
我首先想说的是，合成数据（synthetic data）这条路本身就是错的。它不可能是对的——那是一个合成出来的世界，不是真实世界，而这个区别是要命的。世界复杂到难以想象。你写一个小程序——因为生成合成数据的说到底就是这么一个小程序——那它造出来的注定是一个很小的世界。
[14:35] 主持人
Mhm.
嗯。
[14:36] Rich Sutton
So, for example, what's important to me is what's going on in your mind right now. Okay? And why You're saying, "Why don't I get some synthetic data to tell me what's going on in other people's minds?" No, there's no way we can have synthetic data for other people's minds. And other people's minds matter to us. You know, like I talked to you guys about investing today, so I care what you going on in your minds. And how how can I get synthetic data on such a thing? Really, you can't even get synthetic data on anything. You can't get synthetic data on on the how the drone is going to interact with with its environment in the physical world and the the the the friction and where in the motors of this of this robot. The world is infinitely complex, and any simulation of it is like microscopic. The big world hypothesis, let's say what it is, is that the world is massively more complex than your mind, than any agents any agent. And this is obvious because the world contains many other agents. So, because the world is is massively complex, as you could no way you can do anything like anything that might claim to be optimal or perfect. You're going to be imperfect, and you have to have approximations, and those approximations will will be severe. And so, because of that, that is the ultimately the reason why we have to continue learning, if you want to think of it as a reason.
举个例子，对我来说重要的是你此刻脑子里在想什么。对吧？可你会说："那我为什么不弄点合成数据，来告诉我别人心里在想什么呢？"不行，我们根本没办法为别人的心智造出合成数据。而别人的心智对我们来说恰恰是重要的。你看，我今天跟你们聊投资的事，我当然在意你们脑子里怎么想。这种东西我上哪儿去搞合成数据？说真的，几乎什么东西你都搞不到合成数据。无人机在物理世界里怎么和环境互动、这台机器人的电机里摩擦是什么情况——这些你都造不出来。世界的复杂度是无限的，任何对它的模拟都只是显微镜下的一小点。所谓「大世界假设」（big world hypothesis），说白了就是：世界远比你的心智、比任何 agent（智能体）都要复杂得多得多。这一点显而易见，因为世界里还装着无数别的 agent。正因为世界复杂到这个程度，你根本做不出任何敢自称最优或完美的东西。你注定是不完美的，你必须依赖近似（approximation），而且这些近似的偏差会很大。也正是因为这个，才有了我们必须持续学习（continual learning）的根本原因——如果你非要给它找个理由的话。
[16:08] Rich Sutton
We have to continue learning because we'll encounter some particular part of this immense world, and we'll have to learn an approximation that's tuned to the part of the world we're in, not to the all the other parts that we're not in.
我们必须持续学习，是因为我们只会撞上这个庞大世界里某个特定的局部，我们得学出一个针对当下所处这块地方调校过的近似，而不是针对那些我们根本不在其中的其他部分。
[16:19] 主持人
Yeah. I'm I'm going to push on this one more time. Um, and sorry, I'm being argumentative for the sake of being argumentative, [clears throat] but I'm trying to understand. My understanding is that the newest cohort of self-driving car companies, many of them were primarily trained in sim, and then they, you know, have to do some some sort of post-training, I guess, to to make sure they work in the real world, but but that it's been a very effective pipeline.
好，这一点我还想再追问一次。抱歉，我有点为了抬杠而抬杠，[clears throat] 但我是真想搞明白。据我了解，最新这一批自动驾驶公司里，很多都主要是在仿真环境（sim）里训练的，然后再做某种 post-training（后训练），确保它们在真实世界里跑得通——而这条流水线一直挺有效。
[16:42] Khurram Javed
Yeah. So, I think the important question to ask here is, how many engineers were involved in building that simulation, and are we ready to say that the only problem worth solving are those where we can hire a large team of engineers to first make a simulation. And I'm sure they had to do multiple iterations where they made the simulation, they learned in it, they realized there was a sim-to-real gap that was not acceptable, then they fixed it. So, there is this human in the loop fixing the simulation. Like, they're getting feedback from the real world, humans, and then they're fixing the simulation. Why can't we just remove the human and let the agent do it itself.
对，我觉得这里该问的关键问题是：为了搭起那套仿真环境，前前后后投进去了多少工程师？我们真的愿意接受这个前提吗——只有那些能先雇一大队工程师做出仿真的问题，才是值得解决的问题？而且我敢肯定，他们经历了很多轮迭代：先做仿真，在里面训练，然后发现 sim-to-real（仿真到现实）的差距大到不可接受，再回头去修。所以这里始终有一个 human in the loop（人在回路中）在修仿真。他们从真实世界拿到反馈，然后由人去改仿真。那为什么不干脆把人拿掉，让 agent 自己来做这件事呢？
[17:18] Rich Sutton
And then when it actually drives, again, something unexpected will happen.
而且等它真正上路开起来，照样会碰到意料之外的情况。
[17:22] Khurram Javed
And that's when you really want to learn from experience.
那恰恰就是你真正需要从经验中学习的时刻。
[17:24] 主持人
So, your point is there's just so much more data that's going to come from experience than there possibly can be from humans curating and creating data.
所以你们的意思是，从经验里能拿到的数据量，远远超过人类去筛选、去制造数据所能给出的量？
[17:34] Khurram Javed
Yeah, and think there is a there is obviously value in learning from simulation. And and there is a way of doing it. The agent can learn a model from its own experience. And when the agent learns it, it's very it's much better because if the model is incorrect, it can fix it by continuous learning. If the humans are making a simulator, then the model only gets updated when the humans figure out that something is wrong. So, yes, planning is important. The agents should learn from uh simulators, but simulators they make themselves.
对。而且我觉得，从仿真里学习显然是有价值的，也确实有一种成立的做法：agent 可以从自己的经验中学出一个模型。而 agent 自己学出来的模型要好得多，因为一旦模型不对，它能靠持续学习把它修正回来。可如果仿真器是人做的，那这个模型只有在人发现哪里出了问题时才会被更新。所以，规划（planning）当然重要，agent 也应该从仿真器里学——但那得是它自己造出来的仿真器。
[18:01] 主持人
Okay, I want to move to another part of the bitter lesson, removing human knowledge. From your essay, quote, "Seeking the improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain. But the only thing that matters in the long run is the leveraging of computation." And if I, you know, your former student Dave, uh with AlphaGo and AlphaZero, for me that was a an example of a triumph of removing human priors. Did that result surprise you? I guess why or why not?
好，我想转到《The Bitter Lesson》（苦涩的教训）的另一个部分——剔除人类知识。你在文章里写道：研究者为了追求短期内见效的改进，会去利用自己在该领域的人类知识；但长期来看，唯一重要的是对计算（computation）的利用。对我来说，你以前的学生 David Silver 做的 AlphaGo 和 AlphaZero，就是剔除人类先验（prior）的一次胜利。这个结果让你意外吗？为什么意外，或者为什么不意外？
[18:31] Rich Sutton
Of course, it made me very happy. It made me, you know, feel vindicated. Um you know, it could have gone either way. It wasn't that uh cuz cuz prior knowledge can help. You know, there's nothing wrong with prior knowledge. You know, and and I say this right at the very beginning of the right at the very beginning of the bitter lesson, I say there's no reason why there has to be a conflict between prior knowledge and then learning knowledge. You know, you can put some prior in there and then start learning. There's no reason in principle why these have to be opposed. In fact, they're all about knowledge. You know, life is gaining knowledge and having knowledge. And why are these how somehow, you know, nature and nurture became enemies? but really you know, prior learning is what you already and then and you then you learn more and it's they should be friends. But, as I say at the beginning of the bitter lesson, in practice they have been enemies. In practice, people who who had a an affection for existing human knowledge ended up, you know, wanting that to win and so they wanted to to minimize or or dismiss learning. And so now, I'm sure your sense of me is that I'm someone who who loves learning and wants to dismiss prior knowledge. Um but you know, I'm really someone who's interested in the mind.
当然，这让我特别高兴，有一种被证明是对的感觉。不过你知道，这事本来两边都有可能——因为先验知识（prior knowledge）确实是能帮上忙的。先验知识本身没什么错。我在《苦涩的教训》一开头就说了：先验知识和后天学到的知识之间，没有理由非得对立。你完全可以先塞进去一些先验，然后再开始学。原则上它们没有必要对立。说到底它们都是知识嘛——人生就是不断获得知识、拥有知识。可不知怎么回事，先天和后天（nature and nurture）就成了敌人？其实先验就是你已经有的东西，你在这个基础上再往上学更多，它们本该是朋友。但正如我在《苦涩的教训》开头写的，实践中它们确实成了敌人。实践中，那些对既有人类知识抱有感情的人，最后就希望自己这一边赢，于是拼命弱化甚至否定学习。所以现在你对我的印象八成是：这人偏爱学习、想把先验知识一脚踢开。但其实，我真正感兴趣的是心智。
[19:51] Rich Sutton
The mind is you have prior knowledge and then you get more and then once you've gotten more, then that becomes your prior knowledge as you get more and more and more. And this these two things work together. I end up appearing to be someone who's who's interested in learning primarily because all the rest of the world is is is is talking about all you need is enough knowledge. You don't need to learn. You know, large language models are we're going to put all this knowledge in the into the system and the large language model will not learn when it runs. You know, you know, it's talking to people, it's interacting. It is absolutely the weights never change. So, you know, I am not the weird one. It's you guys that are the weird one that think that that's possible that you could possibly, you know, they claim they can like a PhD level experience and expertise out of something that doesn't learn at all anymore. You know, so you know, I'm not the weird one.
心智就是这样运作的：你先有先验知识，然后获得更多，等你获得更多之后，那些又变成你新的先验知识，如此不断往上叠。这两件事是互相配合的。我之所以最后看起来像个只关心学习的人，是因为整个世界其他人都在说：你只要有足够多的知识就够了，不需要学习。你看大语言模型（LLM），我们把所有这些知识灌进系统里，而这个大语言模型在运行的时候是不学习的。它在跟人说话，在跟外界互动，可它的权重（weights）绝对一动不动。所以你知道吗，怪的不是我，是你们这些人才怪——居然觉得这事能成立，居然声称一个完全不再学习的东西能有博士级的经验和专业水准。所以说，怪的真不是我。
[20:47]
[laughter]
[laughter]
[20:47] 主持人
So, your recommendation is let it let the algorithms run for much much longer period of time before feeding
所以你的建议是，让算法先跑上很长很长一段时间，然后再喂给它——
[20:52] Rich Sutton
Continually learn.
持续地学。
[20:54] 主持人
before you feed it data. with prior data. So, drip or drip the prior data along the way. prior knowledge.
——在你喂数据之前。用先验数据。也就是说，把先验数据沿路一点一点滴灌进去。先验知识。
[21:04] Rich Sutton
So, both are important. Um but in the long run, you've got to gain and structure the gaining of new knowledge. That's that's what all that matters in the long run. And as you are doing this, yeah, there'll be some that you had got previously. Um, like how it would work if we, you know, look into the future when we have intelligent robots. We will we will will we uh, have them all learn from scratch? Or will we like copy them and ask them to keep learning from wherever they are? I mean, we they'll be digital and it'll be easy to copy them. And so, instead of having like this huge thing where we're spending zillions of dollars to retrain them from the internet, we'll just copy the agent and keep learning from there. And and so, in some sense, the prior knowledge will be should be dismissed cuz you're just going to copy it from the previous robot.
两者都重要。但长期来看，你必须去获得新知识，并且把「获得新知识」这件事本身结构化。长期来看，只有这件事是重要的。而在你这么做的过程中，确实会带上一些你之前已经拿到的东西。打个比方，往未来看，等我们有了智能机器人，会怎么办？我们是让它们全都从零开始学吗？还是把它们复制一份，让它们从当前的状态接着往下学？它们是数字的，复制起来很容易。所以我们不会再搞那种烧掉天价、从整个互联网重新训练一遍的大工程，而是直接复制那个 agent，从那儿接着学。从这个意义上说，先验知识确实应该被搁到一边——因为你直接从上一台机器人那里拷过来就行了。
[22:02] 主持人
So, why don't you describe for us what you think a machine or computer that learns from experience looks like?
那你能不能给我们描述一下，在你看来，一台能从经验中学习的机器或者计算机，长什么样？
[22:09] Rich Sutton
Well, it could look like a robot. It could be It also could be live entirely on the internet. You could like, for example, routing a package through the internet and do that in a way that's sensitive to experience and and becomes better over time. Or you can interact via the user interface that's interacting with people, like on your phone or on your computer, and uh, it becomes better over time. Yeah, like an intelligent assistant, you know, has to become better over time. It has to know what you want.
它可以长得像个机器人，也可以完全活在互联网上。比如说，在互联网上给数据包做路由，而且是以一种对经验敏感、会随时间越做越好的方式。或者它通过用户界面跟人打交道，在你手机上、电脑上，然后随着时间变得越来越好。就像一个智能助手，它必须随时间变得更好，它得知道你到底想要什么。
[22:37] 主持人
Would your contention be that the current paradigm of, you know, the popular A L M based assistants, would your contention be that these are not experiential learners or continual learners? And if so, what is the fundamental gap?
那你的看法是不是：现在流行的这些基于 LLM 的助手，都算不上从经验中学习的学习者、也算不上 continual learner（持续学习者）？如果是的话，根本的差距在哪里？
[22:52] Rich Sutton
Are you serious?
你是认真的吗？
[22:54]
[laughter]
[laughter]
[22:55] Rich Sutton
I mean, obviously they
我是说，很明显它们——
[22:56] 主持人
They they have memo- they learn memories about me. They're you know, they're they're they're doing some in-context learning.
它们有记忆啊——它们会记住关于我的事。它们在做某种 in-context learning（上下文内学习）。
[23:03] Rich Sutton
Their weights never change.
它们的权重从来不变。
[23:05] 主持人
And and by the way, is a small number of the weights changing sufficient or do you need all the weights to be changing?
顺便问一句，只有一小部分权重在变，够不够？还是说必须所有权重都在变？
[23:10] Rich Sutton
Well, so all think of all the structuring and generation of new concepts that went into creating the large language models. All that is the weight learning. The and you you want to continue be able to continue doing that. You don't want that to happen just once.
你想想，为了造出这些大语言模型，其中做了多少结构化的工作、生成了多少新概念——那全都是权重学习（weight learning）干出来的。而你想要的，是能继续做这件事，而不是只做这么一次就完了。
[23:27] 主持人
Is another way of saying it is we do too much pre-training and post-training before we launch the the models. They they don't learn after that.
换个说法是不是可以这么讲：我们在模型上线之前做了太多 pre-training（预训练）和 post-training（后训练），上线之后它们就不学了。
[23:35] Rich Sutton
The only point that the big disagreement is we don't let them learn after that.
唯一的、也是最大的分歧点就在这儿：我们不让它们在那之后继续学。
[23:38] Rich Sutton
Yeah, we don't let them learn after that.
对，那之后我们就不让它们再学了。
[23:40] Khurram Javed
as much pre-training as we want, that's okay. Post-training is fine, but then when I'm when I'm using the model I cannot it's it's it stops learning. Uh you can give it more context. You can change the state of the model by giving it more context. And so it it already learned that if the state is different, if the state says something new, then it will use that to make the next prediction. But the model is not learning.
想做多少 pre-training（预训练）都行，没问题；post-training（后训练）也没问题。但等我真正开始用这个模型的时候，它就不再学习了。你可以给它更多 context（上下文），通过喂上下文来改变模型的 state（状态）。它在训练阶段已经学会了这件事：如果 state 变了、如果 state 里出现了新东西，它就会拿这些去做下一步预测。但模型本身并没有在学习。
[24:01] 主持人
Cursor's tab other complete model. It is, you know, it does get updated based on
那 Cursor 的 tab 自动补全模型呢？它确实是会被更新的，基于——
[24:07] Khurram Javed
Those models those weights change.
那些模型——那些权重是会变的。
[24:09] 主持人
weights change. Those are two examples like Cursor's tab and I think the composer they were also updating. Those are two examples of continual learning.
权重会变。这就是两个例子——Cursor 的 tab 补全，我记得 Composer 他们也在持续更新。这算是 continual learning（持续学习）的两个例子。
[24:15] Khurram Javed
Okay.
好吧。
[24:16] Khurram Javed
it's can be much better. So the way they do it as far as I understand is a lot of people are using tab, they collect all this data so coming from millions of users or thousands of users and then they do one update of the policy from this batch data. Um so now this could work, but imagine I want to teach this model something specific. I don't want to fight with 100,000 other peoples about what they want to treat teach their models. I want to teach my model something very specific and I want to do it to my version of the model. I like I don't care about the shared knowledge that the model has coming from other people. And so it's a very inefficient way of doing it.
但这件事其实可以做得好得多。据我了解，他们的做法是：很多人在用 tab 补全，他们把这些数据全都收集起来——来自成千上万、甚至上百万用户——然后拿这批 batch 数据对策略做一次更新。这当然能跑通。但你想象一下：我想教这个模型一件很具体的事，我并不想跟另外十万人去争「该教模型什么」。我想教的是我自己那一份模型，教它一件非常具体的事。别人贡献的那些共享知识，我其实并不在意。所以这种做法效率非常低。
[24:50] 主持人
Mhm. It seems like the way that this is currently done is that there's fundamental skills maybe that are learned in the weights that are common to everybody. And then there's personalization that happens in the form of context, right?
嗯。看起来现在的做法是这样：一部分基础能力被学进了权重里，这部分对所有人是共用的；然后个性化的部分靠 context 来实现，对吧？
[25:04] Khurram Javed
Yeah.
对。
[25:05] 主持人
Is that not the right mental model for how learning should work? Like should should all the context live in the weights themselves?
那这个心智模型是不是错的？学习本来不该这么工作？难道所有的 context 都应该落进权重本身？
[25:11] Khurram Javed
So, context can be in the state, too. Could be both. But, you still need to be able to update the weights. So, if I give you an example, some really good use studies are for with human disabilities. When human go through something that changes their mind or or some sensors, you can see them adapt. So, for example, we have proprioception, we have internal sensors that tell us where the how the body is positioned, and we use this for walking. There are cases where people lose this ability completely, and then they can't walk at all because that is literally the foundation of their walking policies. It is ingrained in the brain. But then over the course of 2 3 years, they can learn to walk again by looking at their feet. So, visual feedback through that. So, brain is insanely plastic in the sense that it it can learn a lot of things. Something that has been true for 20 years, when it stops being true, it can go and update that and get rid of that. And that is the capability I think that's extremely useful we would want in our systems.
context 也可以放在 state 里，两边都行。但你终归得有能力去更新权重。举个例子，人类残障领域有一些特别好的研究：当一个人的大脑或者某些感官发生了变化，你能看到他们是怎么重新适应的。比如我们有 proprioception（本体感觉），身体内部有传感器告诉我们四肢处在什么位置，走路全靠它。有些人会彻底失去这个能力，于是完全不会走路了——因为那是他们走路策略的根基，是刻进大脑里的东西。但在接下来的两三年里，他们能靠盯着自己的脚重新学会走路，也就是换成视觉反馈。所以大脑的 plasticity（可塑性）强得惊人：它能学会大量东西，而且一件事哪怕已经成立了二十年，一旦不再成立，它也能去更新、把旧的那一套扔掉。我认为这正是我们希望自己的系统具备的能力，非常有用。
[26:09] 主持人
Hm. What is there for us to learn from how human babies or animals learn? And how much inspiration do you take from that?
嗯。人类婴儿或者动物的学习方式，有哪些是我们该借鉴的？你们从中取了多少灵感？
[26:16] Rich Sutton
Well, we take uh a lot of inspiration. We don't take it as a requirement that the AI has to behave like uh the natural system, like babies or people or animals. Um but it's it's a source of inspiration. Inspiration, but not constraint from animal learning.
灵感取了很多。但我们不会把「AI 必须像自然系统那样运作」当成硬性要求——不管是婴儿、人还是动物。它是灵感的来源，不是约束。动物学习给我们的是启发，不是限制。
[26:35] 主持人
Consistent with the better lesson?
这跟 bitter lesson（《苦涩的教训》）是一致的？
[26:36] Rich Sutton
Yeah.
对。
[26:37]
[laughter]
[laughter]
[26:37] Rich Sutton
Yeah.
对。
[26:39] 主持人
Where do you think we should most seek to draw inspiration from the way that biological learnings works that is not present in today's systems?
生物的学习方式里，有哪些是今天的系统完全没有、而你觉得我们最该去借鉴的？
[26:48] Rich Sutton
I feel like I'm just giving opinions now, but they're just obvious opinions. So, so I think it's apparent that no animal learns by supervised learning. Because we don't get examples of how our muscles should twitch. And that's our output.
我感觉我现在纯粹是在发表个人看法了，不过都是些明摆着的看法。我觉得很明显：没有任何动物是靠 supervised learning（监督学习）学会东西的。因为没人给我们「肌肉该怎么抽动」的标准样例——而那才是我们的输出。
[27:07] 主持人
But all of school is supervised learning.
可上学不全都是监督学习吗？
[27:09] Rich Sutton
I I know. Absolutely not. Uh, but even if it was, school is like a tiny fraction of what we learn. Like we learn to see, we we learn to walk. And we learn, um, how the world works. But even Yeah, and even in school, you know, no one tells us how we should twitch our muscles.
我知道你会这么讲。但完全不是。而且就算真是，学校也只占我们所学的极小一部分。我们要学会看、学会走路，学会这个世界是怎么运转的。就算在学校里，也没人告诉你肌肉该怎么抽动。
[27:31] 主持人
The knowledge skills I acquire were were from supervised learning in school.
我掌握的那些知识和技能，确实是在学校靠监督学习拿到的。
[27:35] Rich Sutton
don't want to say that that that learning from from others, transmission from others, is not important. It's like extremely important. And language is extremely important. Um, but what are we what are we missing? You know, there is there is no supervised learning. There's no targets that are given to us. You know, you you hear the right answer is, you know, where where is what's the capital of France? And we know the answer is Paris. Okay, but no one tells me how I should pronounce Paris. You say the answer is Paris, and I listen to you, and I hear your words, and, you know, I will make some other uh, muscle motions to produce the answer Paris. It's not literally supervised learning. Um, anyway, yeah. So, I think it's really true. I mean, well, anyway, the first thing is a school is is irrelevant. Like, you know, squirrels don't go to school and and and learn [laughter] that.
我不是说向他人学习、知识在人与人之间传递不重要——那极其重要，语言也极其重要。但我们漏掉的是什么？那里其实并没有监督学习，没人给我们喂标准答案。你听到有人问「法国的首都是哪儿」，我们知道答案是巴黎。可没人告诉我「巴黎」这两个字该怎么发音。你说出答案 Paris，我听着你说，听到你的声音，然后我得自己做出另一套肌肉动作，把 Paris 这个答案发出来。这不是严格意义上的监督学习。总之——我觉得这点是真的成立。反正第一件事是：学校根本不相关。松鼠又不上学，也不靠上学学会那些本事 [laughter]。
[28:37] 主持人
They might.
说不定它们还真上呢。
[28:37] Rich Sutton
Animals don't learn that way. It's And school is a very special thing that that even we didn't have up until, you know, I don't know, a few hundred years ago. It's not part of in not part of the essence of intelligence? And it's a distraction to think of that as your primary example of learning is this thing which we didn't do as animals.
动物不是那样学的。学校是个非常特殊的东西，连我们自己在几百年前——具体多久我也说不准——都还没有它。它不属于智能的本质。把它当成你理解学习的首要范例，其实是一种干扰，因为那根本不是我们作为动物时做的事。
[28:59] 主持人
I wish you had been around to tell my parents that before I was made to have good go to school and deal with all the structure.
真希望当年你在场，在我被送去上学、去应付那一整套条条框框之前，跟我爸妈讲讲这番话。
[29:08] Khurram Javed
The thing is like squirrels are wonderful at jumping off trees, but squirrels can't prove math theorems. And if I want to learn how to prove a math theorem, I go to school.
问题在于，松鼠特别擅长从树上往下跳，但松鼠证明不了数学定理。而我要是想学会怎么证明一个数学定理，我就得去上学。
[29:14] Rich Sutton
Yeah. Uh they also don't have uh DVDs and
是啊。它们也没有 DVD——
[29:20]
[laughter]
[laughter]
[29:21] Rich Sutton
and iPods. You know, there are a lot of things they can do things that we can't do. Um but
——也没有 iPod。它们能做很多我们做不到的事。不过嘛——
[29:27]
[sighs]
[sighs]
[29:28] Rich Sutton
math theorems uh Yeah, and they don't play chess. You know, it's sort of like more of X paradox. They're uh they're these advanced things that we think of as really intelligent. But uh they're sort of easy for computers to do as opposed to all these regular things that are hard. Like moving and seeing with attention and everything. Um I think supervised learning is a is a good thing. You know, just mentions I like to think look for obvious things. No one tells us how to twitch our muscles by giving us examples cuz they couldn't possibly cuz we have had to twitch our muscles. We've had to figure that out.
……数学定理，对，而且它们也不会下棋。这有点像莫拉维克悖论（Moravec's paradox）：那些我们觉得特别高深、特别能代表智能的事，计算机反而做得挺轻松；难的恰恰是那些日常动作——移动身体、带着注意力去看东西之类的。supervised learning（监督学习）本身当然有它的价值，不过我习惯先盯住最显眼的事实：没有人是靠给我们打样、举例子来教我们怎么控制肌肉的——他们也根本教不了，因为肌肉该怎么动，只能靠我们自己摸索出来。
[30:06]
Yeah.
嗯。
[30:06] Khurram Javed
And their answer would be wrong, right? So, if I moved my mouth and my tongue and my vocal cords exactly the same way that Rich does to pronounce Paris, I'm sure a very different sound would come out. So, in some sense Rich or no one knows the right way of producing a sound with my body. Only I know that.
而且就算他们教，给出的答案也是错的，对吧？你想，如果我把嘴、舌头、声带完全照 Rich 的方式去动，念出来的 Paris 肯定是另一个声音。所以从某种意义上说，Rich 不知道、谁都不知道我这具身体该怎么发出这个音——只有我自己知道。
[30:24] 主持人
Yeah. It seems to me that many of the most, I guess the most raw like sensory motor capabilities, especially related to movement in the physical world. I agree with you that that seems something that is inherently learns from experience. It seems to me though that there are higher levels of abstraction that bring us closer to, you know, what makes humans great. And much of that doesn't live in this low level of sensory motor learning. Does your world model, I guess, span sensory motor learning all the way up?
嗯。我觉得很多最原始的感知—运动能力，尤其是跟在物理世界里移动有关的那部分，我同意你，那确实天然只能从经验里学。但在我看来，再往上还有更高层的抽象，那些才更接近「人之所以了不起」的地方，而它们大多并不住在这种底层的感知—运动学习里。你们的世界模型（world model）打算从感知—运动一路往上覆盖到那一层吗？
[30:56] Rich Sutton
Yeah, that's the ambition, absolutely. And squirrels, by the way, can do some enormously abstract things.
对，这就是我们的野心，绝对是。顺便说一句，松鼠也能干出一些抽象程度极高的事。
[31:03] 主持人
What's the coolest thing a squirrel can do?
松鼠最厉害的一手是什么？
[31:05] Rich Sutton
Well, it can always get into your bird feeder,
它总有办法弄开你家的喂鸟器，
[31:08]
[laughter]
[laughter]
[31:08] Rich Sutton
no matter what obstacles you put in the way, you know, it can find new ways to jump and climb and
不管你在中间设多少障碍，它都能找出新的跳法、新的爬法，
[31:14] 主持人
Okay.
好吧。
[31:14] Rich Sutton
and do lots of things.
还能干很多别的事。
[31:16] Rich Sutton
Calculate trajectories pretty well. Animals are pretty good at understanding the physical world without the mental calculations that we think we are doing when we think about launching ourselves into space.
弹道也算得挺准。动物理解物理世界的能力相当强，而且完全用不着我们以为自己在做的那套心算——比如我们琢磨着要腾空跃出去的时候，脑子里以为在跑的那一堆计算。
[31:29] Khurram Javed
Breaking a fall, they can do it in real time in the right way to prevent injuries.
还有摔落时的缓冲，它们能实时做出正确的动作，把伤害避掉。
[31:33] 主持人
Okay, fair enough.
好吧，这点我认。
[31:34] Rich Sutton
I think it's just a question of degree between and I like to think that animals, other animals, are are very close to humans. I think it's hubristic to try to emphasize what we do differently, you know, how we're different from animals. It's better to see the commonalities. And I I think we are just a question of degree. It's degree and of course society and culture give us big advantages. Language give us big advantages.
我觉得这只是程度上的差别。我倾向于认为，其他动物跟人类其实非常接近。老想强调我们跟动物有什么不同，我觉得那有点自大，不如去看两边共通的地方。差距只是程度问题。当然，社会和文化给了我们巨大的优势，语言也给了我们巨大的优势。
[32:00] 主持人
Can I just push on some of this?
我能就这点追问一下吗？
[32:02] Rich Sutton
Yeah, good.
可以，来。
[32:02] 主持人
Because I want to back up Sonia. So, I believe animals and children learn from experience and do incredible things learning from experience. And when my son was two or three or four, I'm like, "Wow, this is really interesting that they're my son can learn these things without nobody really teaching him how to do these things." But at the same time, what Sonia's saying is like what makes human uniquely human, to be able to go to outer space, build a rocket. Those Those not things that are learned 100% from experience because before you launch the rocket, you actually have to abstract thinking through it in a way that is not learned from {quote} {unquote} experience. Because you don't know if it's going to work or not. You have to imagine it. How do you we teach a machine to imagine things that were not available before? That's probably the thing that we're trying to like push on because that we're not quite understanding that.
因为我想帮 Sonia 补一刀。我确实相信动物和小孩是从经验里学的，而且能学出很不可思议的东西。我儿子两三岁、四岁的时候，我就在想：太有意思了，没人真教过他，他自己就把这些学会了。但另一方面，Sonia 想说的是：让人之所以为人的那些东西——能上太空、能造火箭——不可能 100% 从经验里学来。因为在火箭发射之前，你得先在脑子里抽象地把它推演通，那种推演不是所谓「从经验中学」来的，因为你根本不知道它会不会成，你只能去想象。那我们要怎么教一台机器去想象一件从来没出现过的东西？这大概就是我们最想追问的点，因为这一块我们还没真的想明白。
[33:04] Rich Sutton
actually going to agree with you there. You have to be able to plan. You have to be able to imagine.
这一点我其实同意你。你必须能做规划，必须能想象。
[33:09]
Yeah.
嗯。
[33:10] Khurram Javed
Would you say that humans 1,000 years ago, before they had done all most of the thing that we're talking about, were they as intelligent? Um if for example someone from that era was exposed to this new culture, would they be able to get the same skills and and start doing useful things?
那你觉得，一千年前的人类——在我们刚才聊的这些事大多还没发生的时候——他们的智能水平是一样的吗？比如把那个年代的人放进今天这套文化里，他能不能学会同样的技能、开始做出有用的东西？
[33:27] 主持人
Even over the last 10,000 years, I don't think the human brain has evolved that much. Because
哪怕再往前推一万年，我也不觉得人脑进化了多少。因为——
[33:31] Rich Sutton
Fundamentally the same machine.
本质上还是同一台机器。
[33:33] 主持人
Fundamentally the same machine, but we've built up 10,000 years of knowledge.
本质上还是同一台机器，但我们额外积累了一万年的知识。
[33:37] Rich Sutton
Yes.
是。
[33:37] 主持人
And I get to learn 10,000 years of knowledge by going to school through supervised learning.
而我可以靠上学、靠 supervised learning（监督学习）把这一万年的知识拿到手。
[33:42] Rich Sutton
Right.
对。
[33:42] 主持人
And I get all that much, much faster than trying to learn through experience.
而且比自己从经验里一点点摸索要快得多得多。
[33:47] Rich Sutton
Right.
对。
[33:48] Khurram Javed
So I think you're like totally right. So we we would want our systems to learn from experience and part of their experience would be getting exposed to our culture and then learning from about our culture. They should learn from that. That's all good. But let's talk about when someone goes and does a paradigm shifting thing. So everyone gives the example of Einstein, but I think there are many examples. Learning is that too, like looking at learning thing versus programming thing. So when these paradigm shifts happen, I would say it's a human who has accumulated all this knowledge and then from their experience they're building new abstractions and they're planning with them and then they're discovering new knowledge. And that skill of of coming up with new abstractions and then learning what models and planning with them, that problem is that skill is totally missing in our current systems. And you can expose this at the edge of human knowledge, but you can also study this problem at the sensory motor stream level.
所以我觉得你说的完全对。我们当然希望系统能从经验中学习，而它经验的一部分，就是接触我们的文化、从文化里学到东西——这都没问题。但我们来聊聊有人做出范式转移（paradigm shift）的那种时刻。大家都爱举爱因斯坦的例子，但其实这样的例子很多，「学习」这件事本身就是一个——比如从「编程」转向「学习」。这些范式转移发生的时候，我会说：是一个人先积累了所有这些知识，然后从他自己的经验里构建出新的抽象，用这些抽象去做规划，进而发现新的知识。而「提出新抽象、学出模型并用模型规划」这项本事，恰恰是我们现在的系统里完全没有的。你可以跑到人类知识的最前沿去研究这个问题，但你同样可以在感知-运动流（sensory motor stream）这个层面上研究它。
[34:41] Rich Sutton
So, we're not arguing with the with the principle. We need to form abstractions so we can reason at a high level. You guys are coming close to doing that thing that I said we should never do, which is argue is prior knowledge important or gaining knowledge important. You know, that's what you guys just said. You said It's a you're you're still going to have to learn things. And you're saying, "Oh, I can get things from my culture and from prior knowledge." But these should not fight for each other.
我们并不是在质疑这个原则本身——我们确实需要形成抽象，才能在高层面上做推理。但你们俩现在越来越接近我说过绝对不该做的那件事了，就是去争论「先验知识重要，还是获取知识重要」。你们刚才说的就是这个：一个说「你终归还是得自己去学」，另一个说「我可以从文化里、从先验知识里拿到东西」。可这两件事不该互相打架。
[35:10] 主持人
On the exact thing around paradigm shifts, how do we create a machine that understands when to shift the paradigm?
就说范式转移这件事——我们怎么才能造出一台知道「什么时候该换范式」的机器？
[35:16] Rich Sutton
Yeah, I think through its experience, right? So, it would have to through its own experience. It can't rely on human knowledge because we're assuming the humans see one paradigm and we want a different way of looking at things. And so, through its experience, it has to find something that is better. Maybe it it generalizes better and makes better predictions. Maybe it's better in some other ways, but it has to be through its own experience.
嗯，我觉得只能靠它自己的经验。它没法依赖人类知识，因为我们的前提就是：人类看到的是一个范式，而我们想要一种不一样的看问题方式。所以它必须通过自己的经验，找到某种更好的东西。也许是泛化得更好、预测得更准，也许好在别的地方，但一定得来自它自己的经验。
[35:40] Rich Sutton
The big challenge that we don't see in our field, the ability we don't see in our field yet, is the ability to learn a model and then plan with the model. We can do the the math things and we can do AlphaGo because the games, we know the model. We know how the moves work. And in math, we know what the operators are. We you know, we know lean will take us from one state of knowledge to the to the state of the proof to the next state. But if we have to learn the models, there are no I'm I'm going to say it. It's probably maybe a a weird example, but a counterexample, but I can see that there's no instances of learning the model and then planning with the model in our field.
我们这个领域现在还看不到的一个大难题、一项还不具备的能力，就是：先学出一个模型，再用这个模型去做规划。数学那类事情我们能做，AlphaGo 我们也能做，因为那些游戏的模型是已知的——我们知道每一步棋怎么走。数学里我们知道算子是什么，知道 Lean 会把我们从一个知识状态带到证明的下一个状态。但如果模型得靠自己学出来……我就直说了，可能会冒出个奇怪的反例，但据我所知，我们这个领域里还没有「先学模型、再用模型规划」的先例。
[36:23] Khurram Javed
At least not with uh uh, like self-discovered abstractions. So, there are people who say, "I'm just going to learn a model of what happens in the next second or next millisecond." But, that's not how our models work. Our models are more abstract. Our models are, uh, quite different.
至少在「自己发现的抽象」这个意义上是没有的。确实有人说「我就学一个模型，预测下一秒、下一毫秒会发生什么」。但我们人的模型不是这么运作的，我们的模型要抽象得多，完全是另一回事。
[36:42] 主持人
So, one of the things I like about what you're doing here is you're not just sitting around pontificating or lamenting the state of the world as it is. You're very action-oriented. It's why you started a company. So, let's let's start talking about that a bit. Uh, in 2022, Rich, you laid out a very specific 12-point plan, the Alberta plan for AI research. Maybe tell us about that.
我很欣赏你们的一点是：你们没有坐在那儿空谈大道理，也不是光哀叹世界现在这个样子，你们非常行动导向——所以才去开了公司。那我们聊聊这个。Rich，2022 年你提出过一份非常具体的 12 点计划，也就是 AI 研究的 Alberta 计划（Alberta Plan）。能不能讲讲它？
[37:04] Rich Sutton
So, the Alberta plan came about because we just have general ideas, but we also needed to convert them into smaller chunks. And so, the 12 steps are the attempt to uh, crystallize particular chunks. There's a very important early step, step two, uh, which is continual deep learning. And And we think that one is like almost the most important because it unlocks everything else. If you could do continual deep learning, you could then continually update your model of the world. And then, if you knew how to do the abstraction rights in like the second half of of the the steps are all about how to get the abstractions right. So, and not only my abstractions right, what I mean by I don't mean get the right abstractions cuz no one can say what the right abstractions are. That depends on the world that you're in. Your agent would have to learn the correct abstractions for whatever world it's in. And so, you know, if you maybe those are the two key things. You have to find the right abstractions, and then you have to do able to continual deep learning.
Alberta 计划的由来是：我们手上只有一些笼统的大方向，但需要把它们拆成更小的块。这 12 步就是想把一个个具体的块结晶出来。其中很靠前的一步非常重要，第二步：continual deep learning（持续深度学习）。我们认为这一步几乎是最重要的，因为它能解锁后面所有的东西。如果你能做到 continual deep learning，你就能持续更新自己对世界的模型。而如果你还知道怎么把抽象做对——后半程那几步几乎全在讲怎么把抽象做对……我说「把抽象做对」，不是指找到「正确的抽象」，因为没人说得清什么才是正确的抽象，那取决于你身处什么样的世界。你的 agent 必须自己学会适合它所处那个世界的抽象。所以，关键大概就是这两件事：找到对的抽象，以及能做 continual deep learning。
[38:10] Khurram Javed
I think that a lot of the people in the field realize that we need models, we need to plan with them. But, the abstractions tell us what the model should be conditioned on. So, what should you What should the model predict? What are you going to do and then something is going to happen. And more importantly, where would that come from? So, I really like the example of elite athletes. If you ask elite athletes about how they do certain things, they would have weird niche terminologies for doing very specific things. They were like, you know, I do this thing and they would have a name for it. If they communicate, sometimes they don't even have a name for it if they're just doing it alone. So, how did they come up with those abstractions? That's in some sense a crucial thing that's missing that the later half of Alberta plan answers.
我觉得领域里很多人都意识到我们需要模型、需要用模型做规划。但抽象决定的是：模型该以什么为条件？模型该预测什么？你要做什么动作，然后会发生什么？更重要的是，这些东西从哪儿来？我特别喜欢顶尖运动员这个例子。你去问顶尖运动员某个动作是怎么做出来的，他们会冒出一堆奇怪的小众术语，专门指代某个非常具体的东西，说「我会来这么一下」，还给它起了个名字。有时候如果只是自己练、不用跟人交流，他们连名字都懒得起。那这些抽象到底是怎么来的？某种意义上，这正是现在缺的关键一环，也是 Alberta 计划后半部分要回答的问题。
[38:53] 主持人
Can we talk about the continual deep learning part? Is it an algorithmic gap that exists today or is it a just a practical deployment infrastructure data privacy gap? Because if I wanted to do call it naive updating of weights based on user interaction, I can do that today, right? And so, what in your opinion is the biggest thing that we're missing to kind of get to continual deep learning?
能展开讲讲 continual deep learning 这部分吗？今天卡住我们的到底是算法上的缺口，还是只是部署基础设施、数据隐私这类工程现实问题？因为如果我想做那种最朴素的、根据用户交互直接更新权重的做法，今天就能做，对吧？所以在你们看来，离 continual deep learning 最欠缺的那一块是什么？
[39:18] Khurram Javed
Yeah, so it's absolutely an algorithmic gap. You You can do the naive thing, but then you'll see all sorts of problems. So, for example, if you say um I'm going to take one sample and then I'm going to update my whole model with that one sample. You will run into this problem that now all of the previous knowledge in the model it's impacted negatively. And the way currently we we get around this is exactly what cursor does. They don't use one example. They use a large batch coming from a lot of users. So, in use cases where you can have that, you can do continual learning. But, most use cases you don't have that. Most use cases you have a single stream of data. And then if you apply it to the naive thing, it just completely destroys your prior knowledge in a very um destructive way.
对，这绝对是算法上的缺口。朴素做法你确实能做，但马上会撞上各种问题。比如说，我拿一个样本，就用这一个样本去更新整个模型——你会发现模型里原有的所有知识都被负面影响了。现在大家绕开这个问题的办法，正是 Cursor 在做的事：他们不用单个样本，而是用来自大量用户的一个大 batch。所以在能凑出这种 batch 的场景里，你是可以做 continual learning 的。但大多数场景没有这个条件，大多数场景你面对的是一条单一的数据流。这时候你再上朴素做法，它会以一种极具破坏性的方式，把你原有的知识彻底毁掉。
[40:00] 主持人
Catastrophic forgetting.
灾难性遗忘（catastrophic forgetting）。
[40:02] Khurram Javed
Yeah.
对。
[40:02]
That is the Yeah.
那就是——对。
[40:04] Rich Sutton
But, it's totally curable. You have to [laughter] have the right algorithm.
但这完全是能治好的。你只需要 [laughter] 用对算法。
[40:07] 主持人
cure?
治好？
[40:07] Rich Sutton
Well, yeah, exactly.
对，没错。
[40:09] 主持人
What is the cure?
那药方是什么？
[40:09] Rich Sutton
Well, you know, first you need to do what we call step size optimization. And it means every weight in your network has to have a separate step size. So, some will move fast, some will move slow. And you will we you will have to metalearn the step sizes for each weight. Most of your network will be have have weights that have tiny step sizes. So, then when you train on a new example, they don't get destroyed. Happens just to the right places. And then secondly, you have to use some form of generate and test. Um which is in feature space. So, you come up with new features or new units and and without following gradients. Cuz gradients are very slow process. You only move in a direction if you know it's the helpful one. And that's always going to be very slow and doesn't give you a path to grow more and more complex and and to have sustained learning. You need to have something that just proposes a bunch of new units. And and and then goes from there. I guess so, there is a specific thing I can say that make it at least concrete, which is to say we have this algorithm called continual backprop. We used published in in nature a couple years ago. And it it is exactly like backprop, but every but you also plant new seeds of units that are newly initialized with random weights.
首先，你得做我们所说的 step size optimization（步长优化）。意思是网络里每一个权重都要有自己独立的步长——有的走得快，有的走得慢。而且这些步长还得靠 meta-learning（元学习）学出来。你网络里绝大部分权重的步长会非常小，这样当你在新样本上训练时，它们不会被破坏，变化只发生在该发生的地方。第二，你还得用上某种形式的 generate and test（生成与检验），而且是在特征空间里做：你不断提出新的特征、新的单元，并且不靠 gradient（梯度）。因为梯度是个很慢的过程——只有当你已经确定某个方向有用，你才往那儿挪一点。这注定很慢，也没法让你长出越来越复杂的结构、维持长期不断的学习。你需要有个东西，能直接抛出一批新单元，然后从那儿往下走。我可以举一个具体的例子，至少让这事落地一点：我们有个算法叫 continual backprop（持续反向传播），几年前发表在 Nature 上。它跟普通 backprop 几乎一模一样，只不过你还会不断地种下新的「种子」单元，这些单元用随机权重重新初始化。
[41:37] Rich Sutton
Backprop only has random weights at the beginning of time. And then as you go on, all that randomness all that variety from the randomness gets used up. And with continual backprop, we keep injecting a bit of randomness, a bit of generate and test, a bit of generate and then the the operation of backprop is the tester. So, you need you need that. And and if you put those together really well, I think you'll have a new generation of massively superior continual deep learning. And that's what we hope to do in the next couple years.
普通 backprop 只在最开始有随机权重，随着训练推进，那些随机性、那份由随机性带来的多样性会被一点点耗光。而 continual backprop 会持续注入一点随机性、一点 generate and test——不断 generate（生成）新单元，而 backprop 的运作本身就充当了 test（检验）那一半。这个东西你必须要有。如果把这几件事拼得足够好，我认为你会得到新一代、性能远超今天的 continual deep learning。这就是我们希望在未来这两三年里做成的事。
[42:10] 主持人
Wonderful. Do you think that these algorithms can be applied to the current state of affairs with people scaling LLMs and trying to get them to do continual learning without catastrophic forgetting?
太好了。你们觉得这些算法能用到眼下的局面里吗？就是现在大家拼命 scale LLM（大语言模型），又想让它做 continual learning、还不能出现灾难性遗忘。
[42:22] Khurram Javed
Yeah, absolutely. I think it's So, I don't think that you could take an existing model and say I'm going to just start updating it with these algorithms because these algorithms meta learn how to learn. So, really you have to say, I'm going to learn from scratch. So, let's say I learn a new foundation model, but I'm going to learn with this these new algorithms. These new algorithms in addition to learning the knowledge, they're also going to learn how to learn future things. So, they're learning two things at the same time. And then um then I think you would be able to learn new things without catastrophic forgetting.
对，完全可以。不过我不认为你能拿一个现成的模型，说「我从现在开始就用这些算法更新它」——因为这些算法是 meta-learn（元学习）「怎么学」的。所以你其实得说：我要从头开始学。比如我要训一个新的 foundation model（基础模型），但我用这套新算法来训。这些新算法在学知识的同时，也在学「以后该怎么学新东西」，等于同时在学两件事。这样一来，我觉得你就能在不发生灾难性遗忘的前提下持续学到新东西。
[42:57] 主持人
the most radical thing in that you're trying to do in your company in terms of from the current state of affairs to try to do these two things at the same time?
……相对于现在的主流做法，你们公司在尝试做的事情里，最激进的一点是什么？就是要同时做成这两件事吗？
[43:06] Rich Sutton
Most radical thing.
最激进的一点啊。
[43:07]
Is I think that's
我觉得那大概是——
[43:09] Rich Sutton
This goes back to I'm not crazy, everyone else is crazy.
这又绕回那句话了：不是我疯了，是别人都疯了。
[43:11]
[laughter]
[laughter]
[43:12]
Yeah.
对。
[43:13] Khurram Javed
That's perhaps not totally radical. There was a point in like 2016 to 2018 where a lot of people were exploring these ideas quite a bit. They were doing it in a much more limited setting. So, they would say, we have a distribution of problems and then in this specific case we'll do it whereas we want to do it from a single stream of experience. So, our method should be more generally applicable. So, I think many people have explored this, but no one has explored this in the general setting where the resulting algorithm would be applicable everywhere.
这大概算不上有多激进。2016 到 2018 年那阵子，有不少人相当深入地探索过这些想法，只不过他们是在一个受限得多的设定里做的。他们会说：我有一个问题分布，然后在这个特定场景下来做；而我们想做的，是从单一的一条经验流里做。所以我们的方法应该更具普适性。我觉得很多人探索过这个方向，但没有人在真正通用的设定下探索过——那种做出来的算法哪儿都能用的设定。
[43:43] 主持人
So, what would be the most radical thing that your company your new company is trying to do that other people are not doing?
那么，你们公司——你们这家新公司——在尝试做而别人都没在做的、最激进的事情，究竟是什么？
[43:49] Rich Sutton
What's the most ambitious thing? Remember, I don't think I'm weird, so I don't want to say it's radical.
最有野心的事是什么？先说好，我不觉得自己有多离经叛道，所以我不想用「激进」这个词。
[43:56] 主持人
the most ambitious
那就说最有野心的。
[43:56] Rich Sutton
ambitious thing, I think is to try to have the full spectrum of knowledge both about the tiny things and about the big things. You know, like thinking about how you take an airplane from one city to another. That's a a very big thing. You know, it's it's more it's like your your space flight example, but it's just kind of more common sensical to think about cuz we all many of us take airplanes and any all of us use abstractions on all kinds of our life. And even even the squirrels use abstractions. So, to have that spectrum of of knowledge uh from the small to the big and to treat it in a uniform way and to be able to help have it self uh maintaining. You know, the big question is always you have your knowledge-based system and what keeps the knowledge in it correct? Well, what keeps the knowledge correct in a large language model is well, people did a lot of post-training and and they they they made it sure it was correct. And then they freeze it after that. So, that's what keeps it correct. But really our minds, we are we're always changing things and yet something keeps it organized and coherent and and and settling back into a good place rather than drifting off into crazy land.
最有野心的事，我觉得是拥有一整个光谱的知识——既有极细微的东西，也有极宏大的东西。比如你想「怎么坐飞机从一座城市到另一座城市」，这是件很宏观的事，有点像你刚才举的太空飞行的例子，但更贴近日常，因为我们很多人都坐飞机，而且我们在生活的方方面面都在用抽象（abstraction）。连松鼠都会用抽象。所以，要拿到从小到大这一整个光谱的知识，用统一的方式去处理它，还要让它能自我维护。你知道，一个绕不开的大问题永远是：你有一个基于知识的系统，那是什么在保证里面的知识是对的？在大语言模型（LLM）里，保证知识正确的办法是——人做了大量 post-training（后训练），确认它是对的，然后就把它冻结住。是「冻结」在保证它正确。但我们的心智完全不是这样，我们一直在改东西，可总有某种机制让它保持条理、保持自洽，跑偏了还能沉回一个好的状态，而不是一路飘进胡言乱语的世界。
[45:16] Rich Sutton
That is an I think our our biggest ambition to have a mind that is self-consistent and and can keep training itself and making it coherent.
我觉得这就是我们最大的野心：造出一个自洽的心智，它能不断训练自己，并且始终保持连贯。
[45:26] 主持人
that. Can I ask? It almost seems that it's it's such a ambitious vision and the idea that all these things can be unified into a single mind is so ambitious.
我能追问一句吗？这个愿景听上去野心实在太大了——把所有这些东西统一进同一个心智里，这个设想本身就野心勃勃。
[45:37] Rich Sutton
It's within reach. I think it's within reach. It's here it's 2026 and our computers are so fast. You know, is it is it so ambitious that it's out of reach? Uh or or do we have already inklings of how all the steps can be done? And we I I think we have a vision and inklings. Um I don't think it's I don't think it's inappropriate.
但它是够得着的。我认为够得着。现在已经是 2026 年了，我们的计算机这么快。问题在于：这份野心是不是大到根本够不着？还是说，每一步该怎么走，我们其实已经有眉目了？我觉得我们既有愿景，也有眉目。我不认为这个目标不合时宜。
[46:02] 主持人
Your vision involves a trillion parameter model with 20 watts. That seems pretty ambitious.
你们的愿景里有一条是：一个万亿参数的模型，只跑 20 瓦功耗。这听着相当有野心。
[46:10] Khurram Javed
That is ambitious. Uh, in some sense with current technology, I would say it's also impossible. Like just storing a trillion parameters in memory would probably use more than 20 watts of energy with current memory technologies, but we are really thinking of okay, things are getting better, computation is getting cheaper or it is getting more energy efficient. So, where would be would be in 5 to 10 years? And I think 5 to 10 years with the right algorithms and we can totally be in a world where this would be possible.
确实有野心。某种意义上，用今天的技术我会说这根本做不到——光是把一万亿参数存进内存，按现有的存储技术，耗电就不止 20 瓦。但我们真正在想的是：技术一直在变好，算力越来越便宜、能效越来越高，那 5 到 10 年后会是什么局面？我觉得 5 到 10 年后，只要算法对路，我们完全可能进入一个这件事可行的世界。
[46:43] Rich Sutton
So, 5 to 10 years is two orders of magnitude of Moore's law. It's a standard improvement. If we double every 18 months, 10 years would give you two orders of magnitude. And so, for Kurzweil's statement to be plausible, then today you should be able to do it for for what? 20 watts? Two orders of magnitude?
那 5 到 10 年正好是摩尔定律的两个数量级，属于常规进步。如果每 18 个月翻一倍，10 年就是两个数量级。所以，要让 Kurzweil 那个说法站得住脚，今天你就应该能用多少瓦做到？20 瓦往上两个数量级？
[47:08] Khurram Javed
2,000
2000。
[47:08] Rich Sutton
2,000 watts. If you can do it with 2,000 watts today, yeah, then in 10 years you'll be able to do it for 20 watts.
2000 瓦。如果今天你能用 2000 瓦做到，那对，10 年后你就能用 20 瓦做到。
[47:14] 主持人
You think you can do it for a 2,000 watts? You have I think lots of people at research labs that have access to way more than that.
你觉得你们今天能用 2000 瓦做到？可是各家研究机构里那么多人，手上能用的功率远远不止这个数啊。
[47:21] Khurram Javed
Yeah, I think we can can be more efficient than that even now with the right algorithm.
是，我觉得只要算法对，哪怕在今天我们也能比这更省。
[47:28] 主持人
If we can be more efficient than that, then why aren't we? It's not like people just want to spend all their money on spend all their money.
如果真能更省，那为什么没人这么干？又不是说大家就是一心想把钱全烧光。
[47:34] Rich Sutton
Sometimes it seems like they
有时候看起来他们还真——
[47:36] 主持人
I'll pay money.
我出钱。
[47:38] Rich Sutton
Sometimes it [laughter] seems like they want to.
有时候[laughter]看起来他们还真就想烧钱。
[47:39] 主持人
Yeah, I I
是啊，我、我——
[47:40] Rich Sutton
Doesn't it? I think that's how they show they're they're real men by using lots of energy.
不是吗？我觉得他们就是靠烧掉海量的电，来证明自己是真汉子。
[47:46] Khurram Javed
At least when I look at different research groups, I don't even see anyone believing in that it's possible. And I think if you don't believe in it, you're just not going to work on the technical problems and work through them.
至少我去看各个研究团队，我压根没见到有人相信这件事是可能的。而如果你根本不信，你就不会去啃那些技术难题，也不会一个个把它们磨穿。
[47:56] 主持人
Is it that it's not possible or it's that there's so much waste in the system? Like one which one is it? Like is it is is there someone who knows how to do it efficiently?
那到底是做不到，还是这个体系里浪费太多？是哪一种？有没有人其实已经知道怎么高效地做？
[48:05] Khurram Javed
Yeah.
对。
[48:06] 主持人
And then there's 10 times the number of people in the same lab doing all these other things. And so nine out of 10 people are wasting
然后同一个实验室里还有十倍于他们的人在做别的事。所以十个人里有九个是在做无用功——
[48:16] Khurram Javed
In some sense I the way I think about it is that we are stuck in a local minimum. So, if we want to move towards these new kind of algorithms, it is almost impossible that things will not get worse before they get better. So, when we start exploring these new directions, you're not going to get state of the art performance from day one, but it is because it is a different paradigm. Um but that path leads to similar performance at a higher energy scale. And these big labs, they are so locked into a product that they like it is not possible for them to pursue a path where things get worse first.
某种意义上，我的理解是：我们卡在了一个局部最小值（local minimum）里。所以如果要转向这类新算法，几乎不可能绕开「先变差、再变好」这一段。你刚开始探索这些新方向时，第一天不可能做出 state of the art 的性能，因为它是另一套范式。但那条路走下去，能在能耗量级完全不同的前提下达到同样的性能。而那些大厂实验室已经被自家产品死死绑住了，他们根本没法去走一条「先变差」的路。
[48:49] 主持人
Because their current paradigm allows them to keep scaling and this new paradigm they have to take a bet. And then
因为现在的范式让他们可以一路往上堆规模，而新范式意味着他们得押一注。而且——
[48:55] Khurram Javed
And they have to figure out some some technical things that are difficult that we have thought about it for many years. We know people who have thought about these things for many years and when I talk to them, it makes sense that it's doable, but you need to think about those challenges for a long period of time.
而且他们还得解决一批很难的技术问题，而这些问题我们已经想了很多年。我们也认识一些琢磨了很多年的人，跟他们聊下来会觉得这事确实可行，但前提是你得在这些难题上耗很长时间去想。
[49:13] 主持人
So, if everything goes right with Oak, what happens with the company? What do you what kind of company are you building?
那如果 Oak Labs 一切顺利，这家公司会走到哪一步？你们在建的是一家什么样的公司？
[49:20] Rich Sutton
Uh if everything goes right, we uh implement the architecture, we can have a genuine uh continual learning and we can form abstractions so that we can do planning and reasoning and and we have so sort of like true intelligence. And then, you know, it's hard to imagine just exactly what will happen by then. But I think
如果一切顺利，我们把这套架构实现出来，就能做到真正的 continual learning（持续学习），也能形成抽象，从而进行规划和推理——某种意义上就是真正的智能。到那时候究竟会发生什么，其实很难具体想象。但我觉得——
[49:41] 主持人
Humans will become irrelevant.
人类会变得无足轻重。
[49:43] Rich Sutton
I I don't think that's true at all. I I I
我完全不这么认为。我、我——
[49:46] Rich Sutton
We don't either.
我们也不知道。
[49:47] Rich Sutton
I think the world becomes exciting and even more exciting and interesting and and for humans. But in particular, I think there are the You have to wonder about the large language models. They might be at risk. Uh when this eventually happens, you know, I'm sure they'll get a good run. They've already had a good run. You know, they've been very successful. And let me say, just for for clarity that uh large language models are an amazing scientific breakthrough, a breakthrough in the skillful use of language by neural networks, wholly unanticipated. You know, it was a It was always a hold out for uh symbolic methods in language, and they They have totally changed how that's thought about now. Yeah. It's a It's a big breakthrough. It's so It's frustrating to me that we have to you know, just celebrate that we've made this great progress in the sub subset of the problem of AI, and enjoy that. Instead, it has to pretend to be all of AI. All of intelligence is not fluid, capable use of language. There's so much more. It's an important part. You know, it's like 20% or a quarter of intelligence. There's There's more.
我觉得那个世界会变得很精彩，甚至更精彩、更有意思，对人类来说也是。但具体到某些东西上——你不得不替大语言模型（LLM）捏把汗，它们可能是有风险的。等这一天真的到来的时候……当然，我相信它们还能风光一阵，而且它们已经风光过了，非常成功。为了不引起误会我先说清楚：大语言模型是一项了不起的科学突破，是神经网络在娴熟运用语言这件事上的突破，完全出乎所有人的意料。语言原本一直是符号方法（symbolic methods）最后守住的阵地，而大语言模型彻底改变了今天大家对这件事的看法。是的，这是个大突破。让我难受的是：我们本可以就这么庆祝一下——我们在 AI 这个大问题的一个子集上取得了巨大进展，享受这份成果就够了。可它偏偏非要假装自己就是 AI 的全部。智能并不等于流畅、娴熟地使用语言，智能远不止这些。语言是重要的一部分，大概占智能的 20%，或者说四分之一，但还有更多东西在后面。
[50:59] 主持人
Yeah.
嗯。
[51:00] Rich Sutton
We're not done.
我们还没做完呢。
[51:01] 主持人
Yeah. If everything goes right, are you imagining that there's a single minds that can do everything from learn how to swing from tree branches to make a spaceship, uh to you know, all these various things we've talked about today. Is it a single mind, and is it a single set of weights that can that can do all these things, or is it
是。如果一切顺利，你设想的是有一个单一的心智（mind），既能学会在树枝之间荡来荡去，又能造出飞船，还能做我们今天聊到的这各种各样的事？它是一个单一的心智吗？是同一套权重（weights）就能干成所有这些事，还是说……
[51:22] Rich Sutton
It's a single design.
是同一套设计。
[51:23] 主持人
Okay.
好。
[51:24] Rich Sutton
And there'll be many different There'll be many minds.
然后会有很多个不同的……会有很多个心智。
[51:26] 主持人
Mm. Okay. So, it's a single design that reacts to different environments.
嗯，明白了。所以是同一套设计，但会对不同的环境做出不同的反应。
[51:30] Khurram Javed
And it did it And different versions of that mind would learn different things because they have different experience. This sort of goes back to the big world hypothesis that there are infinitely many things to learn, so one system cannot learn infinitely many things. And like I think Rich already mentioned this, but if you have two of these systems, if you have two of the largest systems in the world, then it is trivial that they cannot model each other because they are equally complex. So, single system would never be able to get to a point where it can learn everything. It would always be multiple systems that are learning from their own experience.
对，而且同一个心智的不同版本会学到不同的东西，因为它们的经验不一样。这其实又回到了「大世界假设」（big world hypothesis）：可学的东西有无穷多，所以单个系统不可能把无穷多的东西都学完。而且我记得 Rich 刚才也提到过，如果你有两个这样的系统，比如世界上最大的那两个系统，那么显然它们没办法互相建模，因为它们复杂度是相当的。所以单个系统永远走不到「什么都学得会」那个点上。永远都会是多个系统，各自从自己的经验里学习。
[52:03] 主持人
Guys, are you hiring? What kind of people are you looking for?
两位现在招人吗？你们想找什么样的人？
[52:06] Khurram Javed
We are hiring. The initial team, most of it we already have in our mind. So, these are people who have thought about these ideas in the past in many uh you know, for a long time. And we are going to take a slightly different approach because this is a different paradigm. It doesn't make sense to become large very quickly because uh in some sense, everyone we hire has to uh come to see what we see. And not everyone sees that. So, we're going to start small, slowly grow to maybe a handful or two or three uh and then go from there.
我们在招。最初这支团队，大部分人选我们心里其实已经有数了——都是过去很长时间里一直在琢磨这些问题的人。我们会采取一种稍微不一样的打法，因为这是一个不同的范式（paradigm）。很快把团队做大是没有意义的，因为某种意义上，我们招的每一个人都得能看见我们看见的东西，而不是每个人都看得见。所以我们会从很小的规模起步，慢慢长到也许一小撮人，或者两三撮，然后再往下走。
[52:40] Rich Sutton
We want to be super aligned.
我们希望大家高度同频。
[52:42] Khurram Javed
We want to be super aligned.
对，我们希望大家高度同频。
[52:43] Khurram Javed
So, that we can be um very productive working together and and scaling the progress.
这样我们一起干活才能特别高效，把进展推起来。
[52:50] Rich Sutton
Absolutely.
完全同意。
[52:51] 主持人
Very, very cool.
非常非常棒。
[52:52] 主持人
Wonderful. I love this conversation. Thank you for taking the time to share what you're up to. You are, you know, an an unusually deep thinker about where reinforcement learning and algorithmic design will go. And it was a true pleasure to get to explore it together with you today. So, thank you.
太好了。我很喜欢今天这场对话。谢谢两位抽时间来分享你们正在做的事。你在强化学习（RL）和算法设计的走向上，是极少有的想得这么深的人，今天能和你们一起把这些东西展开聊一遍，真的很享受。所以，谢谢。
[53:10] Rich Sutton
Thank you very much. Thank you. It's our pleasure.
非常感谢。谢谢你。是我们的荣幸。
[53:15]
[music] [music] [music]
[music] [music] [music]