Memory and Continual Learning: Engram's Dan Biderman and Jessy Lin

频道: Sequoia Capital
视频: https://www.youtube.com/watch?v=aiR7F4jqjXY
原文语言: en
统计: 共 100 轮 · Jessy 20 · 主持 42 · Engram 3 · Dan 32

[0:00] Jessy

What about pre-training or even post-training makes it possible for the models to generalize in these magical emergent ways and controlling that process so that a company has a set of private data? How do we make the models [music] learn that just as well as the models know like the capital of France or you know like how to write Python? Um so I think it it's a really fun problem to think about.

到底是 pre-training、还是 post-training 里的什么东西，让模型能以这些近乎神奇的涌现方式去泛化？而我们又该如何掌控这个过程，让一家公司哪怕只有一批私有数据，也能让模型把它学得像「法国的首都是哪里」「怎么写 Python」一样牢？所以我觉得，这是个特别好玩的问题，值得琢磨。

[0:32]

[music] [music]

（音乐）（音乐）

[0:41] 主持

Welcome to training data. We are delighted to have Don Beerman and Jesse Lynn, co-founders of Engram today. Engram is a neolab focused on memory and continual learning and two of the hottest topics in all of AI research today. Okay. And Sean and I are delighted to dig in on those topics with you today.

欢迎来到《Training Data》。我们今天非常高兴请到 Engram 的两位联合创始人——Dan Biderman 和 Jessy Lin。Engram 是一家专注于 memory（记忆）和 continual learning（持续学习）的新实验室，而这两个正是当下整个 AI 研究界最热门的话题。我和 Sean 今天很期待跟你们一起深入聊聊这些。

[0:57] Engram

Awesome. Happy to be here.

太好了，很高兴来这儿。

[0:59] 主持

Great. So maybe to kick off, the engram website says, "We don't see the world through the lens of pre-training or post-training. Our models are always training. What does that mean?"

那我们就开个头吧。Engram 的官网上写着：「我们不从 pre-training 或 post-training 的视角看世界，我们的模型一直在训练。」这是什么意思？

[1:09] Jessy

So I think like models today obviously know a lot of things. They're incredibly smart. Um but we kind of think the bottleneck for making these models more useful these days is not really raw intelligence, but understanding like new and evolving context. So whether it's like you know a new task that you're doing or a particular context um for you know like a job or something like this um how do you bake that into the model weights the same way that you know pre-training and post- training bakes into the model weights very deeply and this is kind of why we think of ourselves as working on these fundamental problems of memory and continue learning which are really two sides of the same coin. How do you make the models learn new things um and bake them deeply into the weights of the model? And is your premise then that memory as a separate database or separate you know thing that you shove into the context window is not true memory and it's not true continual learning.

我觉得吧，今天的模型显然懂得很多东西，它们聪明得不得了。但我们认为，现在让这些模型变得更有用的瓶颈，其实并不是纯粹的智能，而是对新的、不断变化的 context 的理解。不管是你正在做的一个新任务，还是某份工作里特定的背景信息，你要怎么把这些东西刻进模型的权重里，刻得跟 pre-training、post-training 把知识深深刻进权重里一样深？这也是为什么我们觉得自己是在解决 memory 和 continual learning 这两个根本问题——它们其实是一枚硬币的两面：你怎么让模型学到新东西，并把它深深刻进模型的权重里？

那你们的前提是不是说：把 memory 当成一个独立的数据库、或者塞进 context window 的某个东西，这都算不上真正的 memory，也算不上真正的 continual learning？

[2:01] Jessy

I think all of these tools will kind of come together. So these days like the way that people are solving these problems is with context engineering. So you take like a huge prompt maybe you like keep talking to the model over many many turns and hours and um you know reorganize the context to better understand like what you're trying to do. And we think like these kinds of things like tool use, context engineering will play a part. But I think an underleveraged tool these days is using the same kind of training um pipeline or framework or kind of workflow that the frontier labs are using to make these models really good at frontier math or code but applying that to every kind of domain, every kind of context that you have like let's say in a company.

我觉得这些工具最终会融合到一起。现在大家解决这类问题靠的是 context engineering——你写一个超长的 prompt，可能跟模型来回聊很多很多轮、聊上好几个小时，然后不断重组 context，让它更好地理解你想干什么。我们认为像 tool use、context engineering 这些都会发挥作用。但我觉得现在有一个被严重低估的手段，就是：前沿实验室用来把模型练得特别擅长前沿数学或写代码的那一整套训练 pipeline、框架、工作流，其实可以套用到每一个领域、每一种 context 上——比如一家公司内部的各种场景。

[2:40] Dan

Yeah. And to me it's like as an individual um taking notes and having sticky notes is a very valuable thing. we should never discard this. But whenever we get back to business the next day, we always have some sort of trace of memory in our brain, some new intuition about how things should should be and where should we look. So these two things should come together. And current solutions are more kind of externalized memory. Um and this has two two issues. One is that the amount of tokens we will all collectively individually generate is going to be in the tens of millions of tokens per day soon. So just keeping it and searching through it is going to be and rereading it's going to be pretty expensive but it's gonna also be pretty hard pretty confusing for the models unless we have major major breakthroughs and how

对，而且在我看来，就像一个人做笔记、贴便利贴，这本身是很有价值的事，我们绝不该把它扔掉。但每次第二天回去工作时，我们脑子里其实都已经留下了某种记忆的痕迹，对事情该怎么做、该往哪儿看，有了某种新的直觉。所以这两样东西应该结合起来。而现在的方案更多是一种「外置记忆」，这有两个问题。第一，我们每个人、加在一起每天生成的 token 量，很快就会达到每天几千万个 token 的级别。光是把它存下来、再去检索、再去重读，成本就会很高；而且除非在「怎么……」这件事上有重大重大的突破，否则对模型来说也会相当吃力、相当容易犯晕。

[3:23] 主持

billions of tokens for Sean [laughter]

对 Sean 来说是几十亿个 token 啦（笑）。

[3:26] Engram

that's good depends on the day

哈哈这个看心情，得看是哪一天。

[3:29] 主持

could you maybe tell us a little bit about the engram architecture or the engram product and and how it works

能不能给我们稍微讲讲 Engram 的架构、或者说 Engram 这个产品，以及它是怎么运作的？

[3:35] Jessy

yeah I mean at a high level I think what we're trying to do is take any context like there's all these different um workspaces let's say Um so we're working with partners like notion and Microsoft and Harvey that have these places where people are doing a lot of work over a long period of time. There's all this context um both in terms of like you know documents that you've already written as a team as well as like now people are interacting with these agents more and more on these products um or having conversations giving them feedback um and figuring out how to have a model that deeply understands that context. So not just reading the files at test time, but really understanding it the way that an employee that's worked at your company for years has. So you kind of understand at a high level, oh, these are the initiatives across the company. Um, this is the way that we do things. Um, you've studied like how to run the hiring pipeline or how to, you know, do this kind of thing within the company. Um, and can operate just as well as, um, like any anybody else can in the company. And so what we're doing is training per team models within these workspaces that deeply understand those contexts and can improve with time on the things that people care about. Um so the way that we do this at like a technical level maybe is um training these into weights. So we do a lot of like adapter fine-tuning. So adapters of many types like I think people have looked into this for decades at this point like whether it's Lauras or prefixes or you know sparse architectures. I think like all of these tools are at our disposal. Um and then figuring out what the right data is. So how do you turn any kind of raw like document or interaction into useful training signal for the model? So again we have like a variety of tools now like supervised fine-tuning you know um RL um you know on policy distillation like all these things that you know the field has kind of developed um and trying to fit these pieces together into a model that learns continuously on the things that people care about.

好。从宏观上讲，我觉得我们想做的事情是：把任何 context 都接进来。现在有各种各样的工作空间，比如我们正在合作的伙伴有 Notion、微软、Harvey，这些地方都是人们长期、大量工作的场所。那里积累了海量的 context——既包括团队过去写下的文档，也包括现在人们越来越多地在这些产品上跟 agent 互动、跟它对话、给它反馈。我们要想办法做出一个能深刻理解这些 context 的模型，不只是在 test time 去读读文件，而是像一个在你公司干了好几年的员工那样真正吃透它。这样它从宏观上就能明白：哦，公司里有这些重点项目；我们做事是这个风格；它研究过怎么跑招聘流程、或者公司内部某件事该怎么办；然后它干活能干得跟公司里任何其他人一样好。

所以我们做的事情，就是在这些工作空间里、为每个团队训练专属的模型，让它深刻理解那些 context，并且能随时间在大家真正在意的事情上不断变好。至于从技术层面具体怎么做，可能就是把这些东西训进权重里。我们做了很多 adapter fine-tuning，各种类型的 adapter——其实大家在这方面已经研究了几十年了，不管是 LoRA、prefix，还是稀疏架构，我觉得这些工具我们都可以随手取用。然后就是搞清楚什么才是对的数据——你怎么把任何一份原始文档、或一次互动，转化成对模型有用的训练信号？同样，现在我们手上有一堆工具，比如 supervised fine-tuning、RL、on-policy distillation，这些都是这个领域逐渐发展出来的，我们就是想把这些拼图拼到一起，做成一个能在大家真正在意的事情上持续学习的模型。

[5:31] Dan

Yeah. And it's not a bet that tools are not there like our models always work under the assumption that some knowledge is externalized. Some tools are always there. But what you need to do is you need to figure out and that's the hard task is what needs to be internalized and what can be externalized. And for even for stuff that's externalized many individuals and companies have their own bespoke tools and ways of doing things. Not everyone has the same uh you know bash CLI tools that you know the frontier models are training on and how to get the models to better understand your bespoke setup I think is its own interesting thing. H. And so is the premise then that my notion agent will be a custom agent that is Laura fine-tuned or you know it's some way with an adapter tuned so that it's constantly learning on new content that's added into my notion workspace. Is that the premise?

对，而且这并不是在赌「工具会消失」。我们的模型始终是在这样一个假设下运作的：总有一部分知识是外置的，总有一些工具是现成在那儿的。但你真正要做的、也是最难的活儿，是搞清楚什么必须内化、什么可以外置。而且哪怕是那些外置的东西，很多个人和公司都有自己定制的工具和做事方式——并不是每个人用的都是前沿模型训练时用的那套 bash CLI 工具，那么怎么让模型更好地理解你这套定制的环境，本身就是个很有意思的课题。

那么前提是不是说：我的 Notion agent 会是一个定制的 agent，用 LoRA 微调过、或者以某种 adapter 调过，让它能不断地在我 Notion 工作空间里新加进来的内容上学习？是这个前提吗？

[6:20] Dan

Yeah. And they're working with many models and they're the early users of all the Frontier models and they're probably going to keep doing that. Does this approach work on the frontier models or are the closed frontier models or

对。而且他们会同时用很多模型，他们是所有前沿模型的早期用户，估计以后也会一直这样。那这套方法在前沿模型上行得通吗？还是说那些闭源的前沿模型……

[6:30] Dan

we need we need white you know we need white box access to the weights right so you know we can partner with companies that have you know closed source weights and do this with them uh but it's easiest for us to do it with with open source uh models um but any model that's a transformer model uh we can do our thing to it

我们需要——我们需要对权重的白盒访问权，对吧。所以我们可以跟那些拥有闭源权重的公司合作，跟他们一起做这件事；但对我们来说，用开源模型来做是最省事的。不过只要是 transformer 模型，我们都能在它身上施展我们这套操作。

[6:50] 主持

and what's the trade-off then when when people are comparing the before and before and after using you is it that there no longer sending so much context. Um, and so the trade-off is like you burn more compute up front to learn your company's way of doing things into the weights and then you're sending less context to the model on every inference pass. Is that the rough

那当人们对比「用你们之前」和「用你们之后」时，权衡点是什么？是不是说他们不再需要发送那么多 context 了？所以这个权衡大概是：你在前期烧掉更多算力，把公司的做事方式学进权重里，然后之后每一次推理时发给模型的 context 就少多了。大致是这么个权衡吗？

[7:09] Dan

trade-off? That's that's one that's one thing. The fact that you don't have to research things and the fact and reread things and the fact that you don't have to write like monstrous system prompts that already that can give you, you know, two orders of magnitude reduction in token uh inference consumption. It's not like you know 50% or it's it can be 100x fewer tokens because many things especially things that relate to people and teams and organization and priorities these are things that you can't really find in one document unless like you really have it really regimented and document everything. Uh and these kinds of things the model can kind of implicitly learn by training on some of the data and answer you know within 100 tokens what what the best frontier models would consume 100 thousand tokens doing. So these kinds of examples are interesting and also the the quality you know there are tasks that are you know not supernatural for the current generation of the models and we we're we kind of think there's going to be consistently this gap of like three to six months ahead where there's certain things that are bespoke that people are just exploring the models are not fully great for them. The models will at some point be great for them but if you can autonomously learn in in a very lightweight way it will give value in that time in terms of capabilities. Why train on the workspace level versus the individual level for example?

这是其中之一。你不用反复去查资料、不用反复重读文件，也不用写那种巨无霸般的 system prompt——光这些，就能让你的 token 推理消耗减少两个数量级。不是说省个 50%，而是可能少用 100 倍的 token。因为很多东西，尤其是跟人、跟团队、跟组织、跟优先级有关的东西，你根本没法在某一份文档里找全——除非你真的把一切都规整得井井有条、事事都记录在案。而这类东西，模型其实可以通过在部分数据上训练，隐式地学会，然后用 100 个 token 以内就答出来；同样的问题，最好的前沿模型可能要烧掉 10 万个 token 才能搞定。所以这类例子很有意思。

还有就是质量层面：有些任务对当前这一代模型来说并不是「手到擒来」。我们大概觉得，会持续存在一个差不多三到六个月的领先窗口——总有一些定制化、大家刚开始探索的东西，模型还不太能完全胜任。模型迟早会擅长它们，但如果你能用一种非常轻量的方式自主学习，那在这段时间里，它就能在能力上带来价值。

那为什么是在工作空间这个层级训练，而不是比如在个人层级？

[8:22] Dan

Uh either is is fine for us. It's just easier to start with you know it teams of people have you know are more you know disciplined in how they collect context and in the amount of context they have over years and it's easy for us to start there but every person's computer and every person's phone one day is a useful you know target for our technologies and in fact it will be very interesting to to go there. We just think you know the big deposits of information are now in and teams of people collaborating in knowledge work.

对我们来说哪个都行。只是从团队入手更容易些——一群人组成的团队，在怎么收集 context 上更有纪律，多年下来积累的 context 量也更大，我们从这儿起步比较顺。但每个人的电脑、每个人的手机，总有一天都会成为我们技术有用的落地对象，而且真到那一步会非常有意思。我们只是觉得，眼下信息的大矿藏，都在那些协作做知识工作的人组成的团队里。

[8:51] 主持

Is it a feature or a bug that there is so much fact memorization basically built into large language models? And there's a school of thought that you know the me the models just wrote memorizing the fact that the capital of France is Paris is actually a bad thing

大语言模型里基本上内建了这么多对事实的死记硬背，这到底是个特性（feature）还是个 bug？有一派观点认为，模型死记硬背「法国的首都是巴黎」这个事实，其实是件坏事。

[9:05] 主持

and what we would prefer for the models to do is you know abstractly learn the concepts of countries and capital cities. Yeah.

我们更希望模型做的，是抽象地学会「国家」和「首都」这些概念。对。

[9:11] 主持

But not to memorize all these facts in the weights. And so I'm curious what you think about disentangling memorization um versus learning, how it's done in the models today and then how you're thinking of approaching it.

而不是把所有这些事实都背进权重里。所以我很好奇你们怎么看「把记忆和学习拆解开」这件事——它在今天的模型里是怎么实现的，以及你们打算怎么去处理它。

[9:24] Jessy

Yeah, I think it's a really interesting question. Like to some extent you kind of need to remember stuff in order to like compose them into more complex concepts. I think the thing that's kind of missing is figuring out what's important to remember. And I think even now when you think about like learning new knowledge, if you look at a lot of these academic benchmarks, it's like how can we learn very specific facts like you know the length of a bridge in this like African country and that's not something that you really want the models to devote capacity for and it's not something that we devote capacity to. Um, so I think if you look at human memory, I mean you can say a lot more about this, but like it's lossy. Um, because part of the feature of intelligence is compressing what's important and separating that from what's not important. Um, and so I think like you can't really separate fact learning from like non-factarning or skill learning as some people would like to think. um like if you take a model and like some people have done this with models where you like strip out you know like all the facts and just have it like the pure core or something like this it's very unnatural as a model it doesn't know basic things um and you kind of need need that but I think

对，我觉得这是个特别有意思的问题。某种程度上，你确实得记住一些东西，才能把它们组合成更复杂的概念。我觉得现在缺的那一块，是搞清楚什么才是重要到值得记住的。哪怕是现在，当你想到「学习新知识」这件事，你去看很多学术 benchmark，它考的是「我们能不能学会非常具体的事实」——比如某个非洲国家某座桥有多长——这根本不是你希望模型为之分配能力的东西，我们自己也不会把脑容量花在这上面。所以我觉得，你看人类的记忆——这方面你（指 Dan）可以讲得更多——它是有损的。因为智能的一部分特性，恰恰就是压缩出重要的东西、把它和不重要的东西分开。所以我觉得，你没法像有些人想的那样，真的把「学事实」和「学非事实」或「学技能」干净地切开。比如你拿一个模型——有人真这么干过——把里面所有的事实都剥掉，只留下那个纯粹的内核，结果它作为一个模型会非常不自然，连基本常识都不懂，而你其实是需要那些东西的。但我觉得……

[10:33] 主持

why do you need that like why can't you look up facts and then just have

为什么你需要那些呢？为什么不能事实都靠查、然后只……

[10:37] Jessy

I think if you look at like how the models think if you need to recall basic facts in order to like take the next step in your thinking you can't get very far u maybe that's like a high level intuition, but it's part of like the reason why we think training is really important.

我觉得你看模型是怎么思考的就知道了——如果你为了推进思考的下一步、必须先回想起一些基本事实，那你根本走不远。也许这是个比较宏观的直觉，但这正是我们为什么觉得训练真的很重要的部分原因。

[10:51] Jessy

In order to like think more and more complex and deep thoughts about things, you kind of need to internalize something so that you can compose them into more abstract concepts.

为了能对事情想得越来越复杂、越来越深，你多多少少得先内化一些东西，这样你才能把它们组合成更抽象的概念。

[11:00] Dan

Yeah. And there have been efforts before that were hard to scale to try and, you know, disentangle the two and pre-train the models in a way that's, you know, allows it to retrieve and search for things and not internalize them. It's just the recipe we know to hill climb on collectively right now is this you know fact pre-training step and I think the the magic of of or the the mystery of this approach is that you know traditionally in CS we would have you know databases as its own curriculum and we would have algorithms and the databases is like facts about the world and capitals of whatever store them query them there's also algorithms of how do you efficiently manipulate information and get some answers uh in in a sample efficient way. And I think the magic of deep learning is that these two things are now mushed together and we need all these smart people and thropic interpretability to try and and and break them apart. And I think a lot of what we're seeing now in the adoption of AI into the economy is that these things are gradually separating again where companies have their own context and they really handle them with care and engineer them with care and there's a generic model that's completely a stranger through these contexts and the the model is operating on them. But it for us it's clear that there needs to be a certain convergence uh at least with with some cadence uh where where the the facts and the stories and the details are are getting mixed into the model. It has disadvantages as well because if you you have to you know capitals of of of countries are you know they can change but it's not very frequent but there's many other facts are changing all the time and just uh imprinting them into weights is a is a challenging thing to do.

对。以前也有过一些尝试，想把这两者拆开、用一种让模型可以去检索、去查东西而不必内化的方式来 pre-train，只是那些方法很难规模化。我们集体当下知道、能够拿来「爬坡」的配方，就是这个「事实式 pre-training」步骤。我觉得这套方法的神奇——或者说神秘——之处在于：传统上在计算机科学里，数据库是自成一门课的，算法又是另一门；数据库就像是关于世界的事实——某地的首都之类，存起来、查询它；算法则是讲怎么高效地操纵信息、以一种 sample efficient 的方式得出答案。而深度学习的神奇之处，就在于这两样东西现在被糅到了一起，以至于我们需要 Anthropic 那帮搞 interpretability 的聪明人去试着把它们重新拆开。

而我觉得，我们现在在 AI 融入经济的过程中看到的很多现象，是这两样东西又在逐渐分开：公司有自己的 context，他们非常小心地对待它、精心地打磨它，而那个通用模型对这些 context 完全是个陌生人，模型只是在这些 context 上运作。但对我们来说很清楚，至少得有某种程度、某种节奏的「收敛」——让那些事实、那些故事、那些细节重新被揉进模型里。这么做也有坏处，因为你得……国家的首都确实可能变，但不太频繁；可还有很多别的事实是时时刻刻都在变的，光是把它们烙进权重，本身就是件很有挑战的事。

[12:35] 主持

I see. So you're saying it's a false dichotomy that's try to separate algorithms from databases here. Um what really matters is like how to distinguish what's important to remember. Yeah.

我懂了。所以你是说，在这儿想把算法和数据库切开，是个假二分法。真正重要的，是怎么去分辨什么是重要到值得记住的。对。

[12:45] 主持

Versus what's not important.

以及什么是不重要的。

[12:47] 主持

Exactly. And it's an open how we dream and are you guys taking any inspiration from that in in terms of ranking?

完全正确。而且这还是个悬而未决的问题。那「我们是怎么做梦的」这件事，你们有没有从中汲取什么灵感，比如用在排序上？

[12:53] Jessy

Very very loosely. I think just the idea that that's kind of a phase that's missing maybe where you take a context and you deeply internalize it. Right now it's like everything happens at test time. you look at the, you know, context that the user gives you and you do some like thinking on the fly. Um, but again, like you can't get very far. Um, or you can get so far maybe and like you make mistakes along the way. Like how do you digest that back into the model so that next time you do it, you do it the right way and make even more progress.

非常非常松散地有一点吧。我觉得主要是那个想法——就是现在好像缺了这么一个「阶段」：你拿到一段 context，然后深深地把它内化。现在的情况是一切都发生在 test time，你看着用户给你的 context，临场即兴地做点思考。但同样，你走不了多远；或者说你能走到某一步，然后一路上犯各种错。那你怎么把这些经验消化、再「喂」回模型里，让你下次再做的时候就能做对，从而取得更大的进展呢？

[13:21] Dan

Yeah. And and what are dreams? Dreams are pretty crazy things to to say we want to build an AI that's like our dreams sounds a little bit like a nut thing to do. Um, there's not a lot of coherence there. But what's interesting there is like what happens in our dreams we we we see things we talk to ourselves and we we experiment with the affordances of what can we do and can't we do in the world in social situations and in you know u any any it's heavily biased towards social uh stuff right so for us too with things we're building is you know you we give the models the time to then go back retreat from the actual interaction and experiment with its affordances what can it do in an environment what Can it what does it know? How fast can it can it you know handle these kind of tail extreme uh things that same ones that we dream about at night?

对。那梦到底是什么呢？梦是挺疯狂的东西，说我们想造一个像「我们的梦」一样的 AI，听上去有点离谱。梦里头没什么连贯性可言。但有意思的地方在于：我们梦里发生的事——我们看见各种画面、跟自己对话、去试探各种「我们在这个世界上、在社交情境里能做什么、不能做什么」的可能性边界。而且梦严重偏向社交方面的内容，对吧。所以对我们正在造的东西也一样：你给模型留出时间，让它从真实的互动中抽身退回来，去试探它自己的可能性边界——它在某个环境里能做什么、它知道些什么、它能多快地应付那些长尾的极端情况，也就是我们夜里会梦到的那一类情形。

[14:08] 主持

You guys come from academic backgrounds like what's a canonical example that motivates this problem or you know like or that's a win so far.

你们俩都是学术出身——有没有一个经典的例子，能说明是什么激励你们去做这个问题？或者说，目前为止有什么算是一个胜仗的例子？

[14:19] Dan

Yeah, I have one example maybe Jesse can give another one a hypothetical one. For example, imagine one of the AI labs, say OpenAI, has to win some math Olympiad in a week time from now. Would they construct a catalog of all the math textbooks and really have people annotate which chapters to get and which graphs to to see or will they actually collect this, synthesize some training data, launch a training job, see where it lands in five, six days, start evaluating it and stuff like that. So it's it's obvious for anyone who's trained models that there's superior way to integrate across the ideas and capabilities and it involves this kind of magic of training. Um and we are clear that this has to happen in those highstake domains of math and coding and cyber and stuff. We just think much of this magic can actually end up in in the hands of of many more people and in interesting ways

好，我先给一个例子，Jessy 待会儿可以再给一个假设性的。比如说，想象一家 AI 实验室——就说 OpenAI 吧——得在从现在起一周之内拿下某个数学奥林匹克竞赛。他们会去整理一份囊括所有数学教科书的目录、然后真让人去标注「该看哪些章节、该研究哪些图表」吗？还是说他们其实会去收集这些材料、合成一批训练数据、启动一个训练任务，等个五六天看它落在什么水平、然后开始评估，诸如此类？对任何训练过模型的人来说，显而易见有一种更高明的方式，可以把各种想法和能力整合到一起，而这种方式恰恰要用到「训练」这种魔法。我们很清楚，在数学、编程、网络安全这些高风险领域，这件事注定会发生。我们只是觉得，这种魔法其中很大一部分，其实可以以各种有意思的方式，落到多得多的人手里。

[15:09] 主持

like why isn't it just the fi the foundation model labs that own the end product here? you like how do you go between giants?

那为什么这里最终拥有终端产品的，不就是那些基础模型实验室呢？你们要怎么在这些巨头之间立足？

[15:18] Jessy

Yeah. So I think like the worldview that we have is a bit different from the frontier lab worldview where it's like we want one model that's bigger and bigger that's more and more intelligent across a variety of domains. Instead how we see it like we kind of imagine this world where everybody has their own model. A lot of the things that people want to learn are either private like things that will never see the light of day in a post-training data set or even conflicting like oh the way that I want to do the task is different from how another company or another individual wants to. And I think a lot of these things we're already seeing are um hard to train into the models with the same tools that we have used for like decades in machine learning which is like you have really clean supervision. you have like ground truth reward signals um and you like create a nice environment and you like train the model to like use the tools to better accomplish this like coding task and instead a lot of the things that actually happen out in the world are very ambiguous or like um it's hard to say like what makes something good. Um, and so I think a lot of these things are very specific to individuals and I think very kind of misaligned or not very aligned with how the Frontier Labs think about the whole training pipeline and what kind of models will exist in the longer term.

对。我觉得我们的世界观跟前沿实验室的世界观有点不一样。他们的世界观是：我们想要一个越来越大、在各个领域都越来越聪明的单一模型。而我们看待这件事的方式不同——我们设想的是这样一个世界：每个人都有属于自己的模型。人们想学的很多东西，要么是私有的——那种永远不会出现在 post-training 数据集里、不会见天日的东西；要么甚至是相互冲突的——比如「我想用来做这个任务的方式」就跟另一家公司、另一个人的方式不一样。

而我觉得，我们已经看到，很多这类东西很难用我们机器学习里沿用了几十年的那套工具训进模型里——那套工具靠的是非常干净的监督信号、有 ground truth 的奖励信号，你搭一个漂亮的环境、训练模型去用工具更好地完成某个比如编程任务。可现实世界里实际发生的很多事情都非常含糊，或者说很难讲清楚「到底什么样才算好」。所以我觉得，很多这类东西都极其因人而异，而且跟前沿实验室对整条训练 pipeline 的思路、以及他们设想长远来看会存在哪种模型，相当地不对路、或者说不太合拍。

[16:33] Dan

Yeah. And to add to it, I think you know what is the P zero for the Frontier Labs and some of you here are are pretty close with them. It's getting to AGI, getting this one generic model that's extremely capable in coding and math and then using it to to automate the economy or to solve really hard, you know, long-term problems in cryptography and defense or whatever. Um, and it's pretty clear what needs to happen to push this. You know, more pre-training, bigger models, more data, uh, more RL, more inference time compute, that kind of stuff. That's PZ. That's where the majority of of expenditure and talent goes. And definitely all of them are thinking about memory and all of them are thinking about continual learning. It's just more of a product kind of effort right now. Um we think um it deserve it deserves its own its own attention and we think breakthroughs need to happen there and and Demis and the Sequoia event about a month ago said pretty clearly we need new breakthroughs around these topics and obviously they're thinking about them. we're just focusing exclusively on this and we think uh certain things around incentives of where the data is uh and who owns the model are pretty interesting. Um so if you could learn from many humans or organizations at scale uh without necessarily sending someone work with them shoulder-to-shoulder uh that would be a pretty big unlock

对，再补充一点。我觉得，对前沿实验室来说，他们的 P0（最高优先级）是什么——在座有些人跟他们走得挺近——那就是奔向 AGI，造出这么一个在编程和数学上极其强大的单一通用模型，然后用它去自动化整个经济，或者去解决密码学、国防之类真正棘手的长期难题。而要把这件事往前推，该做什么也相当清楚：更多 pre-training、更大的模型、更多数据、更多 RL、更多推理时算力，诸如此类。那是他们的 P0，绝大部分的开销和人才都流向那里。

他们当然都在想 memory，也都在想 continual learning，只是现在那更多是一种偏产品层面的投入。我们认为，这件事配得上、也需要它自己专门的关注，而且我们相信这里必须发生一些突破。大约一个月前在红杉的活动上，Demis 也说得很清楚：我们需要在这些课题上有新的突破——显然他们也在想这些。我们只是把全部精力都聚焦在这上面。而且我们觉得，围绕「数据在谁那儿」「模型归谁所有」这些激励机制方面的问题，相当有意思。所以，如果你能大规模地从许许多多的人或组织那里学习，而不必非得派个人去跟他们肩并肩共事，那将会是一个相当大的解锁。

[17:49] Jessy

and maybe another point on that is like I think a lot of things need to look different in the world. So one is there needs to be new research breakthroughs. Two is new infrastructure for training like you know small models for everybody rather than like one big model one big run. Um and then the third I think is um a different way of kind of combining research and product. Um so right now I think like there's like researchers in these frontier labs. They kind of train the model. They throw it over the fence to the product team who then like prompts or context engineers like new product surfaces on top of the core models. Um, but in this world where the models are always training, I think the inputs that users provide are very intricately tied to what the models learn from, like what the training signal is. And so there needs to be a lot more of a kind of integrated loop between like research and product. And so like while we're focused on tackling a lot of the core research challenges and that's our background, I think we're also very focused on like how to deploy this as quickly as possible to like learn from actual feedback in the real world.

也许在这上面还有一点：我觉得这个世界上有不少东西得变得不一样。第一，得有新的研究突破。第二，得有新的基础设施来支持训练——比如为每个人训练小模型，而不是「一个大模型、一次大规模训练」。第三，我觉得是要有一种把研究和产品结合起来的全新方式。现在的情况是：前沿实验室里有一群研究员，他们把模型训出来，然后「越墙扔」给产品团队；产品团队再在核心模型之上去写 prompt、做 context engineering，搭出新的产品界面。但在这个「模型一直在训练」的世界里，我觉得用户提供的输入，跟模型从什么里学习、训练信号是什么，是极其紧密地绑在一起的。所以在研究和产品之间，需要有一个紧密得多的整合闭环。因此，虽然我们专注于攻克很多核心研究难题——那也是我们的老本行——但我觉得我们同样非常在意的，是怎么尽快把它部署出去，从真实世界里的实际反馈中学习。

[18:47] Jessy

What motivated you to work on this problem? I think like it's obviously like one of the grand challenges in AI. I think everybody's talking about it these days because like the models are so smart. What what else is left? You know, it's I think learning like at the edges like learning the remainders of what makes these models useful. Um it's not just about raw intelligence anymore. It's about like learning new things. Um and I think it also feels very fundamental because it kind of goes back to really understanding what makes the model so good. So right now the models kind of incidentally know a lot of things from pre-training and we don't really understand why. Um it's like the internet was just you know this gift granted to us where um there's like a diverse set of data that contains like all of these different examples of coding and like writing and all these other things and it just happened that way. And now to figure out how to crack this problem of continual learning, it's about figuring out what about pre-training or even post-training makes it possible for the models to generalize in these magical emergent ways and controlling that process so that you know a company has a set of private data. How do we make the models learn that just as well as the models know like the capital of France or you know like how to write Python? Um, so I think it it's a really fun problem to think about.

是什么促使你们去做这个问题的？我觉得这显然是 AI 里的宏大挑战之一。如今人人都在谈它，因为模型已经这么聪明了，还剩下什么呢？我觉得剩下的就是在「边缘地带」去学——去学那些让模型真正有用的「剩余部分」。现在已经不只是关乎纯粹的智能了，而是关乎学习新东西。而且我觉得它还有种特别根本的感觉，因为它把我们带回到一个真正去理解「模型到底凭什么这么强」的问题上。现在的模型，多少是无意之中从 pre-training 里懂了一大堆东西，而我们其实并不真正明白这是为什么。这就好像互联网是凭空赐给我们的一份礼物——里头有一套极其多元的数据，包含了编程、写作以及各种其他东西的海量例子，而它就这么恰好成了那个样子。如今，要想破解 continual learning 这个难题，关键就在于搞清楚：到底是 pre-training、乃至 post-training 里的什么，让模型能以这些神奇的涌现方式去泛化？以及怎么掌控这个过程，好让一家只有一批私有数据的公司，也能让模型把它学得像「法国的首都是哪里」「怎么写 Python」一样牢。所以我觉得，这真是个特别好玩、值得琢磨的问题。

[20:07] 主持

And Don, you came from the neuroscience world. Is that right?

Dan，你是从神经科学领域过来的，对吧？

[20:10] Dan

Yes. Yes. So I was initially interested in in questions around, you know, consciousness and the human condition and things like that.

对，没错。我最早感兴趣的是意识、人类境况这一类问题。

[20:17] 主持

Are the models conscious?

那这些模型有意识吗？

[20:19] Dan

I don't have any any uh advanced thoughts on this more than you would you would read. I don't think so. But it's important that smart people are thinking about it. I would say like I was interested in how humans think, how humans perceive. And as Amos Ferski, the Israeli psychologist, used to say like he's not interested in artificial intelligence. He's interested in natural stupidity. So I would say like I started kind of similarly trying to see how people and animals experience the world. Gradually, you know, my inclinations took me to the stats and AI domains and there I figured that so many of the same problems of memory and continual learning are really really urgent and the kind of solutions we have in the current systems are pretty far from what we have in biology. And I'm not one of these people who would say that the the machine should be like, you know, like the animal or the the human brain. I don't think so. There's many things computers can do better than us. But human memory has these like very different uh things in it. It's you know, if you want to store a whole code base or you you can use a computer, you don't even need AI on the computer to store everything losslessly and just get it. But the human brain evolved to work in these constraints of of, you know, information capacity and to have these fuzzy representations that can then, you know, be abstracted and form connections and informed the next day. Current systems don't really have that beyond the generic pre-training step. And I was really interested in, you know, what are ways to to to build that in what are ways to learn from that.

这方面我没有比你读到的更高深的见解。我觉得没有，但重要的是有聪明人在认真思考这件事。我想说的是，我感兴趣的是人是怎么思考、怎么感知的。以色列心理学家 Amos Tversky 以前常说，他对人工智能（artificial intelligence）不感兴趣，他感兴趣的是「天然的愚蠢」（natural stupidity）。所以我一开始也类似，想搞清楚人和动物是怎么体验这个世界的。慢慢地，我的兴趣把我带到了统计学和 AI 领域，在那儿我发现，记忆（memory）和持续学习（continual learning）的很多同类问题其实非常非常紧迫，而我们现有系统里的那套解法，跟生物学里的机制相比还差得很远。我并不是那种认为机器就应该像动物或人脑一样的人，我不这么看。有很多事计算机做得比我们好。但人类记忆里有些很不一样的东西。比如你想存一整套代码库，你用计算机就行，甚至都不需要 AI，就能无损地把所有东西存下来再取出来。而人脑是在信息容量这些约束下进化出来的，它形成的是一种模糊的表征，这些表征之后可以被抽象、建立连接，再指导你第二天的行为。现有系统除了那个通用的预训练（pre-training）步骤之外，基本没有这种能力。我很感兴趣的就是，有哪些办法能把这种能力构建进去，又有哪些办法能从中学习。

[21:44] 主持

This is more of a philosophical question. You know, you mentioned in the brain how there's a bunch of different real estate there's different co-processing units, whatever modern computer architecture, there's CPUs, GPUs, you know, memory, there's different co-processors. Um, with like the bitter lesson, do you think that what's happening is that like LLMs are, you know, converge to say like one co-processor that's just totally dominant. it's like everything all compute is going to happen in you know the GPU equivalent of like a language model or do you think that these models are kind of building a bunch of co-processors like you know emergently inside the model like like you know and take with memory like do you think that the models themselves will just build you know whatever part of the brain equivalent would be that's good at memory or or do you think there needs to be like another standalone architecture that

这更像是个哲学层面的问题。你刚提到大脑里有很多不同的「地皮」，有不同的协处理单元——就像现代计算机架构里有 CPU、GPU、内存，还有各种协处理器。那么结合「苦涩的教训」（bitter lesson），你觉得现在发生的事情是：LLM 会收敛成某一个完全占主导的协处理器，所有计算都发生在那个相当于语言模型的「GPU」里？还是说这些模型其实是在内部涌现式地（emergently）长出一堆协处理器？拿记忆来说，你觉得模型本身会自己长出某个相当于「擅长记忆的那块脑区」的部分，还是说需要另外一个独立的架构来——

[22:47] 主持

yeah like is is memory an emergent property almost versus like

对，就是说，记忆几乎是一种涌现属性（emergent property），还是说——

[22:51] 主持

and almost everything like is everything that we need in intelligence will just be emergent with better training data and more scaled compute

几乎所有东西都是——智能里我们需要的一切，是不是只要有更好的训练数据、更大规模的算力，就会自然涌现出来？

[22:58] Dan

yeah I would say just on a more like superficial perspective on the current deployment of AI it's way more than than just GPUs and we're seeing all these you know sandboxes exploding and models operating on other computers trying things

我先从一个比较表层的角度说。看现在 AI 的实际部署，远远不止 GPU 这么简单。我们看到各种 sandbox（沙盒）大量涌现，模型在别的计算机上运行、各种去尝试。

[23:11] 主持

I'm more mean on the model architecture level rather than on the

我更想问的是模型架构这一层，而不是——

[23:15] Dan

so other experiment there there have been many previous experiments different architectures that we contributed to uh like the state space family and others to try and handle very very long context more efficiently. Uh the thing with all these methods, it ends up being a trade-off, usually a trade-off between memory and accuracy. And memory not in the behavioral cognitive sense, memory and the computer sense, right? Instead of having, you know, the the memory footprint of the transformer attention which is quadratic in the sequence length. Uh these models have

那就说别的实验。之前有过很多实验、很多不同的架构，我们自己也有贡献，比如 state space（状态空间）那一族还有其他架构，都是想更高效地处理超长的 context。这些方法的问题在于，最后都变成一种权衡，通常是 memory（内存）和准确率之间的权衡。这里的 memory 不是行为认知意义上的记忆，而是计算机意义上的内存，对吧？Transformer 的注意力机制其内存占用是随序列长度二次方增长的，而这些模型——

[23:44] 主持

some are claiming, you know, they have sub quadratic.

有些人声称他们做到了次二次方（sub-quadratic）。

[23:47] Dan

Yeah. Some claim and some do have it, right? And some of the the best Chinese model have layers that are you know inspired by those state space architectures and are you know not quadratic in cost thing is is that in in in our hands we find that you you always compromise accuracy for this memory there's no free lunch uh and what we're saying is like look if you're really bitterless and pilled what you want to do is you want to think how can I burn more compute and how can I burn it on you know new context that I have not seen before. So we're as bitter or less impilled as anyone else. Uh and we are not betting that the overall direction of AGI is is going to you know end anywhere soon. We just think there's more compute to scale and if I truly want to understand Shawn and Sean's work and Sean's context just like rereading files is not going to make it especially for a special person like you.

对，有人是声称，也确实有人做到了。一些最好的中国模型就有那种受 state space 架构启发的层，成本不是随序列二次方增长的。但问题是，在我们自己的实践里，我们发现你总是要拿准确率去换这个内存，没有免费的午餐。我们想说的是：你要是真的被「苦涩的教训」洗了脑，那你真正该想的是——我怎么烧掉更多算力，而且怎么把它烧在我以前没见过的新 context 上。所以我们跟任何人一样被「苦涩的教训」洗脑。我们也不赌 AGI 的整体方向会很快走到头。我们只是觉得还有更多算力可以扩展。如果我真想理解 Sean 还有他的工作、他的上下文，光是反复重读文件是不够的——尤其是对你这么特别的一个人来说。

[24:36] 主持

We got to derogatory.

都开始损人了。

[24:38] Dan

We got to train [laughter] 100 trillion parameter for this guy.

我们得为这位老兄训一个 100 万亿参数的模型才行（笑）。

[24:42]

Y

是啊。

[24:42] 主持

cosine cosine. Um, what are you finding that people care most about their models learning? Like is it memorizing facts about the organization? Is it remembering like ah no, we do CI this way? Is it like what what are people actually hoping to and then maybe this feeds into how you do do the ranking of memory slots and all that?

cosine（余弦）……嗯，你们发现大家最在意让自己的模型学会的是什么？是记住关于组织的事实？是记住「不不，我们 CI 是这么做的」这类东西？大家实际上最希望模型学会什么？而这可能也跟你们怎么给记忆槽（memory slot）排序之类的有关。

[25:02] Jessy

Yeah. Well, I think if you look at what people are spending their time in the app layer doing these days, it's a lot of just trying to make the model work well for your use case. like, oh, I want the model to like, you know, let's say like design my website with my brand style. Like that's like a, you know, very common example these days. But there's many kinds of different tasks that people do with agents. Um, like learning how to run a workflow, um, or, you know, kind of your particular way of like writing, let's say. Um, so there's many many kinds of things. And honestly like I think when we think about these methods kind of going back to this distinction between like facts and skills there really is none. Um I think the methods are kind of agnostic to that.

嗯，我觉得你看现在大家在应用层（app layer）上花时间做的事，很大一部分就是想方设法让模型在你的具体场景里好用。比如「我想让模型按我的品牌风格来设计我的网站」，这是现在很常见的一个例子。但人们用 agent 做的任务种类很多，比如学会怎么跑某个工作流，或者学会你写东西的某种特定风格。所以种类非常多。说实话，我觉得我们在思考这些方法时，回到「事实」和「技能」这个区分上，其实根本没有实质区别。我觉得这些方法对这种区分是无所谓的、不挑的。

[25:46] Dan

Yeah. To me it's like the the natural thing almost all the app layers are basically you know a frontier model wrapped in in a loop with search tools and stuff. And what they're all interested in doing with us is finding ways to kind of interface with their data in a way that's you know faster, more efficient and also is more contextual. So almost all of them it's like [snorts] we want to have our you know our firm knowledge you know be encoded in something that's more efficient that I don't have to research. We want to have the model know in a targeted way who's the person I should triage a thing to. And we're just showing them that with pretty lightweight training, these things can can be instinctual to the models. They don't have to have these very involved long uh ripple loops to to solve them. So it's in a sense it's like um you know it's it's a rag killer kind of um kind of thing. Again, we can always do rag and we can always retrieve, but that's the thing that people are interested in interfacing with very large data planes and automating very repetitive uh things this way.

对，在我看来这几乎是顺理成章的事——几乎所有应用层本质上都是一个前沿模型（frontier model）外面套了一个循环，配上搜索工具之类的东西。而他们想跟我们一起做的，全都是想找到一种跟自己数据打交道的方式，要更快、更高效，也更贴合上下文。所以几乎所有人都是这个诉求：「我们想把公司的知识编码进某种更高效的东西里，这样我就不用每次都去检索。」「我们想让模型有针对性地知道，这件事我该分派给谁。」而我们正是在向他们展示：只要用相当轻量的训练，这些东西就能变成模型的本能，不需要那种牵涉很多步、来回反复的长链路去解决。所以某种意义上，这是个「RAG 杀手」（rag killer）一类的东西。当然我们随时都还能做 RAG、随时都能检索，但大家真正感兴趣的，是想跟超大规模的数据面（data plane）打交道，并用这种方式把高度重复的事自动化掉。

[26:47] 主持

Yeah. And I want to double click on this rag killer thing. And I'm sorry to beat a dead horse. I just don't fully gro it yet.

嗯，我想就这个「RAG 杀手」的说法深挖一下。抱歉我有点抓着一个点不放，我只是还没完全弄明白。

[26:53] Dan

Yeah.

嗯。

[26:53] 主持

Um,

嗯，

[26:54] 主持

is the premise that there's some tradeoff between doing rag versus updating your model weights? Is it the idea that you should be doing both? Like what types of things should be done in the weights versus what types of things should be externalized to rag?

你们的前提是：做 RAG 和更新模型权重之间存在某种权衡吗？还是说你的意思是两者都该做？哪类东西应该放进权重里，哪类东西应该外置给 RAG？

[27:07] Jessy

I think it's a it's an unsolved problem. I don't think anyone has answered it. Um, we're all working on it. Um it's also the fundamental question of like biological memory. What should be internalized versus what not. Um I do think that things that are like you know do you need to internalize the room number in a hotel that you were in like a year ago? Probably no. Not in your neural tissue. Uh probably that's good to write down. But do you need to internalize maybe the password to your home right now? Probably it's useful for the next few years to have that imprinted somewhere. So yeah, how does this translate into like knowledge work and products? This is still something we figure out and we try to take the approach that we try to use as few uristics as possible. It's easy to run filters on the data and say like I'm going to keep this, discard that, train on this, train on that. But as humans, you know, we watch Tik Tok and we, you know, get exposed to a lot of garbage and still the brain is able to learn and not completely go off the rails and we think models should be the same as well. Yeah, maybe concretely in the short term, I think a lot of what people are worried about these days is the huge inference costs of running these agents like for days on end. Um,

我觉得这是个尚未解决的问题，我不认为有谁回答出来了，我们都还在攻这个问题。这其实也是生物记忆的根本问题：什么该被内化，什么不该。我确实觉得，有些东西——比如你一年前住过的某家酒店的房间号，你需要把它内化吗？大概不需要，不用进你的神经组织，那个写下来就行。但你现在家门的密码，需要内化吗？大概在接下来几年里，把它刻在某个地方是有用的。所以，这怎么落到知识工作和产品上，仍然是我们还在摸索的东西。我们采取的思路是：尽量少用启发式规则（heuristics）。在数据上跑过滤器很容易，比如「这个我留着、那个丢掉、这个拿来训练、那个也训练」。但作为人，我们刷 TikTok、接触到一大堆垃圾信息，大脑照样能学习、不会完全跑偏，我们认为模型也应该是一样的。对，也许具体到短期来看，我觉得现在很多人担心的是，让这些 agent 连着跑好几天带来的巨大推理（inference）成本。嗯，

[28:16] 主持

high inference costs a good thing. [laughter]

高推理成本是好事啊（笑）。

[28:18] Jessy

I mean, consuming tokens for what?

我是说，消耗 token 到底换来了什么？

[28:21] 主持

Sonia works with fireworks. She really loves [laughter]

Sonya 跟 Fireworks 有合作，她可太喜欢了（笑）。

[28:25] 主持

inference. We love inference, too.

推理。我们也很爱推理。

[28:28] Jessy

Yeah. So I think it's like in the short term I think that's the immediate pain point like why are you reading the same files over and over again you know even in the same query but like definitely you know across people in the same company they're running the same queries on the same documents over and over again and that should be something the model just knows like in the same way you ask an employee they don't you know type into the search box like what what was I working on yesterday they just know

对，所以我觉得短期来看，最直接的痛点就是：你为什么要一遍又一遍地读同样的文件？哪怕是在同一次查询里。而且很明确，同一家公司里不同的人，会在同样的文档上一遍遍跑同样的查询，这种东西应该是模型直接就知道的——就像你问一个员工，他不会往搜索框里打「我昨天在做什么来着」，他直接就知道。

[28:51] 主持

but doesn't caching kind of solve that

可是缓存（caching）不就在某种程度上解决这个了吗？

[28:53] Jessy

I think to some extent yeah but I think going back to this like question of what should be internalized versus what's um like something you retrieve at test time. I think again like a lot of it is about building on your knowledge. So if you are always doing rag, you can't make associations like oh you know I see somebody you know on the team is doing this kind of research and I kind of like recall at an abstract level. Oh there's this like related thing that you might want to know about. You didn't even ask about it right? But I think like these kinds of associations can only happen in weights because they're not really about you know you ask me to search for this I'm going to search for this.

我觉得某种程度上是的。但回到刚才那个问题——什么该内化，什么是你在测试时（test time）才去检索的。我觉得这里很大一部分是关于在你已有知识之上做积累。如果你永远只做 RAG，你就没法建立联想，比如「噢，我看到团队里某个人在做这类研究，我在一个抽象层面隐约想起来，还有个相关的东西你可能想了解一下」——你压根都没问过这件事，对吧？我觉得这种联想只可能发生在权重里，因为它本质上不是「你让我搜这个、我就去搜这个」那回事。

[29:31] Dan

Yeah. And also the I think the main limitation with retrieval systems in general and in AI specifically is like the problem is not so much what to store and where to put it. It's the problem is like how how to address it like how to query the thing. Do you know what to look for even? Yeah.

对。而且我觉得检索系统总体上、尤其是在 AI 里的主要局限在于：问题不在于存什么、放哪儿，问题在于怎么去寻址（address）它、怎么去查询这个东西。你甚至知道自己要找的是什么吗？对吧。

[29:46] Dan

And this is involves some sort of intuition that sometimes the models don't have. Interestingly enough they don't know where to look. uh and and especially if you're you know limited to the the current way of doing things which is keyword search that is just easier to scale in RL and least involved in terms of like infra for embeddings and stuff. So yeah, knowing what to search is something that's intuitive and can and can happen in the weights and also about caching and inference like much of this company started with us taking like a deep dive into like KV caches and caching and this is a a fascinating thing right KV cache is a monstrosity of the current uh way of doing things that you know think about it a KV cache for a single like Wikipedia article for some you know Taylor Swift or something like this it will be like 80 [snorts] gigabytes of HBM memory on the GPU and an entire llama it's it's for say a 70B llama model and the entire weights of the model would be about 100 gigabytes and you know with with some distortion they remember the entire internet um and how come this thing is so uh one thing is so bit efficient and this we have this proof of existence that gradient descent can pack a lot of information in very few numbers whereas this KV cache thing you take a few tens kilobytes of article and it becomes those 80 gigabytes of of brain state. So sure you can cach this, you can load this, you'll have issues with disk to HBM uh stuff people are working on it. It's pretty interesting. But what if we can take those 80 gigabytes, spend some compute offline, maybe also in fireworks and file, but then compress it and make it really really small so that the thing we load in cache is like a thousandx smaller that would have tremendous implications for how we load things, how fast we can do things, and what the fidelity of the representation is.

这就涉及某种直觉，而模型有时候并不具备。有意思的是，它们不知道该去哪儿找。尤其是当你被限定在现在这套做法里——也就是关键词搜索（keyword search），因为它在 RL 里更容易扩展、在向量嵌入（embeddings）那套基础设施上也最省事。所以，知道该搜什么是一种直觉性的东西，是可以、也应该发生在权重里的。再说说缓存和推理：这家公司很大程度上就是从我们深入研究 KV cache（键值缓存）开始的，这东西非常迷人，对吧？KV cache 简直是现有做法里的一头怪兽——你想想看，光是一篇维基百科文章，比如关于 Taylor Swift 的某篇，它的 KV cache 在 GPU 上就要占大约 80 GB 的 HBM 显存。而一整个 LLaMA 模型——比如 70B 的 LLaMA——它全部权重大约也就 100 GB，而且这 100 GB 多少有点失真地记住了整个互联网。这东西怎么能这么节省比特？这给了我们一个「存在性证明」：梯度下降（gradient descent）能把海量信息压进很少的几个数字里。反观这个 KV cache，你拿几十 KB 的文章，它就膨胀成那 80 GB 的「大脑状态」。当然你可以缓存它、可以把它加载进来，你会遇到从磁盘到 HBM 的那些问题，大家都在攻，挺有意思。但如果我们能把那 80 GB 拿来，离线花点算力——也许也借助 Fireworks 之类的——然后把它压缩到非常非常小，让我们加载进缓存的东西小上一千倍，那对我们「怎么加载、能多快地做事、以及表征的保真度（fidelity）」都会有巨大的影响。

[31:37] 主持

Super interesting.

太有意思了。

[31:38] 主持

Yeah. What are some of the things that could happen in the next year or two that would be like the chat GBT moment of memory? Or do you think that that's not how things will play out?

对。那在未来一两年里，有哪些事情可能会成为记忆领域的「ChatGPT 时刻」？还是说你觉得事情不会这么演变？

[31:49] Jessy

It's a good question. Um, I don't know. I think like the first proof of concept of the thing that people keep talking about with continual learning, which is you have an intern that you can teach things over time and it actually gets better. I think everybody's waiting to see that, you know, and no matter how sophisticated the context engineering approaches are these days, they're not getting there. So, I think you need, you know, all of these tools at your disposal to make that happen. Um, but I think it will be something like that where it's like the model's actually getting smarter like whoa, it it's different from yesterday.

好问题，我也说不准。我觉得第一个概念验证（proof of concept），应该就是大家一直在说的那个关于持续学习的画面：你有一个实习生，你可以随时间教它东西，而它真的会越来越强。我觉得大家都在等着看到这一幕，而不管现在的 context engineering（上下文工程）方法多精巧，它们都还做不到。所以我觉得你需要把所有这些工具都用上才能实现。但我觉得那一刻大概会是这样的：模型真的在变聪明，让你「哇」一下——它跟昨天不一样了。

[32:20] Dan

Yeah. And it's important to say that the Chad GPT model was not anticipated. we've just you know read about all the different product the product directions that certain people had before Chad GPA was different. Um I feel like to me the example is like look if you you know resigned from your job today and your sole mission was to make a model that's better for you and you would use open anthropic and all these frontier models and you just 247 engineer the context right skills your way to move the needle is very limited as an individual. you'll just be better off waiting for the next version of the model and and you'll take it from there. And we would like um to see a future where actually the more time you spend on the thing actually translates to the quality of performance at least in the things and domains you you care about. Um and this is pretty hard to achieve and the only reason it we we think it could be achieved is if you start scaling compute and training on these data without destroying them all importantly which is pretty hard. Just a couple like this is just for fun like rapid fire questions going off just memory.

对。还得说一句，ChatGPT 这个模式当初是没人预料到的。我们都读到过，在 ChatGPT 之前某些人设想的各种产品方向其实都不一样。在我看来这个例子是这样的：假设你今天从工作辞职，你唯一的使命就是打造一个对你而言更好的模型，你会用 OpenAI、Anthropic 这些前沿模型，你 24/7 地去做 context engineering、写 skills——可作为个人，你能撬动的幅度非常有限。你还不如直接等模型的下一个版本，再从那儿继续。而我们希望看到这样一个未来：你在一件事上花的时间越多，真的能转化成性能质量的提升，至少在你在意的那些事和领域里是这样。这相当难实现。我们之所以认为它有可能实现，唯一的理由是：你要开始扩展算力、在这些数据上训练，但关键是不能把这些数据全毁掉——而这非常难。来几个纯粹好玩的、快问快答的问题吧，全凭记忆来答。

[33:23] 主持

When's the last time you reached surprised about something in AI in any area

你最近一次对 AI 任何领域里的某件事感到惊讶，是什么时候？

[33:28] Dan

when reading about fundraising? [laughter]

读到融资消息的时候（笑）。

[33:31] Dan

A lot of surprises every day. I would say all of us felt you know a little bit of a change around the capabilities of the coding agents.

每天都有很多惊喜。我觉得我们所有人都隐约感觉到，coding agent 的能力出现了一点变化。

[33:38] 主持

That's true.

确实是这样。

[33:38] Dan

Uh but we we've been you know dabbling with these things and trying to make them work in in more effortful ways before so it didn't come as a complete surprise. Um but yeah, I think to me the main events were GitHub copilot. That for me was just the main event and chat GPT and then seeing the agentic stuff. We all anticipated I think and and different different people had different expectations on how far it can go and how long horizon it can go. But I feel yeah it's we're we're yet to see something fundamentally different and people are working on completely new ways of doing things now. Um but yeah, to me it's it's models actually changing in a way that's not harmful uh and learning new things uh on the fly that are, you know, personally and economically viable. That's interesting.

不过我们之前就一直在折腾这些东西，想方设法让它们更扎实地跑起来，所以这变化对我们来说也不算完全的意外。但要说对我冲击最大的，是 GitHub Copilot——那对我来说就是标志性事件，还有 ChatGPT，再后来看到 agentic 那一波东西。我觉得这些我们多少都预料到了，只是不同的人对它能走多远、能撑多长的时间跨度有不同的预期。但我感觉，我们还没真正看到什么根本性不同的东西，而现在大家都在探索全新的做事方式。对我来说，真正让我兴奋的，是模型本身以一种无害的方式发生改变，能即时学到新东西，而且这种学习在个人层面和经济层面都是可行的。这点挺有意思的。

[34:25] 主持

Right now there's this idea of like we're each going to have a token wallet that we're going to bring around to companies

现在有一种说法是，我们每个人都会随身带着一个 token 钱包，走到哪家公司都带着它，

[34:30] 主持

or to different apps um different workspaces. Do you think that we're gonna end up with like a memory bank, a memory wallet that we're going to move around

或者带到不同的 app、不同的 workspace 里。你们觉得我们最终会不会拥有一个类似 memory bank、memory wallet 的东西，随身带着它

[34:40] 主持

to across the digital world as we go?

在整个数字世界里走到哪带到哪？

[34:43] Jessy

I think it's an interesting question. I don't know if we've fully figured out what the right kind of like product form factor is in this sense. In a way, even with like chatbt memory, let's say, I kind of don't want it to remember across my like personal and work context. Oh, yeah. like it's like oh you know you might like these sheets because you trained a model on a GPU last week it's like that's totally irrelevant and to some extent it's like because the memory is flawed but also I think you do want memory in your I guess tools and the products that you use to be separated to have control over that. So I personally think like there needs to be some separation there but I guess to be determined what that might look like.

我觉得这是个很有意思的问题。我不确定我们是不是已经完全想清楚了在这件事上正确的产品形态该是什么样。某种程度上，就算是 ChatGPT 的 memory，我其实并不希望它把我的个人语境和工作语境串起来记。对啊，比如它会说「你可能会喜欢这套床单，因为你上周在 GPU 上训了个模型」，这完全是风马牛不相及——一方面是因为 memory 本身还有缺陷，但另一方面，我觉得你确实希望你用的工具和产品里的 memory 是分开的，希望自己能掌控这件事。所以我个人觉得，这里需要某种隔离，不过具体长什么样还有待观察。

[35:24] Dan

Yeah. And like I think a holy grail is like you go to work and you just burn through all these tokens and you create all this value and somehow you know all the IP and stuff stays with the company but somehow the skills you learned the things you invented your ways of doing things. Some of them you can take with you as well to your next job in a way that's you know sanitized and not you know harmful to any other company's IP. So I do think like carrying um a set of skills uh will be interesting. We do it in our biology right now and we just you know sign NDAs and have like ethical rules around it. But I think doing it in a digital world would be pretty interesting and pretty rewarding because it will force each of us to push the frontier and implement AI more deeply in our companies in our individual life and then be rewarded for it. I started a PhD in the SAS firm in 2007 at at Stanford and AI like AI was boring as hell at the time. was all statistical learning and there's basically two areas like computer vision and NLP. So like vision and and language were kind of the two areas and I think that's still true. In 2012 Alexand happened like vision was dominating for six years or whatever. Are you guys surprised that language seems to be like the language approach seems to be like dominating over vision in progress? Question two, do you think vision has any chance of coming back? How do you think about this?

对。我觉得一个终极理想是这样的：你去上班，烧掉一大堆 token，创造出大量价值，所有的 IP 之类的东西都留在公司，但你学到的技能、你发明的东西、你做事的方式，其中一部分你能带走，带到下一份工作里去——而且是以一种「脱敏」过的方式，不会损害任何其他公司的 IP。所以我确实觉得，随身携带一套技能会是件很有意思的事。我们现在在生物层面就是这么干的，无非是签 NDA、加上一些道德规范来约束。但我觉得，在数字世界里也能这么做会相当有意思、相当有价值，因为它会逼着我们每个人去推动前沿、在公司里、在个人生活里更深地落地 AI，然后因此得到回报。我 2007 年在斯坦福读统计的博士，当时 AI 无聊透顶，全是统计学习，基本上就两个领域——computer vision 和 NLP，也就是视觉和语言这两块，我觉得这点到今天依然成立。2012 年 AlexNet 出来，视觉统治了差不多六年。你们会不会觉得很意外——在进展上，语言这条路线好像正在压过视觉？第二个问题是，你们觉得视觉还有机会翻盘吗？你们怎么看这件事？

[36:48] Jessy

Yeah, I think it is pretty surprising to me. I mean, some people maybe saw it coming, but I think I've always kind of been interested in language um as like I don't know, I guess like a medium for communication and like so many kind of complex abstract things can be done in language. Um, I do think like, you know, I imagine like in the longer term, language and vision will kind of like combine in this more like unified system where, you know, we kind of like take in inputs from all of these different modalities and like understand them in this abstract way. But, um, yeah.

嗯，我觉得这对我来说确实挺意外的。当然有些人可能早就预见到了，但我一直对语言挺感兴趣的，怎么说呢，把它当成一种沟通的媒介，那么多复杂、抽象的事情都能在语言里完成。我确实觉得，往更长远看，我想象语言和视觉会融合进一个更统一的系统，我们从所有这些不同的模态里接收输入，然后以一种抽象的方式去理解它们。不过，嗯，就这样吧。

[37:23] Dan

Yeah. To me, like I've never been interested in language. It seemed to me such a such an advanced capability that you know is is very the the entire animal kingdom has very different forms of of speech and language than what we you know and how we communicate with ourselves and in writing. Uh [snorts] and I was always as many other leaders in AI had this thought that you know the natural thing is you have to experience the world act in it and vision and action that will be the the key. But then I've, you know, like anyone else seen the the Chad GPT moment and went to to do some work at Mosaic and stuff like that to learn how the sausage is made on the on the NLP side. And the thing that's striking is that like the the language should be pretty hard like each word has this uh one hot embedding vector that's as dissimilar to any other word uh than it is you know uh to you know it's it's a completely highdimensional space and it's really artificial in a sense and we we learn it with models that are order of magnitude bigger than the best vision models and still you know things work pretty well. I do think there's a lot of juice to be squeezed in in in an image and video and I think you guys doing good good investments in this space but it's I think the two would keep being interesting in different ways.

对。我反倒是从来没对语言感兴趣过。在我看来它是一种非常高级的能力，整个动物界拥有的言语和语言形态，跟我们人类自己交流、跟我们书面表达的方式差别极大。而且我跟 AI 圈很多领军人物一样，一直有这么个想法：最自然的路径应该是你得去体验世界、在其中行动，视觉加行动才是关键。但后来我跟所有人一样见证了 ChatGPT 时刻，还跑去 Mosaic 干了一阵活，去看看 NLP 这边的「香肠是怎么做出来的」。让人吃惊的是，语言按理说应该很难才对——每个词都是一个 one-hot embedding 向量，跟任何别的词都是最大限度地不相似，它是个彻头彻尾高维的空间，某种意义上非常人造；我们用比最好的视觉模型大一个数量级的模型去学它，结果居然还运转得相当好。我确实觉得图像和视频里还有大量「油水」可榨，我也觉得你们在这个领域的投资做得不错，但我想这两块会在不同维度上持续有意思下去。

[38:41] 主持

I'm going now tell you my that was my leadup. Now I'm going to tell you the crackpot theory. [laughter]

我现在要跟你们讲——刚才那都是铺垫。现在我要讲那个「民科理论」了。[笑]

[38:45] 主持

Um I like and this this podcast is not for me to pontificate. It's for you guys. But this is something I've been thinking a lot about and I just you're the right people to share this with. I I was pretty shocked that language kind of surpassed vision and I underestimated what was happening with LLMs in like 2018, 2019, 2020 because I just had this bias towards vision. And when I look back on it now, like I think what is basically happening is that in biology, like vision has a massive fundamental advantage over language in biology. And maybe I'm wrong, but basically like the bit rate that your brain can process optical data through the eye is, and this is my I'm not a biologist. This just kind of my dumb assessment seems many orders of magnitude greater. And there's a lot of like optical processing that happens like even before you reach you know like electrons and so just like the total bit rate that is of training data that's kind of being processed and then making it to your brain seems many or semanted greater than the audio data where you know it's sound waves where sound waves are fundamentally like much slower bit rate than light.

这个播客不是给我来高谈阔论的，是给你们俩的。但这是我一直在反复琢磨的事，而你们正好是最适合分享的人。语言居然超过了视觉，这让我相当震惊；2018、2019、2020 那阵子我严重低估了 LLM 正在发生的事，就因为我对视觉有偏见。现在回头看，我觉得本质上是这么回事：在生物层面，视觉对语言有一个巨大的、根本性的优势。也许我说错了，但基本上，你大脑透过眼睛能处理光学数据的比特率——这是我的、我不是生物学家、这只是我外行的瞎判断——好像要高出好几个数量级。而且在数据到达大脑之前，眼睛里就已经发生了大量光学处理，所以总的训练数据比特率，处理之后送进大脑的那部分，好像比音频数据高出好几个数量级——音频是声波，而声波从根本上说比特率比光慢得多。

[40:10] 主持

Yeah. And then there's almost like like an upscaling from the acoustics to electronics which make it into your brain where it's like there's like a downscaling from photons to electrons with vision. Whereas in computers today everything is electronic. So it's kind of like you nerfed vision and you like promoted language where the it's like all processing is on the same playing field. It's all electronic. And I just I think this might this is like my crazy ass dumb non-technical crackpot theory, but I think this might be part of why just like from an information theory perspective that like maybe language and vision are on a similar playing field by the time you get to like LLMs and then LLMs are we're just a really really smart architecture that's better suited for language than for vision. um how dumb does this sound especially to you Don the neuroscientist

对。然后从声学到电信号、再进到你大脑的过程里，几乎像是一次「升采样」，而视觉那边则像是从光子到电子的一次「降采样」。可是在今天的计算机里，一切都是电子的。所以这就像是你给视觉「减了配」、给语言「加了戏」，因为在计算机里所有处理都站在同一条起跑线上、全是电子的。我就在想，这可能——这是我那个疯癫的、外行的民科理论——可能部分解释了，纯从信息论的角度看，等到了 LLM 这个层级，也许语言和视觉就站在差不多的起跑线上了，而 LLM 又恰好是一种非常非常聪明的架构，比起视觉更适合语言。这听起来有多蠢？尤其是在你——Dan，神经科学家——听来。

[41:11] Dan

Jesse also has some background in cognitive computational science right

Jessy 在认知计算科学方面其实也有些背景，对吧？

[41:16] Dan

so I would say my my point here is like look much of what we're doing in knowledge work we haven't evolved to do right we're sitting on these computers reading these things writing these memos whatever we are not evolved to do this it's new to us our brains are not wired for this still nevertheless it's useful to have LMS to do this for us and you know as humans we're heavily vision biased you Other rodents are more alactory biased and I've worked on these things myself before. So what's the uh real estate in the brain that's allocated to vision and you know exhibital loes versus like language areas as a temporal lobe probably more vision. I'll have to check with check chat GPT but I think that's the situation

我想说的是，看，我们在知识工作里做的很多事，本来就不是我们进化来要做的——我们坐在电脑前读这些东西、写这些备忘录，不管是什么，我们根本没进化出做这些的本事，这对我们是全新的，我们的大脑还没为这个布好线。但即便如此，有 LLM 替我们干这些活还是很有用的。而作为人类，我们是重度偏向视觉的；其他啮齿动物更偏向嗅觉，这些我自己以前都研究过。所以问题就是：大脑里分配给视觉的「地盘」——比如枕叶——相比语言区——比如颞叶——到底有多大？大概视觉占得更多吧。我得回头跟 ChatGPT 核对一下，但我觉得情况差不多就是这样。

[41:56] 主持

you don't know from memory.

你凭记忆答不出来。

[41:57] Dan

No man I'm externalizing. I'm a big rag believer in my personal lifestyle [laughter] but um I think

答不出来，老兄，我把它外置了。在我个人的生活方式里，我是个坚定的 RAG 信徒。[笑] 不过我觉得

[42:04] Dan

in the limit we're all it's all rag. I internalize just u you know important things like um my emotions to you. No, just kidding. Sorry. [laughter] Um

归根到底，我们全都是 RAG、一切皆 RAG。我只内化真正重要的东西，比如说……我对你的感情。不，开玩笑的，抱歉。[笑] 嗯……

[42:15] Dan

anyways um yeah and vision is dominating when people are training vision language models they end up the vi language ends up dominating the vision content there. Um but yeah it's it's hard to say that because a certain brain is more you know biased towards a certain modality doesn't mean necessarily that we're going to more efficiently do it. I do think that efforts on like brain computer interfaces should take this into account. How do you then relay it back to the brain? That's where I think it's really important to think like what real estate do we have there right now? Um but for knowledge work, it's equally fine if it's text. I think

总之，对，还有关于视觉占主导这件事——当大家训练 vision-language 模型时，最后往往是语言压过了里面的视觉内容。但话说回来，很难就此下结论：某个大脑更偏向某种模态，并不一定意味着我们处理那种模态就一定更高效。我倒是觉得，做脑机接口的那些工作应该把这点考虑进去——你到底怎么把信息回传给大脑？这时候我觉得真正重要的是去想：我们现在在那儿到底有多少「地盘」可用？不过对知识工作来说，用文本完全没问题，我觉得。

[42:46] Jessy

last question if if everything goes right, what does the world look like in five 10 years and then what is engram's role in it? I think I'm imagining like a world where everyone has their own model um that is really different from the other person's model and from the frontier model and all of these kind of serve different purposes and to have a model that really you know I think people often talk about like knowing knowing you um but also like um kind of like helping you in the ways that make sense to you um personally um whether it's like an individual or a team. I think there's an element of like having different kinds of intelligence everywhere.

最后一个问题：如果一切顺利，五到十年后这个世界会是什么样，而 Engram 在其中又扮演什么角色？我在想象这样一个世界：每个人都有自己的模型，这个模型跟别人的模型、跟前沿模型都很不一样，而它们各自服务于不同的目的。拥有一个真正……我觉得大家常说的是「懂你」的模型，但同时也是那种以对你个人有意义的方式来帮你的模型——无论对象是个人还是团队。我觉得这里有一层意思是：让各种不同形态的智能无处不在。

[43:24] Dan

Yeah. And to me actually it's it's a variant of the story where like you know in neuroscience we know that memory and navigation are pretty closely related same circuits in the brain that you know represent landmarks in space are in charge of some you know elements of episodic memory and things like this and for me I think the company can be you know the actual LLM interface to the data plane for everyone. So sharing some similarities to great companies like you know data bricks and Oracle where you know we form these memories that happen to be neural memories with models that happen to be personalized and happens to be there's hundreds of millions of them but they're basically a neural interface to the data plane in a way that's that's very different from what we know and it's more efficient it's more associative it's not representing the file system as it is it's representing a brain state u of that file system so that's for me a vision

对。对我来说，这其实是另一个故事的变体：在神经科学里我们知道，记忆和导航关系相当紧密——大脑里那些表征空间地标的同一批神经回路，也掌管着情景记忆的某些环节，诸如此类。所以对我来说，我觉得这家公司可以成为所有人通往数据层（data plane）的那个真正的 LLM 接口。这跟 Databricks、Oracle 这些伟大的公司有些相似之处——我们形成的这些记忆恰好是「神经记忆」，用的恰好是个性化的模型，而且这样的模型恰好有上亿个，但它们本质上是一个通往数据层的神经接口，而且这种方式跟我们熟知的那套很不一样：它更高效、更具联想性，它不是照搬文件系统的原貌，而是表征出那个文件系统的一种「大脑状态」。这对我来说就是一个愿景。

[44:14] 主持

beautiful vision to end on thank you guys so much for coming by the share with building.

用这么美的愿景收尾真好。非常感谢你们俩过来分享、和我们一起共建。

[44:19] Engram

Awesome. Love it. Thank you guys. [music] [music] [music]

太棒了，太喜欢这次聊天了。谢谢你们。[音乐][音乐][音乐]

[44:46]

[music]

[音乐]