SOTA Generative Media Panel — Dumitru Erhan, Shane Gu & Nicole Brichtova, Google DeepMind

频道: AI Engineer
视频: https://www.youtube.com/watch?v=KLDdXOw6jIc
原文语言: en
统计: 共 213 轮 · 主持人 40 · Dumitru 24 · Shane 35 · Nicole 32

[0:12] 主持人

and welcome back for those on the stream and those those in person. um we take tend to basically take these longer sessions between uh all the sort of mainstage keynotes to reflect on things that um you know are particularly important but like don't have like a significant like sort of launch moments. Today we're very lucky to have people working on Omni and VO Nano Banana like the you know the world's best generative models here with us. Uh, Demetrio, I I I first saw you when you were posting about your office. [laughter] Um, I think you're you're probably number one uh Google Google's number one office influencer at least in in San Francisco. I think you like you like to bike as well. You like to take photos of

欢迎回来，不管是看直播的朋友还是现场的朋友。在主会场各场 keynote 之间，我们一般会安排这种时间更长的环节，聊一些特别重要、但又没有什么重磅发布节点的话题。今天我们非常幸运，请到了做 Gemini Omni、Veo 和 Nano Banana 的几位——这些可以说是全世界最好的生成式模型——来到现场。Dumitru，我第一次注意到你，是你在网上晒你们办公室的时候。（笑）我觉得你大概是 Google 的头号办公室网红，至少在旧金山是。我记得你也喜欢骑车，还喜欢拍

[0:56]

bike here.

这儿的自行车。

[0:57] 主持人

Yeah. Um, but you know, but also you work on video models.

对。不过呢，你同时也在做视频模型。

[1:01] Dumitru

That's right.

没错。

[1:02] 主持人

Um, Shane, I I met you I think at like a dinner.

Shane，我记得我们好像是在一次饭局上认识的。

[1:05] 主持人

Yeah. Um and uh and uh and I I remember you were trying to get me invested in like one of the companies. I forget forget which one.

对。嗯，而且我记得你当时还想拉我投资其中一家公司，具体是哪家我忘了。

[1:14] Shane

Forget about that. [laughter]

那事就别提了。（笑）

[1:17] 主持人

But now but now you're um now you're working on Omni Thinking. Um and and just you know a bunch of other

不过你现在在做 Omni 的 thinking，还有一大堆别的

[1:24] Shane

Gemini RL.

Gemini 的 RL。

[1:25] 主持人

Yeah. Yeah. Uh and Nicole also uh the rest of the gen media models uh nano banana and uh all and everything you just launched actually even this week. Uh,

对，对。还有 Nicole，你负责其余那些生成式媒体模型——Nano Banana，还有你们这周刚发布的所有东西。

[1:35] Nicole

yeah. We launched some APIs.

对，我们发布了一些 API。

[1:37]

Yeah. Yeah. Yeah.

对，对，对。

[1:38]

And I haven't tried to convince you to invest in anything, but maybe I should.

我倒还没劝过你投什么，不过也许我该试试。

[1:41] 主持人

I mean, so I try not to be an investor. People just convince me anyway. I'm like just, okay, well, I'm not that rich, but know like you can't not try to invest in some of these things. And, you know, for those of us who are not working at a Frontier Lab, this is the best this closest we'll ever get. Um, so yeah, actually, let's kind of recap since you're closest to it and we just did it, like what was launched this week? What should people go try out?

我的意思是，我其实尽量不去当投资人，但总有人能把我说动。我就想：好吧，我也没那么有钱，但有些东西你就是忍不住想投一点。而且对我们这些不在前沿实验室工作的人来说，这已经是离前沿最近的机会了。好，那我们来回顾一下吧——你离得最近，东西又是刚发出去的——这周到底发布了什么？大家应该去试试什么？

[2:03] Nicole

Yeah. Um so yesterday we had two launch moments. Uh one of them we launched NanoBanana 2 light uh which is our fastest, cheapest um image model in the nano banana model family. Um and it's better than the original NanoBanana. Um so really for most people um that model replaces what you you know used and love the original Nano Banana for across like generation and editing and it gets really close to the frontier quality of of the kind of mainland bigger models. So that that's really exciting. I think if you look at some of the demos or like things that people have been trying like getting kind of that like 3 second latency just unlocks a whole bunch of things that you can do with like ideation and iteration and it's just really fun and the model's getting to a point where like the quality is really good um where um it you know you can use it for iteration but you can also use some of those outputs as just kind of like ready um production output. So that's really exciting. Um and then second launch we finally um launched the Gemini Omni Flash APIs um that we pre-announced at IO. So thank you for waiting. Um and that you know is the first time that we're making the APIs available for developers and it's basically really exciting kind of video generation and editing and we're pricing it the same as Y31 fast. So we're getting you kind of like really really good quality for a really awesome price hopefully. Um

好。昨天我们有两项发布。第一项是 Nano Banana 2 Lite，它是 Nano Banana 模型家族里最快、最便宜的图像模型，而且比初代 Nano Banana 还要强。所以对大多数人来说，这个模型可以接替你们以前用得顺手、也很喜欢的初代 Nano Banana，生成和编辑都能胜任，而且质量已经非常接近那些主线大模型的前沿水准。这点真的很让人兴奋。你去看看一些 demo，或者大家这阵子在试的东西就知道，做到大约 3 秒的延迟，一下子就解锁了一大堆构思和快速迭代上的玩法，用起来特别好玩。而且模型的质量已经好到这个程度：你既可以拿它来迭代，也可以直接把其中一些输出当作能上生产的成品来用。这点也很让人兴奋。第二项发布是，我们终于上线了在 I/O 上预告过的 Gemini Omni Flash API，感谢大家耐心等待。这是我们第一次向开发者开放这些 API，核心就是非常令人兴奋的视频生成和视频编辑能力，定价和 Veo 3.1 Fast 一样。所以希望能让大家用非常划算的价格，拿到非常非常好的质量。

[3:22] 主持人

yeah, I mean that that's incredible. I'm actually really So when you guys launched Omni for the first time, you also did a podcast uh with Logan who couldn't be here today uh and you added like a sloth uh and and Ramen and all these all these things. I actually really want to do that to our videos. I just didn't have an API for it because obviously I have to automate the whole thing. So thank you for the API.

是啊，这真的太厉害了。我其实特别……是这样，你们第一次发布 Omni 的时候，还跟 Logan 一起录了期播客——可惜他今天没能来——你们在视频里加了一只树懒、Ramen，还有各种各样的东西。我其实特别想在我们自己的视频上也这么玩，只是一直没有 API 可用，毕竟整个流程我肯定得自动化。所以谢谢你们出了 API。

[3:41] Nicole

Uh that is my favorite use case. Everybody should do that. Um I got a cat which is probably like the most boring of the animals. Um if you don't know what we're talking about, you should look it up. It's very funny. Feurer. um Furer who's um you know on on the team did that.

这是我最喜欢的用例，大家都应该去试试。我分到的是一只猫，大概是所有动物里最无聊的那个。如果你还不知道我们在说什么，一定去搜来看看，特别好笑。是 fofr……嗯，是我们团队的 fofr 做的。

[3:54]

Furer is the number one guy you should follow for you should follow ideas on okay what can this thing do?

想知道“这东西到底能干什么”，fofr 就是头号必关注的人。

[4:00]

Yes.

对。

[4:00]

Right.

没错。

[4:01]

Yes. He he's he's amazing at that.

对，他在这方面真的特别厉害。

[4:03] 主持人

I've tried to get him for the last two years to come to AIE. He hasn't made it yet. He's actually come in person. He just didn't want to speak because he's anonymous.

过去两年我一直想请他来 AIE，到现在都没请成。其实他本人来过现场，只是不想上台讲，因为他是匿名的。

[4:11]

I know.

我知道。

[4:11]

I I want to say his real name but I can't say his real name.

我真想说出他的真名，但我不能说。

[4:13]

No [laughter] no we won't we won't do that to him. But you should really follow him. He's amazing.

别别，（笑）我们可不能这么对他。不过你们真的应该关注他，他太厉害了。

[4:17] 主持人

He did all that work. I actually met him uh in the office uh when we did the podcast I think and I didn't realize it was him. So his badge doesn't say Popers.

那些全是他做的。其实我好像是录播客那次在办公室见过他，当时完全没意识到就是他。毕竟他工牌上可不会写着 fofr。

[4:27]

Yeah,

对，

[4:27]

I know.

我知道。

[4:28] 主持人

So he used to be part of uh Replicate and Replicate had this joke where like everyone was Deep Fates. Deep Fates is this like kind of mysterious character and replicate. Replicate is very cool company and both was part of it. Um, so, okay, one thing I want to get on there before I go into like sort of the the the the sort of omniper is we added cats, we added sloths, very cool, very cute, very fun.

他以前在 Replicate 待过。Replicate 内部有个梗，说人人都是 deepfates——deepfates 是 Replicate 那边一个挺神秘的人物。Replicate 是家很酷的公司，他们俩都在那儿待过。好，在正式展开聊 Omni 之前，我想先聊一点：我们加了猫、加了树懒，很酷、很萌、很好玩。

[4:51] 主持人

Uh, what are the, you know, inspire people as to like what are the more sort of workhorse use cases that maybe are not just demos, you know?

那能不能给大家一些启发：有哪些更偏实际干活的主力用例，而不只是拿来做 demo 的？

[4:59] Nicole

Yeah. So, so obviously the hero capability of the model or maybe there's two like one is the ability to kind of take in anything as input and then get video on the other side. Obviously in the future and and we've kind of talked about this as a pre-announce like we want to get the other output modalities out as well but basically what that means is you know you can take a set of images that you have as maybe a storyboard. You can take like an audio track as a reference of you know like a voice that you want a character to speak and then you can get a video on the other side. So like that just unlocks a whole bunch of things that you can do in like you know short film production or you know shorts we've launched on YouTube as well um to help creators kind of like create um content more easily. Um and then the other one is obviously video editing. Like that's another thing that we're really excited about that we're just making easier because now you can use natural language to take a video, you know, add something, remove something. Sloth is obviously like fun example. Um, but there there's obviously kind of there's consumer use cases that we kind of had in mind where, you know, you could take your beach vacation video that was too noisy and you want to clean up that noise. Maybe in the past you wouldn't have because you didn't have the tools or you didn't know what the tools were that you needed to go to. So,

好。这个模型的招牌能力，或者说有两个。一个是它几乎什么都能拿来当输入，另一头直接出视频。当然将来——我们预告的时候也提过——我们还想把其他输出模态也放出来。但说白了，这意味着你可以拿手头的一组图片当故事板，拿一段音轨当参考，比如你想让某个角色用什么样的声音说话，然后另一头就能拿到视频。这一下就解锁了一大堆玩法，比如短片制作，还有 Shorts——我们在 YouTube 上也上线了——帮创作者更轻松地做内容。另一个当然就是视频编辑。这也是我们特别兴奋的一点，我们正在把这件事变得更简单：现在你可以用自然语言来改一段视频，加点东西、删点东西。树懒显然是个好玩的例子，但我们心里也有一些面向普通消费者的场景，比如你海边度假拍的视频太吵了，想把噪音清理掉。以前你可能根本不会去弄，因为手头没有工具，或者压根不知道该去找什么工具。所以，

[6:05] Nicole

that's one use case that you can, you know, go to. We've seen a lot of folks use it for kind of marketing ad campaign creation and I'm excited to see more of those use cases as we launch the APIs. um because obviously like we don't we don't see all of it in the first party products but I'm really excited for people to start to explore that um in the API. So those are just some of the kind of like high level um things that have come up. U people also use it to create like education materials. Yes. Um and like like that's really exciting. I think we're all we've all kind of talked about being excited about the future of education where like everything can be kind of customized to you and personalized to your knowledge level and the style that you prefer and and so this is kind of just like a step in that direction.

这就是一个你可以用上的场景。我们也看到很多人拿它来做营销广告活动的素材，随着 API 上线，我很期待看到更多这类用例。因为在我们自家的第一方产品里，显然看不到所有用法，所以我真的很期待大家开始在 API 里去探索。以上就是目前冒出来的一些大方向上的用法。还有人用它来做教育素材。对。这真的很让人兴奋。我想我们都聊过自己对未来教育的期待：一切都能为你量身定制，按你的知识水平和你偏好的风格来个性化。而这算是朝那个方向迈出的一步。

[6:47] Dumitru

Yeah. I I I sort of actually used just none of yesterday, but my my parents are visiting and there was there was a very fun sort of use case. They I bought some gadget off from Amazon that they wanted and the instructions to use it was were only in English and there was plenty of diagrams or whatever and I took a picture of it and said, you know, translate this into Romanian. Yes.

对。我其实昨天刚好就用了一下——我爸妈正来看我，碰上了一个特别好玩的用例。我在 Amazon 上给他们买了个他们想要的小玩意儿，结果使用说明只有英文，里面还有一堆示意图什么的。我就拍了张照片，说：把这个翻译成罗马尼亚语。对。

[7:04] Dumitru

And keep everything else the same, right? So it was amazing, right? Like it was just like, yeah, it looks identical and it has, you know, it's perfectly translated. I mean, more or less, right? But it's it's you know using Gemini under the hood obviously to kind of do the translation and so you can you can see this use case for video as well right like the the power of text rendering in in in Omni is is quite next level. So and you could you could you could think about plenty of use cases of like both text rendering translation internalization all sorts of things that would be actually genuinely useful to a lot of different people and sort of broader access to either you could like redub a video or whatever it is that you wanted to do. like there's plenty of different things that you could you could think about doing.

而且其他所有东西都保持原样，对吧？结果真的很惊艳。出来的图看起来一模一样，而且翻译得很完美——好吧，大体上完美。当然，它底层显然是用 Gemini 来做翻译的。所以你也能想见视频上会有同样的用例——Omni 的文字渲染能力真的是另一个级别的。你能想到大量的用例，文字渲染、翻译、国际化，各种各样的东西，对很多不同的人来说都是实打实有用的，也能让更多人用得上——比如你可以给一段视频重新配音，或者做任何你想做的事。能想到的玩法真的非常多。

[7:45] 主持人

Yeah. Um one of the most enlightening conversations I have on my podcast is with uh just people researchers at the frontier of these things. Um I had one with um Ethan from the XAI video team, the Grock video team who was basically saying like you know the next trend is actually not just like single model, it's more like video agents. Um, and I don't know if that terminology resonates uh obviously for for very relevant for RL. Uh, but it was it was basically kind of like giving up on like trying to do everything in in effectively one pass. Um, do you feel that same way or is it still an open research question which way the trends are going? [snorts]

对。我播客上最有启发的一类对话，就是跟这些领域最前沿的研究员聊。我跟 xAI 视频团队——也就是 Grok 视频团队——的 Ethan 聊过一次，他基本上是说，下一个趋势其实不是单一模型，而更像是视频 agent。我不知道这个说法你们认不认同，它显然跟 RL 关系非常大。但这本质上等于放弃了“一次生成就把所有事都搞定”的思路。你们也这么看吗？还是说趋势会往哪边走，目前仍是个开放的研究问题？

[8:24] Shane

Yeah. So um what kind of excite me most is really when the symbolic kind of foundational models and this kind of like video foundational model can actually kind of really work together and u in a way the if you look at the beginning of the generative sort of like image generation video generation a lot of it kind of started when the language model got good enough to provide a very detailed captioning like from stable fusion days or kind of dowi 2 days. So um so basically like language is extremely u helpful representation uh one is that it's kind of universal but the other kind of more um technical thing like kind of my hypothesis is like um one very difficult thing about machine learning is um this sort of like spirious coordination. So you don't know you know if the if this kind of feature right that's kind of predictive is actually causal factor or not. There are two ways. One is we can have really diverse data training data like from every intervention of the causal graph.

对。我最兴奋的，其实是符号类的基础模型和视频这类基础模型能真正协同起来。某种程度上，你回头看生成式的起步阶段——比如图像生成、视频生成——很大程度上是从语言模型好到能给出非常详细的 caption 开始的，比如 Stable Diffusion 那会儿，或者 DALL·E 2 那会儿。所以说到底，语言是一种极其有用的表征。一方面它很通用；另一方面是个更技术性的点，也算是我的一个假设：机器学习里有件非常难的事，就是这种伪相关（spurious correlation）。你不知道某个有预测力的特征，到底是不是真正的因果因素。解决办法有两种。一种是让训练数据非常多样，比如覆盖因果图上的每一种干预。

[9:19] Shane

The other is you condition the causal information and conditioning the language is kind of like conditioning like a coal information of the of the kind of world. So um

另一种是以因果信息为条件（conditioning）——而以语言为条件，某种程度上就相当于以这个世界的因果信息为条件。所以，嗯……

[9:29] 主持人

which is a prompt or a concept what

也就是 prompt，或者说某个概念之类的？

[9:32] Shane

yeah exactly so if you look at like you know how we going to describe this video how this kind of image is actually very close to you know how would describe this kind of causality you know behind this like how this is kind of generated. So one is like that can really allow for very rich generalization and then uh very kind of just like a good model. Um the other is so eight months ago uh we put the evaluation paper called video models zero shot learners and reasoners.

对，没错。你看，我们会怎么描述这段视频、这张图，其实跟怎么描述它背后的因果——也就是它是怎么生成出来的——非常接近。所以一方面，这能带来非常丰富的泛化能力，也就能得到一个很好的模型。嗯，另一方面，大概八个月前，我们发了一篇评测论文，叫「Video Models are Zero-Shot Learners and Reasoners」（视频模型是零样本学习者与推理者）。

[9:58] Shane

Yes. So that was a kind of you know it's it's a confirmed paper and then later on actually the N banana team followed up with a vision banana paper that basically used n banana to do but essentially the idea is uh video model is extremely good sort of a foundation model for space and time kind of information. So um classic computer vision tasks a lot of could be kind of zero shorted and when you like say feed in some like a visual quiz uh it can you know there's definitely like a lot to improve it can kind of solve and it can um like robotics kind of like seeing it has really good kind of physical intuitions like word model uh and I think the the key is really the kind of mix of the visual kind of reasoning and then the text kind of reasoning kind of all tied together Um obviously you know like whether doing it you know as kind of unified model versus like just kind of agent coation I think that's more like uh it's going to be more kind of incremental you know how it's going to I imagine everything's going to go into like a single model eventually

对。那篇论文算是把这一点验证了，后来 Nano Banana 团队其实还跟进发了一篇 Vision Banana 论文，基本上就是用 Nano Banana 来做……不过核心思路是：视频模型是处理时空信息的一种极其出色的基础模型。所以，嗯，很多经典的计算机视觉任务都能零样本搞定；比如你给它输入一道视觉类的智力题，它也能——当然肯定还有很大的提升空间——基本解出来；而且它还能，嗯，比如在机器人方向，我们看到它具备非常好的物理直觉，就像一个世界模型。嗯，我认为关键其实在于把视觉推理和文本推理融合起来，全都串在一起。嗯，当然，到底是做成一个统一模型，还是用多个 agent 协同的方式，我觉得这更多会是一个渐进的过程，就是它会怎么……我估计最终所有东西都会并进单一模型里

[10:59] Shane

but right now there's like a lot you can do if you uh basically take like really good video understanding image understanding Gemini agentically with anomy and that's actually gonna yeah our team is like exploring a lot

但眼下，如果你把视频理解、图像理解都非常强的 Gemini，以 agent 的方式和 Omni 搭配起来用，其实已经能做很多事了，而且这其实会……对，我们团队正在这方面做大量探索。

[11:12] 主持人

yeah okay that there's a there's a lot in there um I I think uh one question I I am increasingly starting to wonder is does it all trend towards one product for you guys right like now you have multiple models out the naming of omni does imply that eventually everything will go away and it just goes into omnis um is that the plan

好，这里面信息量很大。嗯，我越来越想问的一个问题是：对你们来说，最终是不是一切都会收敛成一个产品？现在你们已经推出了好几个模型，而 Omni 这个命名确实暗示着，最终其他东西都会消失，全都并进 Omni 里。嗯，这是你们的计划吗？

[11:33] Dumitru

is it [laughter] I don't know I I think I think uh maybe I mean I think eventually I I think there's sort of different trade-offs engineering research product trade-offs in like it's like for the same reason like the the sorry how is it called nano banana light I don't know what the product name

是吗？（笑）我不知道。我觉得……我觉得，嗯，也许吧，我是说，我觉得最终……我觉得这里面存在各种不同的权衡，工程、研究、产品上的权衡。就好比，出于同样的原因，那个……抱歉，它叫什么来着，Nano Banana Lite？我不知道产品名叫什么

[11:52] Nicole

nanob banana tite

Nano Banana 2 Lite。

[11:53] Dumitru

nano banana too light yeah right it's it's it's it serves a particular niche right and it probably doesn't necessarily fit immediately in the same model literally checkpoint as uh something that can do 4K you know uh 30 secondond videos right like they're probably not like trainable in the same quite way, right? Like, so I I don't know. It depends on how how far into the future you look like. Sure, in five years from now, will they all be the same model? Probably. Uh but like, you know, six months from now, we'll we'll probably still have, you know, multiple different models doing different things because kind of from pragmatically the trade-offs are such that we we should have multiple different kinds of models.

Nano Banana 2 Lite，对。它服务的是一个特定的细分需求，对吧？它不一定能马上和那种能生成 4K、30 秒视频的模型装进同一个模型里——我是说字面意义上的同一个 checkpoint，对吧？它们大概没法用完全相同的方式来训练，对吧？所以我也说不好，这取决于你往未来看多远。当然，五年后它们会不会都是同一个模型？大概会。但六个月后，我们大概率还是会有多个不同的模型各司其职，因为从务实的角度看，这些权衡决定了我们就应该有多种不同的模型。

[12:35]

Yeah, I

对，我……

[12:36] Nicole

I think that's right. And and just on that note, I mean, we did call it Gemini Omni because we wanted to hint at the future where Gemini just becomes fully multimodal in and out, right? And so so it's definitely a move in that direction. I think we'll probably see a move in the direction where Omni also generates images and edits images and all those kinds of things. But Doo is right that I think on the way there, there's a bunch of really really useful applications of some of these more specialized models. And so we we will probably continue to work on those as well because like that serves a certain need at this point in time that may not exist you know a year from now. There's also like a research question about like just how much transfer there is between different kinds of modalities, right? I think you may believe that there's some transfer between coding and video generation and I think most people don't necessarily believe that but they you know you could try to think that there is some some there something there or it could be a waste right to put them together to try to learn these both tasks at the same time right so I think it's it's it's interesting sort of question to which extent like image and video obviously kind of there's some transfer like kind of not that different there's value in in learning to output video and audio at the same time because joint audio visual is you know that's how that's how it is. Um and then there's you know other kind of intersections of modalities that are not super obvious right like 3D representation coding I don't know maybe uh things like that right so like I think it's worth sort of exploring the different corners there and we are actively doing that um with a focus towards like what people actually want to do with these models

我觉得没错。顺着这个话题再说一句，我们之所以把它叫做 Gemini Omni，就是想暗示这样一个未来：Gemini 会变成输入输出都完全多模态的模型，对吧？所以这无疑是朝那个方向迈出的一步。我觉得接下来我们很可能会看到 Omni 往生成图片、编辑图片这类方向发展。但 Dumitru 说得对，在走向那个未来的路上，一些更专精的模型有一大堆非常非常有用的应用。所以我们很可能也会继续做这些模型，因为它们满足的是当下的某种需求，而这种需求一年后可能就不存在了。另外还有一个研究层面的问题：不同模态之间到底有多少迁移（transfer），对吧？你可能相信编程和视频生成之间存在某种迁移，但我觉得大多数人不一定这么认为。不过你也可以试着去想，这里面也许真有点东西；也可能把两者放在一起、试图同时学这两个任务纯属浪费，对吧？所以我觉得这是个挺有意思的问题：到底在多大程度上……比如图像和视频之间，显然是有一定迁移的，毕竟两者差别没那么大；同时学会输出视频和音频也有价值，因为音画一体，本来就是这么回事。嗯，此外还有一些不那么显而易见的模态交叉，对吧，比如 3D 表示、编程，我也说不准，也许还有类似的东西。所以我觉得值得去探索这些不同的角落，我们也正在积极地做，嗯，重点放在人们实际想用这些模型做什么上。

[14:05] 主持人

yeah um what one thing I feel I feel like uh I'm surprised by but also I feel like it's insufficiently answered is what is the correct intermediate representation Um, so captioning, right? XI does captioning. Omni does captioning. Um, and I I I understand how captioning works for images. Um, and I understand that you can extend it into to video and and sort of guide it across time. It just feels very inefficient. It there's got to be I feel like there should be something better. Uh maybe it's code and maybe we generate you know and obviously I think a lot of um ffmpeg and mapplot um what's the three blue one brown one manm um a lot of like video is generated through code and maybe that's like the optimal representation uh any hypothesis as to like is is it better or is just English all you need

好。嗯，有一件事我觉得挺意外，同时也觉得一直没被充分回答，就是：正确的中间表示到底是什么？嗯，比如 captioning（给画面生成文字描述），对吧？xAI 在做 captioning，Omni 也在做 captioning。嗯，我能理解 captioning 在图像上是怎么起作用的，嗯，也理解你可以把它扩展到视频上，沿着时间维度去引导生成。但就是感觉效率非常低。一定有……我觉得应该有更好的东西。也许是代码？也许我们生成……显然，很多像 ffmpeg、matplotlib，嗯，还有 3Blue1Brown 用的那个叫什么来着，Manim，嗯，很多视频都是用代码生成的，也许那才是最优的表示。你们有什么假设吗——代码到底更好，还是说 English is all you need，只要英语就够了？

[15:01] Shane

well as so I'm in the Gemini and they know we do like a lot of RL agent and of course kind of coding so yeah We we're definitely exploring the coding representations.

嗯，我在 Gemini 团队，你知道，我们做大量 RL、agent，当然还有编程相关的东西。所以对，我们肯定在探索用代码来做表示。

[15:10]

Yeah.

对。

[15:10] Shane

As kind of better kind of way to represent. Yeah.

作为一种更好的表示方式。对。

[15:13] 主持人

But you know like do you what's your probability estimate on like [laughter] if we just output binaries like we just you know like just it's just ones and zeros.

不过，你们……你们估计有多大概率——（笑）——我们干脆直接输出二进制，就是，全都是 0 和 1？

[15:23] Shane

Um I I guess maybe a kind of similar discussion was like um basically is the language the right representation like right. So uh one kind of question for example uh professor you know like some ask is like you know why why does the channel of thought need to be in the natural language?

嗯，我想一个类似的讨论是：语言到底是不是正确的表示，对吧？比如有个问题，一些教授会问：为什么 chain of thought（思维链）非得用自然语言？

[15:42]

Yes.

对。

[15:42] Shane

Can it just be the kind of any kind of like continuous tokens just any amount of you know additional computations.

它能不能就是任意形式的连续 token，也就是任意多的额外计算？

[15:48] Shane

Um so one is like obviously the test like adaptive compute is going to give like you know better results. So it's that but what really kind of made CH thought so you know like four years ago I wrote you know the larger model zero sort reasoner and then self-improvement. So I kind of know from the very early day but the reason like it works really well is um right now the recipe that works is the pre-training that scales a lot and then that basically like learns a lot of intelligence. there are a lot of you know scaling RL but those are still like extremely kind of comput incent intensive to extract the information and um you really want to rely the intelligence on that so basically by tying the sort of like a reasoning in the natural language you basically directly use the intelligence of the pre-training to it while if you remove that kind of constraints then you're not um and these days uh I feel the a lot of advancements in the texts but also doing this kind of multimodal space is very driven by this uh kind of text as a kind of great uh sort of representation.

嗯，一方面，测试时（test-time）的自适应计算显然会带来更好的结果，这是一点。但真正让 chain of thought 奏效的……你知道，四年前我写了「Large Language Models are Zero-Shot Reasoners」，后来又写了 self-improvement（自我改进）那篇，所以我算是从很早期就了解这件事。它之所以效果这么好，原因在于：目前真正奏效的配方，是大规模扩展的预训练，模型基本上就是从中学到了大量智能。现在也有很多人在扩展 RL，但用 RL 去提取这些信息依然极其耗费算力，所以你会非常想依靠预训练得来的那部分智能。把推理绑定在自然语言上，你基本上就能直接调用预训练的智能；而如果去掉这个约束，你就用不上了。嗯，如今我感觉，文本领域的很多进展，包括多模态领域的进展，很大程度上都是靠「文本是一种极好的表示」这一点推动的。

[16:54]

Yeah, it's a good backbone.

对，它是个很好的主干（backbone）。

[16:56]

Yeah,

对，

[16:57] Nicole

I I think to me it's even simpler than that. It's text is is how we communicate. So I think fundamentally if you're building kind of products that humans will be interfacing with um like like that we will be using text somehow if it's a text interface, right? Not not for everything. So I think it's it's natural to default to that.

我觉得对我来说，这事比那还简单：文本就是我们交流的方式。所以从根本上说，如果你在做人要直接打交道的产品，嗯，那我们总会以某种方式用到文本——如果是文本界面的话，对吧？当然不是所有东西都这样。所以我觉得默认用文本是很自然的。

[17:15] Shane

Yeah. Obviously there's like a conf discussion you know some arrow like arrow maximalists is like oh we don't care about you know kind of channel those kind of like stuff it's just just additional compute

对。当然也有不同的声音，你知道，一些 RL 至上主义者（RL maximalists）会说：我们根本不在乎 chain of thought 那类东西，那不过就是额外的计算而已

[17:24]

sure but I personally yeah

当然，但我个人……对

[17:26]

ro maximalists I wonder I wonder who who qualifies in that description David silver

RL 至上主义者……我倒挺好奇谁符合这个描述。David Silver？

[17:32] 主持人

ah okay yeah I mean they they've just left to to start their thing um interesting okay so uh I I mean I think I'm very interested in just like better representations because I think that's one of our themes that we're curating today uh at the world fair is world models. You mentioned the word world models but it's not something that's like super well defined. I think everyone's like sort of converging on some version of it that is like the ideal.

啊，好吧。对，我是说，他们刚离开，出去自立门户了。嗯，有意思。好，所以，嗯，我是说，我对更好的表示方式非常感兴趣，因为我们今天在 World's Fair 上策划的主题之一就是世界模型。你们提到了「世界模型」这个词，但它其实没有一个特别清晰的定义。我感觉大家都在往某个理想的版本上收敛。

[17:57]

Sure. Everything is a world model now. It's sort of a

当然。现在什么都是世界模型了。这有点像是……

[18:00]

it's not it's not that useful, right?

这就没那么有用了，对吧？

[18:02] Shane

So I just gave a keynote at the IER world model workshop. Yeah. And then uh yeah essentially uh I definitely encourage to check out the definition by Jandra Matic. He's like the you know OG computer vision professor UC Berkeley.

我刚在 IER 的世界模型 workshop 上做了一场主题演讲。对。然后，嗯，基本上，我非常推荐大家去看看 Jitendra Malik 给的定义。他算是 UC Berkeley 元老级的计算机视觉教授。

[18:15] Shane

Uh he has pretty you know bit of word to say about world model

嗯，关于世界模型，他颇有一番见解

[18:18] Shane

but also kind of Schmidt Herburver's kind of how he defined the world model from 2019 like 1990 sort of uh uh you know like Wayne was just basically just that kind of model base. Uh for me the word model is basically just the model in the model based RL and I feel that has sufficient to describe but obviously you know there are like a lot of uh FE had a kind of nice blog post about what about yeah this kind of broken down

不过还有 Schmidhuber 对世界模型的定义，从 2019 年，呃，应该说 1990 年那会儿，嗯，你知道，那时候基本上就是那种 model-based 的思路。嗯，对我来说，世界模型基本上就是 model-based RL 里的那个 model，我觉得这就足以描述它了。当然，还有很多……FE 写过一篇挺不错的博客文章，讲的是……对，把这个问题拆解了一番

[18:41]

um but yeah

嗯，不过，对。

[18:43]

yeah I mean so you know I I'll end this part of the conversation but like I I do think that language to me relying on language as like the sort of like the narrow pipe through which everything goes through um still is like a lossy compression. No, no, no. But we're not seeing that, right? We're basically saying the video model and the language together.

主持人： 对，我是说，这部分话题我就先收个尾。不过我确实认为，语言……对我来说，依赖语言作为那根让所有东西都得从中通过的细管道，终究还是一种有损压缩。

Shane： 不不不，但我们不是这么说的，对吧？我们说的基本上是视频模型和语言结合在一起。

[19:02] Shane

So, so I think the language alone is uh not sufficient. That's why we feel like the video is a very complement model. Right? Now the um you know kind of v omni many people feel as uh you know generating kind of pretty videos but I think our vision it's it's much more than that. It's a missing foundational model that's absolutely required if you want to make the AGI that match to humans not just a jacked one.

所以我认为单靠语言是不够的。这就是为什么我们觉得视频是一个非常互补的模型，对吧？现在很多人觉得 Veo、Omni 就是用来生成好看视频的，但我认为我们的愿景远不止于此：它是一个目前还缺失的基础模型，如果你想做出能与人类匹敌的 AGI，而不只是一个聊天型的 AGI，它是绝对必需的。

[19:25] 主持人

Yeah. Um okay. So one one other thing you know you you mentioned on the vision side um and I'm kind of curious how sort of uh parallel you know in terms of your research careers um this development is like I think basically a lot of vision people have crossed over into more model people um a lot of vision people also become generative video and image people and is it just as simple as you know reversing uh image to text and then now it's text to image like

好。嗯，那另一件事，你们提到了视觉这一侧。嗯，我挺好奇的是，这一发展和你们各自的研究生涯有多大程度上是平行的：我觉得基本上，很多做视觉的人都跨界成了更偏模型方向的人，嗯，很多做视觉的人也变成了做生成式视频和图像的人。那这是不是就像把图生文反过来、变成文生图那么简单？就好像

[19:57]

[laughter]

（笑）

[19:58] 主持人

is is that if I mean that effectively was the diffusion process. Um I I just you know I I just see the career paths of the people that I talked to and and see and I I I see this overall trend of research directions and I just wanted you to guys to sort of reflect on on that.

是不是就这么回事——我是说，那实际上就是扩散过程嘛。嗯，我就是……你知道，我看到跟我聊过的那些人的职业路径，也看到研究方向上的这种整体趋势，所以就想请你们几位聊聊对这件事的看法。

[20:16] Dumitru

I mean I certainly went that way right I I started long time ago uh doing computer vision sort of object detection recognition things like that. Uh I think just that's just simpler problem right just generation is just harder like it's a it's a different kind of mapping right you map from the the inverse mapping is not as simple as just inverting the the kind of network you use right it's it's a it's it's more ambiguous right to go from cat to image of a cat and in some ways it's also a loop because your vision work creates the synthetic labels that then continues

我是说，我确实就是这么走过来的，对吧？我很早以前是做计算机视觉的，目标检测、识别这类东西。呃，我觉得那就是个更简单的问题，而生成就是更难，它是另一种映射。逆向映射可不是把你用的那种网络直接倒过来那么简单，对吧？从「猫」到一张猫的图片，歧义要大得多。而且某种意义上这也是一个循环，因为视觉这边的工作产出了合成标签，然后又继续……

[20:46] 主持人

I mean sure [laughter] I don't know I don't know I try to validate my my sort of theories about how fields develop how how careers has progressed through this

嗯，当然。（笑）我也不知道，不知道，我就是想拿这个验证一下我自己的一些理论——一个领域是怎么发展起来的，大家的职业路径又是怎么随之演进的。

[20:55] Nicole

I mean for like the the the better the understanding side gets like we have seen that the generation side also gets better right so like like

我是说，理解这一侧做得越好，我们看到生成这一侧也会跟着变好，对吧？所以就是……

[21:03]

it's completely bootstrapping yeah it's

这完全是自举式的，对，它是……

[21:05] Nicole

and so like like like there's definitely they're there to that thesis and I think yeah I think a lot of people have kind of like I I definitely worked with a lot of um image understanding people who became image generation people you know and then some of them have moved on to video because it's kind of like the next thing where you have so many more dimensions to work with so yeah I'm curious about you spec as your

所以，你这个论点确实有道理。而且我觉得，对，很多人都有点……我确实跟很多做图像理解的人共事过，他们后来都转去做图像生成了，然后其中一些人又去做视频，因为视频算是下一站，能处理的维度多得多。所以，我也很好奇你的看法，作为……

[21:24] Shane

so I definitely like recommend start with understanding recognition because that's basically discriminator and then that's going to lead to better generation and that's what the bridge is basically reinforcement learning so my um my kind of journey is I initially kind of worked on the algorithmic research in the gent model against some like you know amnest kind of generation and then I worked on like RL and robotics um and then like six years ago I was like leading like a moonshot on the dexterity it was pretty early but I see now everyone's kind of doing uh four years ago I basically kind of figured out that this like symbolic AGI is going to accelerate much faster than the kind of physical AGI kind of counterpart. So uh I decided to kind of like language models and then those things. Um and then recently kind of work with Doomi and then like omni team I quite enjoy kind of collaboration there. the what I quite enjoy uh what I recommend definitely to the researcher is to uh definitely kind of explore or at least like get exposure to what the top people in each of the community are like looking at how they kind of think about problems. So when I look at the video model to me it kind of reminds me like pretty early on sort of like language model where like very early language model was a kind of creative sort of demo right you kind of like try to write like a story like mobile and then like you know GBD2 and then those kind of days like LTM kind of days right and then you know uh instruction tuning you actually kind of make it usable as a chatbot but then at the chatbot stage it still had so much hallucinations and instruction for wasn't good enough so it couldn't use for reasoning and when I got good enough um in pre-training and post- trainining for reasoning then you know this kind of test time scaling the RL really took off to like many of the kind of best performing models and right now I think the video model is as we mentioned it's it is a complimentary

所以我非常推荐从理解、识别入手，因为那本质上就是判别器，它会带来更好的生成，而这中间的桥梁，基本上就是强化学习。说说我自己的路径：我最早做的是生成模型、GAN 方面的算法研究，做一些 MNIST 那种生成，后来做 RL 和机器人。大概六年前，我在带一个机器人灵巧操作方向的登月项目，那时候还挺早的，但现在我看大家都在做了。四年前，我基本上想明白了一件事：符号层面的 AGI 会比与之对应的物理 AGI 加速得快得多。所以我决定转去做语言模型那些东西。最近在跟 Dumi 还有 Omni 团队合作，我挺享受这边的协作的。我特别想推荐给研究者的是：一定要去探索一下，至少去接触一下，每个社区里最顶尖的人在关注什么、他们怎么思考问题。所以当我看视频模型时，它让我想起相当早期的语言模型。很早期的语言模型基本上就是一种创意类的 demo，对吧？你拿它试着写个故事什么的，就是 GPT-2 那会儿，还有 LSTM 那会儿。然后有了 instruction tuning，你才真正让它能当聊天机器人用。但到了聊天机器人阶段，它还是有大量幻觉，指令遵循也不够好，所以没法用来做推理。等它在预训练和面向推理的后训练上都做得足够好之后，test-time scaling、RL 才真正起飞，成就了很多性能最强的模型。而现在我觉得，视频模型就像我们前面提到的，是一个互补的

[23:12]

foundational model and I can imagine it's going to follow a similar path it's going to be very uh it's going to improve a lot instruction following a lot of uh this it's going to improve a lot in reducing coordinations to extend that it become a very reliable world model so we can kind of like intermixed video like space-time simulation was a text simulation to solve like arbitrary AGI problems. Also like I think the difference still is between sort of text models and like image video models is that like we haven't quite unified understanding and generation in in multimedia I'd say yet like I mean I think I think without going to the details of course there's like it depends on on at which level you're thinking about this but generally like there's not that many as far as I know models sot kind of you know frontier models that are genuinely kind of good at both understanding and generation of of let's videos, right?

Shane： 基础模型。我能想象它会走一条类似的路：指令遵循等很多方面都会大幅提升，幻觉也会大幅减少，直到它成为一个非常可靠的世界模型。这样我们就能把视频这种时空模拟和文本模拟交织在一起，去解决任意的 AGI 问题。

Dumitru： 另外，我觉得文本模型和图像、视频模型之间目前还有一个区别：在多媒体领域，我们还没有真正把理解和生成统一起来，我会说是还没有。我是说，不展开细节的话——当然这取决于你从哪个层面去看——但总体来说，据我所知，真正能同时把理解和生成都做好的 SOTA 前沿模型并不多，比方说在视频上，对吧？

[24:04] Dumitru

Like it's a it's a it's an interesting challenge. I'm not saying that we should do this. Uh but but I think uh it kind of stands to reason that like you know understanding and generation are two sides of the same coin. So they they kind of should be in the same model in some ways. Uh but we don't necessarily always do that. So yeah.

这是个很有意思的挑战。我不是说我们就应该这么做。但我觉得，理解和生成是同一枚硬币的两面，这在道理上是讲得通的，所以某种程度上它们应该放在同一个模型里。只是我们并不总是这么做。嗯，就这样。

[24:20]

Uh you mentioned audio as well, right? Yeah. Uh is that as hard as video or qualitatively different? If if so, in what way? Uh, one of the interesting directions three years ago was people using um, I guess diffusion to do audio uh, as in like the the sort of refusion approach. I don't know if you you guys saw that. Um, and I just think it's like very interesting if a modality that we perceive which is audio is different than video actually two machines is exactly the same like there's they see no difference. I mean I think on a technical level there are some differences but I think they're like relatively minor. I think from my perspective audio came into into my life when we shipped V3 which was I believe the first model that did like a joint

主持人： 呃，你也提到了音频，对吧？对。呃，音频跟视频一样难吗，还是在性质上就不一样？如果不一样，是哪里不一样？呃，三年前有个挺有意思的方向，就是有人用——我想是扩散模型——来做音频，就是 Riffusion 那种路子，不知道你们有没有看过。嗯，我就觉得很有意思：音频这种模态，我们人类感知起来跟视频不一样，但对机器来说其实完全一样，它们看不出任何区别。

Dumitru： 我是说，我觉得技术层面上确实有些差别，但都相对次要。从我的角度讲，音频真正进入我的生活，是在我们发布 Veo 3 的时候，我认为它是第一个做到那种联合……

[25:12]

with the slicing of the

就是切那个……

[25:14] Dumitru

Yeah. Yeah. the gold bars or whatever. Um it it was the first model that did this sort of joint audiovisisual generation. Yes. uh like in the in the I mean there are there were other models that did kind of you know kind of kind of agentic hacking under the hood but this one was truly sort of you know generating everything at once and we the reason we did that is because we felt and I think was the right choice we felt that like uh it only makes sense to generate them at the same time because there sort of kind of like from a machine learning perspective there's one latent kind of you know causal kind of you know generative process right like there's something that generates you speaking it's not the pixels and then the the audio or somehow somehow generated by some other process like the lips have to move in sync with with the with the audio, right? So, I think that that solved a lot of the issues that previous models had or the way that people did video generation before where it was like, okay, we generate the pixels and then we're going to hack something on top of it that like moves the lips with the audio that we generate. And that's was very bad.

对，对，切金条什么的。嗯，它是第一个做到这种音视频联合生成的模型。是的。呃，就是说——当然也有别的模型，是在底层用某种 agentic 的方式东拼西凑出来的，但这个是真正一次性把所有东西都生成出来。我们之所以这么做，是因为我们觉得——而且我认为这个选择是对的——只有同时生成它们才说得通。因为从机器学习的角度看，背后只有一个潜在的、因果性的生成过程，对吧？就是说，有某个东西生成了「你在说话」这件事，而不是先有像素，音频再由另一个什么过程单独生成出来——嘴唇得跟音频同步动才行，对吧？所以我觉得，这解决了以前模型的很多问题，也就是以前大家做视频生成的那种方式：好，我们先把像素生成出来，然后在上面硬拼一个东西，让嘴唇跟着我们生成的音频动。那种做法效果非常差。

[26:11] Dumitru

And so, I think I think that was that's to me that's the the I mean after V3 like you know people were like what do you mean like there's no audio in your model? like that makes no sense like once it's there like you you have to have it. So I think that was that was the right choice and doing it to one single generative model I think was was the right choice.

所以我觉得，对我来说这就是……我是说，Veo 3 之后，大家的反应都变成了：什么？你的模型居然没有音频？这说不过去啊。一旦有了，就必须得有。所以我觉得那是正确的选择，把它放在一个单一的生成模型里来做，我认为是正确的选择。

[26:28] Shane

One thing I kind of want to also can ask you guys an opinion as well once one difference I find the audio and then against the image and video is like the audio information is less verbalized. I mean of course the TTS and stuff is trivial right but the when you get her outside like how to describe music how do you describe this like this person's tone kind of pitch I feel the sort of the verbalization is insufficient and the interesting thing is that you kind of see that in two other things like taste taste sense and also uh say um touch

有件事我也想听听你们几位的看法。我发现音频跟图像、视频相比有一个区别：音频信息更难用语言表达出来。当然，TTS 这类东西很简单，对吧？但一旦超出这个范围，比如怎么描述音乐，怎么描述一个人的语气、音高，我感觉语言化是不够用的。有意思的是，你在另外几样东西上也能看到这一点，比如味觉，还有，嗯，触觉……

[27:00] Shane

like smell and then the another interesting thing is the skin color so skin color the the language is pretty limited to describe the skin color and the reason is that we're extremely uh sensitive to the small difference perturvations or not skin color because that basically shows us is this person going to kill me or is can I befriend this person kind of those kind of information and then I feel the smell tastes um skin color and like sound kind of stuff is very very tied into primitive a like survival kind of stuff and so our sort of sensory system is so sensitive that it's intractable to Um, so for example, I asked like one the wine sort of taster and then like professional and then he basically said he kind of use like a language from like a dating, you know, describing like a, you know, partner as a way to describe the taste because there's no sufficient vocab to describe. Um, so I'm kind of curious. Yeah. Do you guys feel that?

还有嗅觉。另一个有意思的是肤色。描述肤色的语言相当有限，原因是我们对肤色上细微的差异、扰动极其敏感，因为这基本上是在告诉我们：这个人会不会杀了我，还是说我可以跟他交朋友——就是这一类信息。我感觉嗅觉、味觉、肤色，还有声音这些东西，都跟很原始的、关乎生存的东西紧紧绑在一起，所以我们的感官系统敏感到根本没法去……嗯。举个例子，我问过一位专业品酒师，他基本上说，他会借用约会时形容伴侣的那种语言来形容味道，因为没有足够的词汇可用。嗯，所以我挺好奇的。对，你们有这种感觉吗？

[27:58] Nicole

I think well to some extent I think the same is true for visual information, right? when you think about like a certain style or a certain aesthetic, right? Like like there are some people who just have a much more kind of developed like whether it's palette or kind of visual taste and aesthetic, right? Like I I think language just tends to be a bit of a limiting factor when you are trying to describe any of these things that like we experience with sensory information. And to your point earlier, I think that is the kind of the reason why we are investing in world models and why we are pushing on kind of the like perception and like generation side of things because it it is such a large part of how we as humans navigate the world. It's a large part of how like embodied AI navigates the world. Um, and and I do I do think language like does have a lot of it's it's gotten us very far and it can probably get us really far, but it it feels limiting in a lot of these kind of areas. And yeah, I don't I don't really know how to describe, you know, like sense and taste. Um, but yeah, I'm curious to me.

我觉得，某种程度上视觉信息也是一样的，对吧？比如你想想某种风格、某种审美。有些人就是有发达得多的鉴赏力，不管是味觉上的品鉴力，还是视觉上的品味和审美，对吧？我觉得，当你试图描述这些我们通过感官体验到的东西时，语言往往就有点成了限制因素。回到你前面说的，我觉得这也正是我们投入做世界模型、在感知和生成这一侧持续发力的原因，因为这在很大程度上就是我们人类认识和应对这个世界的方式，也在很大程度上是具身 AI 认识和应对世界的方式。嗯，我确实认为语言……它已经带我们走了很远，而且很可能还能带我们走得非常远，但在很多这类领域里，它让人感觉受限。对，我也不太知道该怎么描述，你知道的，气味和味道这种东西。嗯，不过，Dumi，我也很好奇你怎么看。

[29:03]

Um, I I yeah, I don't know that I have thought that deeply about this yet. So, uh, yeah, I mean yeah, I don't have a good answer about audio. I mean like I don't know the limit because I'm thinking about like well what is what is Omni bad at in terms of audio but they're all like solvable problems I find uh so like with more data or better data or whatever it is so I don't know like that we have pushed the frontier so much that like we are have hit some sort of limits that are rooted in evolutionary uh kind of you know limits imposed by humans. I don't know. He's feeling the limits of captioning which is the the thing I was

Dumitru： 嗯，我……对，我还没把这个问题想得那么深。所以，嗯，对，我在音频这块没有什么好答案。我是说，我不知道极限在哪，因为我在想：Omni 在音频上到底哪里做得不好？但我发现那些都是可以解决的问题，靠更多的数据、更好的数据之类的就行。所以我不确定我们是不是已经把前沿推进到了那种程度，撞上了某种根植于进化、由人类自身施加的限制。我不知道。

主持人： 他感受到的是 captioning 的局限，这正是我之前……

[29:41] Shane

Yeah, exactly. [laughter] There there's a lot of information in the world and it connects to basically why we do work modeling you mentioned. You just need srefs sref476 and then that's your that's what your journey does, right? I guess maybe I can't describe this vibe but

对，没错。（笑）世界上有太多信息了，这基本上也就连到你刚才提到的、我们为什么要做世界建模。你只需要 sref，sref476，这就是 Midjourney 在做的事，对吧？我大概是没法描述这种 vibe，但是……

[29:54] Nicole

well well I think that that's kind of the point of providing some of these references, right? Because because like even just describing how someone talks and like their tone and and like procity and all of these things like I think I think some of these terms even like I didn't used to know what they mean, right? Well, now

嗯，我觉得这差不多就是提供这些参考素材的意义所在，对吧？因为哪怕只是描述一个人怎么说话，他的语气、韵律（prosody）这些东西——我觉得其中有些术语，连我以前都不知道是什么意思，对吧？嗯，现在……

[30:10] Nicole

yes. Dispuencuencies ex like like there there's kind of an entire vocabulary that even if you're not kind of steeped in a domain, which is true for actually like most human domains that like you don't even know what it means. Um and sometimes it's also a question of like if we haven't focused on those things, you know, with the large language models that they may also have gaps in those areas, right? And then we feel them on the other side with generation because we're like fundamentally relying on on the language models understanding of the world to then be able to like represent it. Um, so I yeah, it all kind of goes back to your question about like the the language as an intermediary. Um, but yeah, I think to De's point like some of these might just be like focus areas and things that we haven't necessarily pushed on as much as we can and like as we will we will discover what the actual ceiling is.

对，disfluency（言语不流畅）……就是说，存在一整套词汇，如果你没有沉浸在某个领域里——其实大多数人类领域都是这样——你根本不知道它们是什么意思。嗯，有时候这也是一个问题：如果我们在大语言模型上没有专门关注过这些东西，那模型在这些方面也可能存在缺口，对吧？然后我们会在另一端、也就是生成这一侧感受到这些缺口，因为我们从根本上就是依赖语言模型对世界的理解，才能把它表现出来。嗯，所以，对，这一切又回到了你之前那个关于「语言作为中间媒介」的问题。嗯，不过我觉得，就像 Dumi 说的，其中有些可能只是投入重点的问题，是我们还没有尽全力去推的方向，随着我们继续推进，我们会发现真正的天花板在哪里。

[30:55] 主持人

Yeah, as a podcaster I think a lot about sound.

是啊，作为一个做播客的人，我经常琢磨声音这件事。

[30:59] 主持人

Um, and and I I'll just offer a couple things for discussion in case in case it triggers anything with you guys. Um I have three domains of rough audio which is like a music voice SFX you know is that rough okay covers everything and then also even within voice let's just let's just focus on voice forget the other two um room sound like the the echoiness of like big room small room in person in a car over a phone all these like are labelable but we experience them very differently and I I often think like one of the tells of a AI video is that it is studio quality because it was recorded in a studio video because that's your training data and like and and to me that's one thing actually like the most interesting thing is just uh when I tell this is how I convince people who are kind of skeptical about the need for world models because you need it even for audio about well I'm further away from you so I should sound a little bit softer or more diffused and like the the video models need to pick that up because if they're going to do immersive video and audio you need that

嗯，我就抛几点出来讨论，万一能让你们几位想到点什么。嗯，我把音频大致分成三个领域：音乐、人声、音效（SFX）——大致是这样吧？好，基本全覆盖了。然后，就算只看人声——咱们先只聊人声，另外两个先不管——嗯，还有房间声，比如回声感：大房间、小房间、面对面、在车里、通过电话，这些都可以打上标签，但我们对它们的体验非常不同。我经常觉得，AI 视频的一个破绽就是它是录音棚音质，因为它听起来就是在录音棚里录的——你的训练数据就是这样的。对我来说，其实最有意思的一点是——这也是我用来说服那些质疑世界模型必要性的人的说法——哪怕是音频也需要世界模型：比如，我离你更远，所以我的声音听起来应该稍微轻一点、更散一点。视频模型需要把这一点学到，因为如果要做沉浸式的视频和音频，这个是必需的。

[32:01] Shane

I I I love that example of basically like studio quality or not in a way like we don't have enough language to really describe like like this kind of echoing or like some kind of noise kind of happening we just like don't have precise enough and uh if you um you know basically the reason that I think it's quite important to have like relatively information rich like kind of captioning is that we kind of rely on the natural language as a representation but if you basically don't have enough uh representation that basically means the condition on the language the generation is very multimodal and if you anything can learn from the BAE kind of like you know very old you know GMBA kind of research the idea is we really want to capture most of the stoasticity in the later representation and then the the X given the Z should be kind of like deterministic so yeah

我特别喜欢这个例子，就是到底是不是录音棚音质。某种意义上，我们没有足够的语言去真正描述这种回声，或者正在发生的某种噪声，我们的语言就是不够精确。嗯，这基本上就是我认为 captioning 要有相对丰富的信息量很重要的原因：我们依赖自然语言作为表征，但如果表征不够，就意味着以语言为条件时，生成的分布会非常多峰（multimodal）。如果说能从 VAE 那类、很早以前的 GMBA 那类研究里学到点什么，那就是：我们真正想要的，是把大部分随机性都装进潜在表征里，然后给定 Z 之后的 X 应该基本是确定性的。就这样。

[32:46] 主持人

yeah yeah um well I hope I hope there's more uh progress there and I'm sure you guys are doing

对对，嗯，希望那方面能有更多进展，我相信你们也在做……

[32:51] Nicole

I even actually like facial expressions right and maybe this gets to your point about like things that we're very sensitive to right I think you can tell a lot of AI content also just by from like people's facial expressions stressful.

其实我觉得甚至还有面部表情，对吧？这也许就说到你那个观点了，就是我们对某些东西非常敏感。我觉得很多 AI 内容光看人物的面部表情就能认出来。压力好大。

[33:04]

Yes.

对。

[33:06] Nicole

Yes. And we try not to contribute to it, but you know, um and or or like skin textures, right? Like like the things that kind of make things look real in real life. Like I you know, I can tell from the way you're nodding or from the way like your micro expressions are kind of changing of like how you're reacting to what I'm saying. Like we haven't quite crossed that chasm. I think like we're we're so much better than we were a year ago.

对。我们也尽量不去助长它，不过你知道……嗯，还有皮肤质感，对吧？就是那些让东西在现实里看起来真实的细节。比如，我能从你点头的样子、从你微表情的变化，看出你对我说的话是什么反应。这道鸿沟我们还没有完全跨过去。我觉得我们已经比一年前好太多了。

[33:28] Nicole

Yeah. Um, but there's so much more headroom kind of in a lot of those things that like we as humans are super sensitive to. And like I think image arguably probably is there because there's there's a lot of kind of images that I will see that like really do look indistinguishable from reality and I can't tell if they're generated or not.

对。嗯，但在很多我们人类超级敏感的地方，还有大得多的提升空间。我觉得图像可以说大概已经到那一步了，因为我看到的很多图片真的跟现实毫无二致，我根本分不出是不是生成的。

[33:46]

Better than reality

比现实还好

[33:47]

um or well that's a different

嗯，或者说……好吧，那是另一个……

[33:49] Dumitru

No, I I think that one of the parad [laughter] better than what I would take on my vacation as a photo. Yes. One of the one of the fun experiments that we did a while ago in the team is is like can we generate videos that are better than than real videos, right? So you just take the same caption from like oh yeah some video and then

不，我觉得其中一个……（笑）比我度假时自己拍的照片还好。对。我们团队前阵子做过一个很好玩的实验，就是：我们能不能生成比真实视频还好的视频？对吧？就是你从某个视频那里拿来同一段 caption，然后……

[34:06] Dumitru

recycle it. Yeah. Just just try to like describe a real video and then generate the equivalent version with omni and then do a human eval. How does how does it do? And then humans largely prefer AI generated

拿来复用。对。就是试着去描述一段真实视频，然后用 Omni 生成对应的版本，再做一次人工 eval，看看表现怎么样。结果人类大多更喜欢 AI 生成的

[34:18]

margin.

差距。

[34:20]

But because it's because it's the RL process,

但那是因为……因为那是 RL 过程，

[34:22]

that's the RL process working.

那就是 RL 过程在起作用。

[34:23] Dumitru

It's however you want to rationalize it. It's not necessarily the old process. It's just like I think it's just I'm not saying this is a good result. I'm just saying is we have optimized in a way that like kind of potentially sort of, you know, triggers something in the human brain that like, oh, it looks it looks all a lot of the videos just look better. Like I'm not Yeah. Yeah. Yeah. on on inspection on on deeper inspection they they would not actually be more useful or whatever but like if you just say side by side random YouTube video versus

这就看你想怎么解释了，不一定是 RL 过程的原因。我觉得就是……我不是说这是个好结果，我只是说，我们优化的方式可能在某种程度上触发了人脑里的某种东西，让人觉得：哦，看起来……很多视频就是看着更好。我不是说……对，对，对。仔细看、深入去看的话，它们其实并不会更有用什么的，但如果你只是并排对比一段随机的 YouTube 视频和

[34:53] Dumitru

generated version of it will you will just have a it will just look better because it's more it's a sharper more HDR uh you know the skin tone is is is better it's not again it's not more realistic

它的生成版本，你就会……生成的就是会看起来更好，因为它更……画面更锐利、更 HDR，嗯，你知道，肤色也更好看。再强调一下，它并不是更真实

[35:06] Dumitru

uh it doesn't solve your problem necessarily but it it looks better

呃，它不一定能解决你的问题，但它就是看起来更好。

[35:09] Shane

I I since also depend on the sensitivity of the people. Uh I was born raised in Japan and I think one thing I kind of know is like they're extremely extremely like sensitive about like you know that's why you know like architecture like food and stuff like they have.

我觉得这也取决于人的敏感度。嗯，我是在日本出生长大的，我比较了解的一点是，日本人真的极其、极其敏感，你看，这就是为什么他们会有那样的建筑、那样的食物之类的。

[35:23] Shane

Um so I talked to like a mangar like like artist there and he's like he's kind of disgusted by like the generation AI and one kind of thing he mentioned is like the eye gaze

嗯，我跟那边的一位漫画家聊过，他对生成式 AI 挺反感的，他提到的一点就是眼神，

[35:32] Shane

eye gaze that slight difference

眼神上那一点点细微的差别，

[35:35] Shane

makes me makes him kind of feel creepy about like unnatural

会让我……让他觉得有点瘆人，很不自然，

[35:38]

like if you're looking a little bit off.

比如视线稍微偏了一点。

[35:40] Shane

Yeah. It's just uh Yeah. just like uh it looks too fake. Yeah. To the point. So So I think it does depend on the sensitivity and

对。就是，呃，对，就是，呃，看起来太假了。对，就是这个点。所以，所以我觉得这确实取决于敏感度，而且

[35:47] Dumitru

Yeah. Yeah. All I'm saying is like you know human preferences are like a not particularly like uh reliable barometer of like what you should be optimizing for like if you just ask people do you like this or not you not necessarily get what you wanted.

对，对。我想说的只是，你知道，在「该优化什么」这件事上，人类偏好并不是一个特别可靠的风向标。如果你只是去问大家「你喜不喜欢这个」，你得到的不一定是你真正想要的。

[36:01]

Yeah. Let let me just kind of add one thing but like four years ago there was a like debate that if the prompt engineering is going to disappear and uh my my like you know some very powerful people say you know it's going to disappear but I basically said like it shouldn't because the prompt engineering like sort of you know specifying that is like the the only way you can sort of control the output sort of you know when you have like sort of control by the AI and what allows you to prompt engineer is really that sensitivity. So sure maybe like right now the AI can do a lot of autoprompting and that and it can generate something that's sufficient but uh if it's like that never be satisfied like never be satisfied with the AI's generated content always fine-tune your sensitivity and always kind of keep prompting the differences. I I think to the there's also a big difference between like the average human untrained eye which I I would put myself in that bucket you know like I have I have some aesthetic sensibilities and I've done this long enough that you know like I have I have a preference um but you know like your example of a manga artist like that's somebody who has honed a craft like over possibly many decades. Um, and anybody who does that, whether it's like design, architecture, right? Like you you you just have a very different level of like expertise and you see things that like the average human will not see. But Doom is right. Like when we look at if you were to just, you know,

Shane： 对。我就补充一点。四年前有过一场争论，讨论 prompt engineering 会不会消失。当时有些很有分量的人说它会消失，但我基本上是说它不应该消失，因为 prompt engineering，也就是把你想要什么讲清楚，是你把控制权交给 AI 的时候唯一能控制输出的办法；而让你能做好 prompt engineering 的，恰恰就是那种敏感度。所以当然，也许现在 AI 能做很多 autoprompting，能生成一些够用的东西，但就算是这样，也永远别满足——永远别满足于 AI 生成的内容，要一直打磨你的敏感度，一直针对那些差别继续写 prompt。

Nicole： 我觉得这里还有个很大的差别：一边是普通人那双没受过训练的眼睛——我会把自己也归进这一类，你知道，我是有一些审美感觉，这行也做得够久了，所以有自己的偏好——但另一边，就像你举的那个漫画家的例子，那是一个可能花了几十年去磨炼手艺的人。任何这样打磨过手艺的人，不管是做设计还是做建筑，对吧，专业水平完全是另一个层级，能看到普通人看不到的东西。不过 Dumitru 说得对。我们去看的时候……如果你只是，你知道，

[37:25] Nicole

um, poll 10 people on the street, they would probably prefer the like overly smooth like very saturated kind of

嗯，在街上随便找 10 个人投个票，他们大概会更喜欢那种过度平滑、饱和度很高的那种

[37:33]

It's called the Instagram filter.

那叫 Instagram 滤镜。

[37:35]

It is. It is the Yeah. [laughter] Um, and you know, and and so there's also a little bit of a question of like what does your default aesthetic look like if you don't specify? But then to Shane's point, one of the things we always try to get these models better at is instruction follow so that like when you want to get them to a different outcome like you should be able to whether that's through language or whether that's through references because language is sometimes too limiting. Um, and so like these models continue to get better at it but they so much at work. Do do you feel pressure as a as a product director to set the default for the world like I mean [laughter]

Nicole： 确实是。确实就是……对。（笑）嗯，你知道，所以这里也有一点值得琢磨：如果你什么都不指定，你的默认审美是什么样的？但回到 Shane 说的，我们一直努力让这些模型变强的一个方向就是指令遵循，这样当你想让它们产出不一样的结果时，你是能做到的，不管是通过语言，还是通过参考图，因为语言有时候局限太大了。嗯，所以这些模型在这方面一直在进步，但还有很多工作要做。

主持人： 你作为产品总监，要给全世界定下默认审美，会不会有压力？我是说……（笑）

[38:10]

kind of

算是吧。

[38:11] Nicole

maybe I should I don't know I haven't thought about this

也许我是该有点压力吧，我也不知道，我还没想过这个问题。

[38:14]

you know you know it's like someone has to have a default the default has to exist

你知道，你知道，就是总得有人来定个默认的，默认值总归得存在。

[38:18] Nicole

actually I will say like we have thought about this um and I I think one of the so for example actually like if you look at nanobanana generations we had like an explosion of nanobanana infographics when nanobanana pro came out

其实我得说，我们确实想过这个问题。嗯，我觉得其中一个……比如说，其实你去看 Nano Banana 的生成结果，Nano Banana Pro 刚出来的时候，Nano Banana 做的信息图一下子爆发了

[38:31]

I tried it yeah

我试过，对。

[38:32]

um yeah I think Nurb's papers were like all you know so so many had like infographics generated. Can you run your uh watermarking on it and see how many

嗯，对，我觉得 NeurIPS 的论文好像全都……你知道，好多好多都配了生成的信息图。你们能不能拿你们的水印检测跑一下，看看有多少

[38:41] Nicole

uh we pro we probably could we have we haven't done that but I saw so like my Twitter was maybe this is just also like the bias of my algorithm but they were everywhere um and it was actually very painful because um I think our default aesthetic was a little bit too it was too cluttered like I think that the the model is like a bit of an overeager student that just like learned you know it was like oh I know all these like I know all this information about this concept let me like shove into the same image. Japanese infographics 5x that

呃，我们应该可以，但还没做过。不过我看到……我的 Twitter 上——也可能只是我的算法有偏差——到处都是这种图。嗯，其实看着挺痛苦的，因为我觉得我们的默认审美有点太……太杂乱了。我觉得这个模型就像一个有点急于表现的学生，刚学会了点东西，就想着：哦，这些我都知道，关于这个概念的所有信息我都知道，那我全塞进同一张图里吧。日本的信息图还得再乘以五。

[39:13]

or maybe it was you know um but it just and and

或者也可能是……你知道，嗯，但它就是……而且，而且

[39:17]

wait so same prompt same content if it's in Japanese it's

等等，所以同样的 prompt、同样的内容，如果是日文的，它就会

[39:20]

density density

密度，密度。

[39:21]

oh wow

哇哦。

[39:22]

because that's the style in Japan

因为日本就是这种风格。

[39:24]

yeah some like very you know bureaucrat and [laughter] there's a famous word for it yeah

对，就是那种很……你知道，很官僚的，而且（笑）还有个很出名的词专门说这个，对。

[39:29] Dumitru

no but we do do go through this process with Omni we did it together right like where like we had like a bunch of like we like at the very end okay like this is we did some tuning and like okay what kind of style do we prefer right like you know

不，不过我们在 Omni 上确实走过这个流程，我们一起做的，对吧？就是我们有一堆……到了最后阶段，好，我们做了一些调优，然后就是：好，我们更喜欢哪种风格？对吧，你知道的

[39:41]

is it more muted more saturated

是更柔和一点，还是更饱和一点

[39:43]

we had a lot of saturation

我们那时候饱和度很高。

[39:44] Dumitru

yeah there was there were I think Nicole just has PTSD so has forgotten about it but she was very much involved in this of like okay which which kind of color palette do we basically prefer right and it's you know it's it's it's not something that like you have to make a a trade-off there like uh

对，当时是有过的……我觉得 Nicole 只是有 PTSD 了，所以把这事忘了，但她当时深度参与了，就是在定：好，我们到底更喜欢哪种配色，对吧？而且你知道，这不是那种……你就得在这上面做个取舍，就是，呃

[40:00] Nicole

and and and it's because it ends up being us right like actually it is true like it it ends up being the modeling teams and you could ask the question legitimately of like are we the best people to do that or should we actually work with someone who like has a really creative point of view and is more of like you know an art director and like has like and we kind of go back and forth on this um

而且，而且这是因为最后定这个的就是我们自己，对吧？其实确实是这样，最后就是落到模型团队头上。你完全有理由问：我们是做这件事的最佳人选吗？还是说我们其实应该找一个真正有创意视角的人合作，更像是艺术总监那样的人，而且有……这件事我们一直在反复权衡，嗯

[40:19] 主持人

we have the trusted testers I'm on

不是有 trusted testers 嘛，我就在里面。

[40:21] Nicole

we do we have trusted testers who give us a lot of feedback and we take that serious

对，我们有 trusted testers，他们给了我们大量反馈，我们也很认真对待。

[40:25] 主持人

very well organized by the way to have these like weekly calls and stuff like it's it's amazing

顺便说一句，组织得特别好，每周都有电话会之类的，真的很棒。

[40:29] Nicole

um Logan's team does a lot of that so kudo kuda kudos kudos to Logan um who couldn't be here today um and we have a lot of people actually internally at Google like Fulfur who give us like a ton of No, no, no. Truly like who give us a ton of feedback on like when we when we release new checkpoints and like sometimes it will be stuff that we like don't see right like we would be like oh yeah this optimization seems okay and then they would come back what have you done like you completely ruined my grass you know because now the detail is all blurry.

嗯，这块很多是 Logan 的团队在做，所以要给 Logan 大大点个赞，他今天没能来。嗯，其实我们在 Google 内部也有很多人，比如 fofr，他们会给我们大量……不不不，真的，他们会在我们发布新 checkpoint 的时候给我们大量反馈，有时候是一些我们自己看不出来的东西，对吧？比如我们会觉得：哦，这个优化看起来还行，结果他们回来就说：你们干了什么？你们把我的草地全毁了——因为现在细节全糊了。

[40:57]

I think he just noticed not not a super secret at this point but like that our model tends to put rings wedding rings on on on hand. That's yeah

我记得他刚发现了一件事——这事现在也不算什么大秘密了——就是我们的模型老爱往手上加戒指，婚戒。这真是，嗯

[41:04]

very strange. I had never noticed that but he's like he I just saw it and there's a faux fur channel basically.

特别怪。我之前完全没注意到，但他就……我也是刚看到的，基本上就是有个 fofr 频道。

[41:09]

Uh he posted I was like why is there wedding ring in every hand? I'm like that's strange.

呃，他发出来以后，我就想：怎么每只手上都有婚戒？我心说，这也太奇怪了。

[41:14]

That sounds very common reward hacking.

听起来就是很常见的 reward hacking。

[41:15]

Yeah. Yeah. Yeah. So but you know something that we would not have we would not have noticed necessarily while while developing this right is an oral artifact or

对，对，对。不过你看，这种东西我们在开发的时候未必会注意到，对吧？这是 RL 带来的副作用，还是……

[41:22] Shane

I I don't know you do have like a lot of preference based and then you know you may can prefer that sperious correlation reward hacking it can happen like in many weird ways. Yeah

我、我也说不准。你们确实用了大量基于偏好的数据，然后模型就可能偏好上那种虚假相关——reward hacking 会以各种奇奇怪怪的方式冒出来。嗯

[41:33]

it does. It is

确实会。确实是。

[41:34] 主持人

uh this was related to another topic that again I I try to use these mainstage things as introductions or ties in. Uh we have the eval track we have character AI and YouTube talking about how they evaluate videos. Um how do you evaluate videos

呃，这就连到另一个话题了——我还是老习惯，想把主舞台这些环节当作引子，跟其他场次串起来。呃，我们有 eval 专场，Character.AI 和 YouTube 会来讲他们怎么评估视频。嗯，那你们是怎么评估视频的

[41:51]

apart from furer [laughter]

除了靠 fofr 之外（笑）

[41:53] 主持人

not everyone has a fauxur but also you know I think there needs to be something more quantitative

不是每家都有个 fofr 啊。而且你看，我觉得总得有点更量化的办法

[41:57]

well I mean it's you improve Gemini to improve the evaluation for video. Um that's that's no no that's that's definitely one way uh it's actually very hard.

嗯，我是说，就是靠提升 Gemini 来提升视频评估。嗯，那、那——不不，那肯定是一条路，呃，只是其实非常难。

[42:08] Nicole

It's very hard. It's very hard um to get like you know audators to evaluate things in a video like including especially things like aesthetics right like that it's like there are some things that are a little bit more objective like especially when we talk like let's say we talk about images and we look at like infographics text rendering that's actually fine right because like you can kind of OCR things out and then you can look at like okay this letter is like messed up and then the whole thing is actually useless because if like literally if a letter is off in render text you just can't use that asset. Right. So th those things are like a little bit more auto ratable. Um from what we found we do rely a lot on humans looking at things and so we do do a lot of human evals. We do a lot of human evals.

真的很难。要让自动评分器（autorater）去评估视频里的东西非常难，尤其是审美这类的，对吧。有些东西会相对客观一点，特别是——比如说图像，我们看信息图、文字渲染，这个其实还好办，对吧，因为你可以用 OCR 把字识别出来，然后一看：好，这个字母画坏了，那整张图其实就没用了——说真的，渲染出来的文字哪怕错一个字母，这张素材你就根本没法用。对吧。所以这类东西相对更容易自动评分。嗯，但从我们的经验看，我们确实非常依赖人去看，所以我们做大量人工 eval。做大量人工 eval。

[42:50] Nicole

Do a lot of human ev and every time Jane is like um and every time we have a new model we like want to do more things and we want to like gem in more capabilities and then we have like more emails that we have to run. Um, and then at some point you do get two models that are like kind of close to each other and then like we literally make decisions based on like looking at outputs side by side. Sometimes like in a room like I've been in rooms where there's like 10 of us and we're just like looking at video side by side and we're like do you prefer this or do you prefer that? like oh wow it's

大量人工 eval，每次 Shane 都在那“嗯……”——每次我们出新模型，都想让它多干点事，想往里塞更多能力，然后要跑的 eval 就更多了。嗯，然后到某个时候，你确实会碰到两个模型水平很接近，那我们就真的是靠把输出并排摆着看来做决定。有时候就在一个会议室里——我待过那种屋子，十来个人，就盯着并排的视频看，互相问：你更喜欢这个，还是更喜欢那个？就是，哇，这……

[43:25] Dumitru

I mean but it is it is genuinely very complicated the more capabilities you add like you know even just the one capability but it's like almost AGI complete capabilities like video editing right like think about video editing as a and like editing with audio and

我是说，但这确实非常复杂，能力加得越多就越复杂。哪怕只是一项能力，但它是那种几乎 AGI-complete 的能力，比如视频编辑，对吧。你想想，把视频编辑当作一个……还有带音频的编辑，还有

[43:38] 主持人

my editor will be very happy to hear this

我的剪辑师听到这话会很开心的

[43:40]

edit the hardest problem in g media

编辑——生成式媒体里最难的问题

[43:43] Dumitru

I mean I don't know if it's the hardest but it's definitely there right like uh in terms of like complexity of of evaluation like free form video editing is you can do anything like

我是说，我不确定它是不是最难的，但肯定排得上号，对吧。呃，就评估的复杂度来说，自由形式的视频编辑就是——你想做什么都行，比如

[43:55] 主持人

yes uh and like I I spent a lot of money on that and it's very hard to tell me

对，呃，而且我、我在这上面花了很多钱，而且这很难说清……

[43:59] Dumitru

like adding those we don't have like add a sloth eval right like uh that we

比如加这些东西——我们可没有什么“加一只树懒”的 eval，对吧，呃，我们

[44:04]

well now we should

那现在该有了

[44:05] Dumitru

now we should yeah yeah yeah but like things like that like it's it's it's not that easy to track

现在是该有了，对对对。但像这类东西，真、真的没那么容易追踪

[44:10] 主持人

I think I'm just surprised at the sample size that you have right like to to test the entire surface of your models you still rely on a magnitude of hundreds

我只是有点惊讶你们的样本量，对吧——要测试你们模型的全部能力面，居然还是靠几百这个量级

[44:20] Nicole

no no no so we do like yeah well we do we do a ton of human evals on like on like you know thousands of things. Um I I think there's also like an element of you know we can talk about things like live experiments right like which which is also where you get signal on like like some of these more minute differences at like much larger scale then there's auto raers which is definitely kind of a more it's a very well defined space I think for LLMs much more nent for media models and then like sometimes you still do rely on human judgment and we do rely on things like feedback from people who just like have a very owned like aesthetic and and people who just like use these models in their workflows dayto-day, right? Because we could also like you could have a model that does really well on some slice of human evals, but then it like really breaks a workflow for somebody. And so this is why we do like early access programs and we try to get feedback and then we like try to incorporate it before we release something more broadly. I feel like Shane had a hot take based on his

不不不，我们其实——对，我们做了海量的人工 eval，是在几千个样本上做的。嗯，我觉得还有一块是，比如说线上实验，对吧，在那里你也能在大得多的规模上，拿到一些更细微差异的信号。然后还有 autorater，这个在 LLM 上已经是个定义得很清楚的领域了，但在媒体模型上还新得多。再然后，有时候你确实还是得靠人的判断，我们也确实很依赖一些人的反馈——那些审美特别老练的人，还有那些每天在工作流里用这些模型的人，对吧？因为你完全可能有一个模型，在某一部分人工 eval 上表现很好，结果却把某个人的工作流彻底搞崩了。所以这就是为什么我们会做抢先体验计划，想办法收集反馈，然后在更大范围发布之前把反馈吸收进去。我感觉 Shane 有个暴论要讲，从他的

[45:19] Nicole

expression always when we were talking about this

表情就看得出来，每次我们聊到这个他都这样

[45:21] Shane

every kind of human sort of you know work should be gradually kind of amortized and then the interesting thing is the video understanding especially like against like AI gener like detecting air stuff is extremely interesting uh visual task

所有人工的那类工作，都应该逐步被摊销（amortize）掉。然后有意思的是视频理解，尤其是针对 AI 生成的内容——识别 AI 做出来的东西，是个极其有意思的视觉任务

[45:35] Shane

and then like some of it kind of aesthetics or this kind of visual quality but for some of the kind of cases like semantically doesn't make sense for example you're taking like some like a famous scene from a movie and try to sort of um construct that and then if you kind of generate it uh it can generate something there but at some point some of the semantic information doesn't make sense like it's actually inconsistent. So can the AI actually detect that? So when I evaluate the AI videos like oh I feel I'm so smart you know like like AI is still kind of behind but we should make like a lot of effort. I think the video understanding is extremely uh important intelligence task uh beyond just the pure aesthetics or the preference. Um and yeah we we should always try to amatize the human

然后其中一部分是审美，或者说视觉质量这类问题；但有些情况是语义上说不通。比如你拿电影里某个经典场景，想把它重新构建出来，然后你去生成，呃，它确实能生成出点东西，但到某个地方，有些语义信息就说不通了，其实是前后不一致的。那 AI 到底能不能检测出来？所以我自己评估 AI 视频的时候，就会觉得：哦，我可真聪明——你懂吧，AI 在这方面还是有点落后，但我们应该在这上面花大力气。我认为视频理解是一个极其重要的智能任务，呃，远不只是纯粹的审美或偏好问题。嗯，所以没错，我们应该始终努力去摊销人类的

[46:22]

human label. Yeah.

人工标注。对。

[46:24] 主持人

Yeah. Um, what data do you need? A lot of people I talked to wanted to get in front of you actually. Uh, they I mean they want to be nice about it. They have a lot of video data. They have gaming data. They have real world video data. They have images. They have labelers. What do you want?

对。嗯，你们需要什么数据？其实我聊过的很多人都想来找你们。呃，他们——我是说，他们想客客气气地来谈。他们有大量视频数据，有游戏数据，有真实世界的视频数据，有图片，还有标注员。你们想要什么？

[46:44]

Are you like offering?

你这是要提供给我们？

[46:46] 主持人

I'm just like this is your request for like Okay. Okay. We get I'm sure you get a lot of pitches, right? You get a lot of people want to talk to you. what's like I think actually it's the signal is this problem this sorting out signal from noise is the main problem so creating a nice API of like okay if you actually do a b and c we are interested in that

我就是想说，这相当于你们的需求征集嘛。好，好。我们——我相信你们肯定收到很多推销，对吧？很多人都想找你们聊。所以，我觉得其实问题在于信号——从噪音里把信号筛出来才是主要难题。所以不如搞一个清晰的 API：好，只要你真能做到 A、B 和 C，我们就感兴趣。

[47:09]

um loaded question there so uh I don't know that there's like an easy like you know did you do I think we we do already have a lot of data I think it's it's

嗯，这个问题有点敏感啊。所以，呃，我不确定有没有那种简单的——比如，你有没有做过……我觉得我们已经有很多数据了。我觉得这、这

[47:19]

hard to talk about this

这个不太好聊

[47:21] 主持人

you know you want to talk about the public I don't want to get you in trouble Yeah,

你懂的，你们想聊公开的部分就好，我可不想给你们惹麻烦。对，

[47:23]

but like I think

但我觉得

[47:24]

No, no. What I just want to say is like hard to talk about this in a sort of you know without trying to without I have to think about the what I am revealing about our project and what where we're going. Um generally high quality data I think maybe maybe let's just put it this way right it's not not the secret

不不。我只是想说，这个话题很难在不……的前提下聊——我得想想我会透露出我们项目的哪些东西，还有我们的方向。嗯，总的来说是高质量数据。我觉得也许、也许就这么说吧，对吧，这也不是什么秘密

[47:40]

embodied I'm sorry

具身……不好意思

[47:42]

embodied data

具身数据

[47:43]

I mean

我是说

[47:44] Dumitru

yeah sure I mean we have sort of announced I think publicly right that we we have some sort of robotics collaboration right like so I think it's like a like or or but you because we have a robotics team at GDM so you know they're always interested in things like that um I mean for Omni specifically I think we're just quite interested just high quality data right like you know it it's not some sort of not necessarily like oh random YouTube video but like you know some a some more professional shop things like that right the things that those are those are things that we're always on the lookout for like uh and yeah

对，当然。我是说，我想我们应该公开宣布过，我们有某种机器人方面的合作，对吧？所以我觉得这算是……或者说……因为我们 GDM 有机器人团队嘛，你知道，他们一直对这类东西很感兴趣。嗯，至于 Omni 本身，我觉得我们就是对高质量数据很感兴趣，对吧？你知道，不一定是那种随便什么 YouTube 视频，而是更专业的拍摄素材之类的，对吧？这些都是我们一直在留意的东西。呃，就这样。

[48:18] Nicole

and I think for you know maybe this is easier to some extent to answer for like some of the agentic work as well like like like actual kind of like what are the tests that people are trying to do right these things are actually kind of difficult to manufacture if you're doing it yourself or if you're like doing it with a vendor, like what is the actual like if you're creating a marketing campaign, like what does that look like, right? Like do do you start from here's like a picture of my new product and then I want to turn that into a video ad and I want to turn that into a bunch of assets that like fit fit all these different ad formats that I need to push onto various platforms to promote and then like so you kind of go from this to that and like what is that kind of trajectory of tasks that you're that you're like you know experiencing along the way like that is really useful and that is actually kind of difficult to get right u because like we don't always have the right firstparty surface where people are actually doing some of these things or like you might work with someone who's a vendor but they don't also don't have that product surface right like like a lot of this kind of information lives in the places where people are doing these tasks and so that's kind of difficult to get like if anyone's figured that out you should reach out to us

而且我觉得，你知道，对一些 agentic 类的工作来说，这个问题某种程度上可能更好回答：大家实际想完成的任务到底是什么，对吧？这些东西其实挺难凭空造出来的，不管是你自己做，还是找供应商来做。比如实际情况是……假如你在做一场营销活动，那具体是什么样子，对吧？你是不是从一张新产品的照片开始，然后想把它做成一支视频广告，再把它变成一堆素材，去适配你要投放到各个平台做推广的各种广告格式？你就这样从这一步走到那一步——一路上你经历的这一整条任务轨迹是什么样的，这个非常有用，而且其实挺难拿到的。因为我们并不总有合适的第一方产品入口，让大家真的在上面做这些事；或者你可能跟某家供应商合作，但他们也没有这样的产品入口，对吧？这类信息很多都存在于人们实际做这些任务的地方，所以很难拿到。如果有谁把这个问题搞定了，欢迎来找我们。

[49:31]

every channel of thought yeah every [laughter] thought

每一条 chain of thought——对，每一个念头（笑）

[49:33]

every thought yeah and maybe the data the Chinese lab is using

每一个念头，对。说不定还有中国实验室在用的那些数据

[49:38] 主持人

yes yeah uh you know as a media person myself, right? Like there's so many podcasters and people in in marketing departments and all these like they would happy to be your data like you know just like put a BCI on my head

是的，对。呃，你知道，我自己也算是做媒体的，对吧？有那么多播客主、市场部门的人等等，他们都会很乐意给你们当数据，就是说，干脆往我头上装个 BCI（脑机接口）

[49:52] 主持人

and podcast [laughter] watch my things uh because like you know there's just endless amount of work to do like there's so much work and this is all like this needs to somewhat be commodity like obviously you can be an art like an artisan like you can be Hollywood for like the really high quality stuff but actually a lot of work is commodity and like should be modelable and we want you to do

然后录播客（笑），看着我做事。呃，因为你知道，要做的活根本做不完，活儿实在太多了，而且这些在某种程度上都得变成大路货。当然，你可以当艺术家、当手艺人，可以像好莱坞那样去做真正高质量的东西，但实际上大量工作就是大路货，应该是能被建模的，我们希望你们来做

[50:13]

[laughter]

（笑）

[50:13] Nicole

And but we we want the high quality to Demi's point right like we do want we want the high quality.

不过我们也要高质量的——就像 Dumitru 说的，对吧，我们确实要高质量的。

[50:18]

We want commodity. Yeah. Yes. Yes. You want on both sides.

我们要大路货。对。是，是。你们是两头都要。

[50:21]

Um I I just

嗯，我、我就是

[50:24]

Thank you for the solicitation.

谢谢这波吆喝。

[50:27] 主持人

Uh I you know we we we also I also added a data quality track. I I think that uh people want to understand like what uh at AI like how to raise the bar, right? like like the and a lot of it is just educating the market and educating researchers and engineers and founders on like this is where we're going a lot of this is stop doing that do this do this instead and I'm like people will listen yeah I don't know uh to that extent you know

呃，我，你知道，我们也……我还加了一个数据质量专场。我觉得大家想搞明白，在 AI 这件事上怎么把标准往上拉，对吧？而这里面很多就是去教育市场，教育研究员、工程师和创始人：这是我们要去的方向；很多时候就是“别那么做了，改成这么做”。我心想，大家总会听的吧——嗯，我也说不好，能听进去多少吧，你懂的。

[50:56] Nicole

but I think to that to that point like there's a lot of again just like craft that goes into this right and there's a lot of process like you even to the marketing campaign example you don't create that in like five minutes right you like go you go through a process and you iterate and you like pick something over something else because you liked it for whatever reason like maybe the eye gaze was correct right like we just we don't know these things right because none of us are marketing directors and like the models don't know these things

但我觉得说到这一点，这里面同样有大量的手艺在，对吧？也有大量流程。就拿营销活动那个例子来说，你不会五分钟就把它做出来，对吧？你要走一整套流程，反复迭代，你会选这个不选那个，因为出于某种原因你就是更喜欢它——比如可能是眼神的方向对了，对吧？这些东西我们根本不知道，因为我们没有一个人是市场总监，模型也不知道这些。

[51:21] Shane

I even kind of say this for the natural like a language as well like I I always kind of say 99% of information is inside people you can only extract it through active dialogue and befriending them so most of the stuff on the internet is like sort of the outcome the output of that yes but you know what are what are all the trajectories you know how did this person have this inspiration to write this paper

我甚至会说，自然语言也是这样。我老说，99% 的信息都藏在人的脑子里，只能通过主动对话、跟人交朋友才能挖出来。所以互联网上大部分东西，某种程度上只是那个过程的结果、产出。是的，但你想想，所有那些轨迹是什么？这个人是怎么有了灵感去写这篇论文的？

[51:41] Shane

what is the starting point what is the inspiration what are the dialogue that sparked it those kind of stuff is kind of inside people so even you know those kind of like even the language space is kind of that I think the creative is kind of similar as well there's a lot of dark knowledge

起点是什么，灵感是什么，是哪些对话激发了它——这类东西某种程度上都藏在人的脑子里。所以哪怕是语言空间，差不多也是这样；我觉得创意领域也类似，里面有大量的暗知识（dark knowledge）。

[51:53] Nicole

yeah it's like when you write a novel right like a novel speaks to you because like usually there's some sort of like a personal connection that you feel to like the story or the trajectory or the characters right like if you read most of the stuff that's written by LLM's today like it's, you know, it's it's it starts it falls into these like default par patterns and like the language starts to feel really similar and all the descriptions sound really similar. You can kind of like quickly read it as like, oh, this is not that interesting because like I can't connect to it, right? Um, and again, that's that's kind of like a human expertise.

对，就像写小说，对吧？一部小说之所以能打动你，通常是因为你对故事、故事走向或者人物有某种个人层面的联结，对吧？而如果你去读现在大部分由 LLM 写出来的东西，你知道，它会掉进一些默认套路里，语言开始变得很雷同，所有描写听起来都差不多。你很快就能读出来：哦，这没什么意思，因为我代入不进去，对吧？嗯，所以说到底，这又是一种人类的专业能力。

[52:26] Shane

One nice thing recently is the Google Cloud and the Google Deep Mind are kind of starting to invest a lot more in the FTEEs for the product engineers. And I also kind of saw some uh recruiting for the creative you know gem media kind of space as well. So I think those are kind of really the effort because we we kind of feel that you know what we can kind of do with a lot of public data there's limits but really you know partnering with that we can provide kind of better models and products and yeah we kind of feedback

最近有件好事：Google Cloud 和 Google DeepMind 开始在面向产品工程的 FDE（前线部署工程师）上投入多得多了。我还看到一些面向创意、你知道，生成式媒体（gen media）方向的招聘。所以我觉得这些才是真正在下功夫的地方，因为我们确实感觉到，光靠大量公开数据能做的事是有上限的，但真正跟这些人合作，我们就能提供更好的模型和产品，嗯，也能拿到反馈。

[52:50] 主持人

uh we have an FD track here for the first time every lab is announcing it. It's it's crazy. Um, one thing I'm actually very keen on doing and I push I push for this at Cognition as well is to turn the FDES not just into sales and solutions but also to EVAL's uh eval workers.

呃，我们这次第一次设了 FDE 专场，每家实验室都在官宣这个，简直疯狂。嗯，有件事我其实非常想推动——我在 Cognition 也在推——就是让 FDE 不只是做销售和解决方案，还要去做 eval，当 eval 工作者。

[53:08]

FD is not the sales FD is way way bigger than that. How do you frame FDs then? Because [laughter] I do think about it as sales like you're you know the more the more you customize the solution for

Shane： FDE 不是销售，FDE 远远不止这些。

主持人： 那你怎么定义 FDE？因为（笑）我确实是把它当销售来看的，就是说，你越是为……定制解决方案

[53:20] Shane

so I define post training as anything between the pre-training and the final user experience anything anything is a post training

所以我把后训练（post-training）定义为：从预训练到最终用户体验之间的任何环节，任何环节都算后训练。

[53:28] Shane

and to me when I first sort of you know learned a lot about I mean FD kind of I guess originally you know came from like path here and then that so I guess the kind of history is different but yeah I think the key is really that um you know the key is like not only to kind of work uh with them and ensure that they kind of know how to but also to sort of code like derive kind of insights that can basically kind of help both parties. They can put the like a lot of harness how they use the model. We can improve like very upstream. So how to get the customer feedback to the modeling I feel is the kind of more the the role I I kind of want for the fds. Yeah.

对我来说，当我最初真正开始了解……我是说 FDE，我猜它最早是从 Palantir 那边来的，然后……所以我想它的历史渊源不太一样。但我觉得关键在于——嗯，关键不只是跟他们一起干活、确保他们知道怎么……而是还要提炼出能帮到双方的洞察。他们可以围绕怎么用模型搭很多 harness，我们则可以在非常上游的地方做改进。所以怎么把客户反馈传导到建模上，我觉得这更像是我希望 FDE 扮演的角色。嗯。

[54:03] Nicole

Yeah. Yeah. and and even for sorry just on that like if you want to talk to us or at least me um I I'm not going to offer up your time um but I it's really helpful for us to actually talk to people who are using our models and like understand where they're struggling uh because again that just like it's it's the real world task that you're actually trying to use them for right like I will talk to people who do kind of interior inter interior design with some of our image models um you know and they will say hey like I really want to take this pattern pattern, but then I want to scale it across like 10 different ruck sizes and sometimes I have like a very custom ruck size and then the model fails at like replicating the pattern the same way or you know I want to do a try on for these earrings and then the earrings have a certain size and then like my head has a certain size like it has to make sense if you're actually trying to try things on and like the models kind of fail at a bunch of these things that like actually happen in the real world, right? Um and so that that's like useful for us because for some of these things like we don't think about because we don't you know we don't use the models for those tasks

对，对。而且，抱歉，就接着这个说：如果你们想跟我们聊，或者至少跟我聊——我就不替别人许诺时间了——嗯，不过能真正跟正在用我们模型的人聊一聊、了解他们卡在哪儿，对我们真的很有帮助。因为说到底，那就是你实际想用模型去完成的真实世界任务，对吧？比如我会跟一些做室内设计的人聊，他们在用我们的一些图像模型，嗯，他们会说：“嘿，我特别想拿这个图案，把它铺到 10 种不同尺寸的地毯上，有时候还有非常定制的地毯尺寸，结果模型没法用同样的方式把图案复刻出来。”或者：“我想给这副耳环做个试戴，耳环有特定的尺寸，我的头也有特定的大小——你要是真在试戴，比例就得说得通。”很多这类在真实世界里确实会发生的事，模型都会翻车，对吧？嗯，所以这对我们很有用，因为有些事情我们根本想不到——我们自己不会拿模型去做那些任务。

[55:06] Dumitru

or like um you know I think to your point about ad campaigns or whatever like people have like notions of brand languages or whatever like which is

或者说，嗯，你知道，我觉得就像你刚才说到的广告活动之类的，大家对品牌语言之类的都有自己的概念，而那个东西是

[55:13]

yes

是的

[55:13] Dumitru

like a a bunch of images or PDFs saying things you know it's a pretty kind of you know ambiguous question as well what is the IKEA brand language you know is it is it blue and yellow I mean that's that's not a very like

一堆图片或者 PDF 在讲一些东西。你知道，这其实也是个挺模糊的问题：IKEA 的品牌语言是什么？是蓝色加黄色吗？我是说，那并不是一个很……

[55:26]

but like what shade of blue you know.

但具体是哪种蓝，对吧。

[55:27]

Yeah. Yeah. Yeah. So there there's like, you know, and the brands are pretty spec, you know, pretty, you know, like they they do care about the shade of blue. It's not shouldn't just be a random blue and a random yellow. That's not going to be IKEA, right? I'm just thinking about an example. But like this is the kind of stuff that, you know, it's not necessarily part of our like, you know, developing frontier models kind of, you know, necessarily mandate, but it's something that we do want to we do want to fundamentally like build products that people will use to solve concrete tasks, not just not just research artifacts, right? So I think it's useful to understand what people do care about. Uh well, I'm sure a lot of people are very grateful for your work and there's a lot more to do that you've made so much progress over the last like even just couple years of like Nano Banana and Theo and Omni and uh I don't know what else you got cooking but we're very excited like you this is one of those things where like I was very disappointed you know when Sora shut down and and I think like there needs to be more general exploration of uh you know generative models and not just you know coding. [laughter] I think I think that is

Dumitru： 对，对，对。所以这里面有……你知道，品牌方都非常讲究，你知道，非常……他们确实在乎蓝色的具体色调。不是、不应该随便一个蓝配随便一个黄，那就不是 IKEA 了，对吧？我只是随口举个例子。但这类东西，你知道，不一定属于我们开发前沿模型的那个职责范围，但它是我们确实想做的——我们从根本上就是想打造人们会拿来解决具体任务的产品，而不只是研究产物，对吧？所以我觉得，了解大家真正在乎什么是很有用的。

主持人： 呃，好，我相信很多人都非常感激你们的工作，要做的还有很多，而你们在过去这几年，哪怕就这短短两年里，已经取得了这么大的进展，像 Nano Banana、Veo、Omni，呃，我也不知道你们还在酝酿什么，但我们都非常期待。这就是那种……比如 Sora 关停的时候，我就挺失望的，我觉得生成式模型需要更广泛的探索，而不只是，你知道，编程。（笑）我觉得、我觉得那是

[56:26]

we obviously like this.

我们当然喜欢这个。

[56:27] 主持人

We love coding. Love coding and and uh yes uh but thank you so much for your time. Uh it's been a real pleasure and I can't wait to see what this looks like next.

我们爱编程，爱编程。呃，是的，不过真的非常感谢各位抽出时间，这次交流非常愉快，我已经迫不及待想看看接下来会是什么样子了。

[56:35]

Thank you for having us. Great question.

谢谢邀请我们。问题提得很好。

[56:36] 主持人

Thank you everyone. [applause]

谢谢大家。（掌声）