HTML Is All Agents Need — James Russo, HeyGen

频道: AI Engineer
视频: https://www.youtube.com/watch?v=Cz4v1WHVyZc
原文语言: en
统计: 共 40 轮 · 主持人 1 · James Russo 39

[0:01] 主持人

[music]

[音乐]

[0:36] James Russo

Since we're talking about generative media, I thought starting with a video might be good. That video was made completely by an agent in a single shot, all utilizing HTML. My name is James Russo. I am the co-creator and tech lead of hyperframes at HeyGen. Today, we're going to be talking about why we think HTML is all your agents need to create great videos.

既然今天聊的是生成式媒体，我想先放一段视频开场应该不错。刚才那段视频完全是由一个 agent 一次性生成的，全部用 HTML 做的。我叫 James Russo，是 HeyGen 的 hyperframes 项目的共同创造者和技术负责人。今天我们要聊的是，为什么我们认为，要做出好视频，HTML 就是你的 agent 所需要的全部。

[0:58] James Russo

Show of hands here who finds creating that launch post or that launch video harder than actually building now with coding agents. All right. Some hands up there. Um Coding agents have democratized building, made it incredibly easy for anyone to create anything that comes to their minds.

举手看一下，有多少人觉得，在现在有了 coding agent 之后，写那篇发布文案、做那支发布视频，反而比真正把产品做出来还难？好，有一些手举起来了。coding agent 让开发这件事变得人人可及，任何人脑子里冒出什么想法，都能极其轻松地做出来。

[1:15] James Russo

Um however, we think that launching your product or your feature, getting it out into the world, is still quite hard. And we at HeyGen have been trying to close that gap. If you're not familiar with HeyGen, our mission is to solve communication through video. We started by creating the best AI avatar on the market.

但是我们认为，把你的产品或者功能发布出去、推到世界面前，这件事依然相当难。我们 HeyGen 一直在努力弥合这个差距。如果你不熟悉 HeyGen，我们的使命是用视频解决沟通问题。我们的起点是做出了市面上最好的 AI 数字人。

[1:35] James Russo

You can think of this as the A-roll, the footage you see here on the left side of the screen. Um it's the narrator, the character, the main subject of the video. However, it's kind of plain, and great videos have a lot more pieces to them. They have B-roll, which is the images and the the other pieces of media and assets. They have animations, they have captions, they have music. All of this is needed to create a great video.

你可以把它理解成 A-roll，也就是你在屏幕左边看到的这部分素材。它是旁白者、是角色、是整支视频的主体。但它其实挺单调的，一支好视频远不止这些。还得有 B-roll，也就是各种图片和其他媒体素材；还有动画、字幕、音乐。这些东西加在一起，才能做出一支好视频。

[2:00] James Russo

And for us, it's important that we nail every layer, not just the avatar, to solve communication through video and give agents the right canvas to create great videos. So, how do you give agents the ability to generate all of these layers and build up this composition needed to create great video? Our bet is on HTML.

对我们来说，要真正用视频解决沟通问题，要给 agent 一块合适的画布去创作好视频，就必须把每一层都做到位，而不只是数字人这一层。那么，怎么让 agent 有能力生成所有这些层次，把做一支好视频所需要的这套组合搭建起来？我们押注的是 HTML。

[2:23] James Russo

HTML, CSS, and JavaScript are the native languages of LLMs. Most of their training data, every webpage that gets scraped at the end of day is essentially just HTML, CSS, and JavaScript under the hood.

HTML、CSS 和 JavaScript 是 LLM 的母语。它们的训练数据里，绝大部分是被抓取的网页，而每一个网页说到底，底层就是 HTML、CSS 和 JavaScript。

[2:34] James Russo

Uh when you try to teach a model a new DSL or even your own custom JSON structure, it's forcing it to speak another language. Uh we like to think of this as trying to ask Shakespeare to write a poem in Japanese or Chinese.

当你想教模型一门新的 DSL，甚至只是你自己定义的一套 JSON 结构时，你其实是在逼它讲另一门语言。我们喜欢这么打比方：这就像让莎士比亚用日语或中文写诗。

[2:48] James Russo

Even if you give them a bunch of examples um and teach them a bunch of things, you will not get the best output from them because it's not their native tongue. So, why not let the LLMs and agents talk in their native tongue when creating videos?

哪怕你给它一堆示例、教它一大堆东西，你也拿不到它最好的输出，因为那不是它的母语。那么，在做视频这件事上，为什么不干脆让 LLM 和 agent 用它们的母语来表达呢？

[3:04] James Russo

We're not the only ones saying this. Over the last few months, Tarik, Andrej Karpathy have been talking a lot about how HTML is a new markdown. It's a great output um for LLMs to give you visual representation. For what's it's worth, I submitted this talk before both of these tweets came out. Uh I can show you proof later if you're curious, but we all kind of came to the same conclusion here, which is that uh let LLMs talk in their native language and you can get better output from them.

这么说的不止我们。过去几个月里，Tarik 和 Andrej Karpathy 都在大谈 HTML 就是新的 markdown。对 LLM 来说，它是一种很棒的输出格式，能给你一个可视化的呈现。说明一下，我这个演讲的提案是在这两条推文出现之前就提交的。你要是好奇，我待会儿可以拿证据给你看。总之我们都得出了同一个结论：让 LLM 用它的母语来表达，你就能从它那里拿到更好的输出。

[3:34] James Russo

It wasn't necessarily an easy thing. We've been working on this for over a year now and tried a bunch of things along the way. Uh there's a bunch of different uh frameworks and tools to help create great videos. And we like to think about them uh on this axis of quality of results and agent-friendly.

这件事并不轻松。我们已经在这上面做了一年多，中间试过一大堆方案。市面上有各种各样的框架和工具来帮你做好视频，我们喜欢把它们放在这两个维度上看：产出质量，以及对 agent 是否友好。

[3:53] James Russo

Um for us, After Effects, Premiere Pro, these are kind of like the gold standards of creatives. They create great output, but they're not very agent-friendly. And even with the more recent connectors into it, it's more of a a co-pilot or something that can help you facilitate things you already know how to do. It doesn't give it creative output.

在我们看来，After Effects、Premiere Pro 这些算是创意工作者的黄金标准。它们产出质量很好，但对 agent 一点也不友好。哪怕最近有了各种接进去的连接器，那也更像是一个副驾驶，帮你更顺手地完成你本来就会做的事。它给不了你创意性的产出。

[4:12] James Russo

We tried things like Lottie and Rive, which are coding uh languages in JSON or custom XML formats. Um and they can get you pretty good output, but agent-friendly isn't necessarily true for those because they're not um their native tongue and they aren't as editable or controllable, which is a big thing for us is that controllability layer.

我们试过 Lottie 和 Rive 这类东西，它们本质上是用 JSON 或者自定义 XML 格式来编码的。它们能给你相当不错的产出，但说它们对 agent 友好就不见得了，因为那不是 agent 的母语，而且可编辑性和可控性都不够。可控性这一层对我们来说是件大事。

[4:36] James Russo

We played around with Remotion quite a bit and honestly thought it was a great example of what LLMs and agents could do with coding. Um however, we noticed that we had to teach them the framework. We had to teach them the language and give them a bunch of examples on how to write code properly, which ultimately took away a lot of the creativity of it.

我们还比较深入地折腾过 Remotion，老实说，我觉得它是一个很好的例子，展示了 LLM 和 agent 用代码能做到什么程度。但我们发现，我们得教模型这个框架，得教它这门语言，还得给它一堆示例来告诉它代码该怎么写才对，这最终把创造力消磨掉了很多。

[4:57] James Russo

Um then we have HTML, which around November of last year when Gemini 3 and the latest models came out, we saw a step function improvement in what LLMs could do. When we just gave them examples of what the output we wanted was, they naturally gravitated toward HTML, CSS, and JavaScript and gave us great output. Um and we decided, let's not fight the model, but find a way where we can let them talk in their native tongue.

然后就是 HTML。大概在去年十一月，Gemini 3 和其他最新模型出来的时候，我们看到 LLM 的能力出现了一次阶跃式的提升。当我们只是把想要的输出效果做成示例给它看，它自然而然就往 HTML、CSS 和 JavaScript 上靠，而且给了我们很好的产出。于是我们决定：别跟模型对着干，而是想办法让它用母语来表达。

[5:24] James Russo

So, how we did this was starting with a very small model, Gemini 3 Flash, as our design partner. We knew that if the smaller models could author workable code in a framework, then the larger models and these coding agents could 100% do it as well.

我们的做法是，先拿一个很小的模型 Gemini 3 Flash 作为设计伙伴。我们心里清楚，如果连小模型都能在一个框架里写出能跑的代码，那大模型和这些 coding agent 就百分之百也能做到。

[5:42] James Russo

We tried with a bunch of different wrappers around HTML, CSS, and JavaScript, adding in a lot of context, and making our system prompts bigger, adding in skills. Um but to our surprise, the thinnest wrapper ultimately won, which is essentially just HTML at the end of the day with a few data attributes as metadata to let the agent know, and to let us know uh about timing and things like that.

我们在 HTML、CSS 和 JavaScript 外面套了各种不同的封装层试，往里塞大量 context，把 system prompt 越写越长，还加了 skill。但让我们意外的是，最后胜出的是最薄的那层封装——说到底就是纯 HTML，再加几个 data 属性当元数据，用来让 agent 知道、也让我们知道时间轴之类的信息。

[6:05] James Russo

Um our Yeah, as I already mentioned, our thinking was very simple. If this smaller model could get it, as the models got better, it would continue to improve. They would naturally understand it more as well as training data came into the picture. Um and this is when we knew the format of what we were building was right.

就像我刚才说的，我们的想法很简单：如果这个更小的模型都能搞定，那随着模型越来越强，效果只会继续变好。而且随着训练数据不断进来，它们对这套东西的理解也会越来越自然。就是在这个时候，我们确认了自己在做的这套格式是对的。

[6:24] James Russo

Here's a little bit more detailed example of what a video looks like uh in hyperframe. So, here is on the left just HTML um with a few attributes, and on the right, the web page that it renders. Um the preview and the render are all done in the browser. The same pixels that the browser sees is ultimately what your video is going to see.

这里给一个更详细的例子，展示 hyperframe 里一支视频长什么样。左边就是 HTML，带几个属性；右边是它渲染出来的网页。预览和渲染全都在浏览器里完成。浏览器看到的是哪些像素，你的视频最终看到的就是哪些像素。

[6:45] James Russo

Um and anything any web page that your LLM or agent knows how to write, it can now write into a video. This gave us hyperframes, our open-source framework that turns your agent's HTML into video. So, now that we had the HTML and the language part of it, um the next hard part was how do we actually turn this into a deterministic MP4 video that anyone can post or utilize. And this was a lot harder because browsers are async on purpose. Um they have a different set of requirements and concerns to video rendering. They need to work across a bunch of different networks and a bunch of different uh constraints.

而且，只要是你的 LLM 或 agent 会写的网页，它现在都能写进视频里。这就有了 hyperframes——我们的开源框架，把你 agent 写的 HTML 变成视频。所以，有了 HTML 和语言这一部分之后，接下来的难题是：怎么把它真正变成一个确定性的 MP4 视频，让任何人都能发布或者使用。这难度大得多，因为浏览器天生就是异步的。浏览器面对的需求和顾虑跟视频渲染完全不同，它得在各种各样的网络环境和各种各样的约束下都能正常工作。

[7:28] James Russo

But, that's not true for video. So, on the left you can kind of see here what a a browser render might do. It's okay to loading things asynchronously like fonts. So, as the page loads in, the font and style of the text might change.

但视频不是这样的。左边你大概能看到浏览器渲染是什么样：它可以接受像字体这样的东西异步加载。所以随着页面加载，文字的字体和样式可能会变。

[7:42] James Russo

Images and videos and other assets can load in asynchronously as well. Um so you might not see them initially on the page, but they'll come in at some point. Whereas for video, we need everything on the page 100% of the time so that we can show you exactly what you expect in the the video.

图片、视频和其他素材也可以异步加载进来。所以一开始你在页面上可能看不到它们，但过一会儿它们就会出现。而对视频来说，我们需要页面上的所有东西百分之百的时间都在那儿，这样我们才能在视频里精确呈现你所期待的画面。

[8:01] James Russo

How we do this in hyperframes is um essentially freezing the clock and seeking frame by frame. For those who aren't super familiar with video, uh basically a video is a series of frames or images strung together to create motion across an entire video.

在 hyperframes 里，我们的做法本质上是把时钟冻住，然后一帧一帧地跳转。对不太熟悉视频的人解释一下，视频基本上就是一连串帧、也就是一连串图片串起来，从而在整支视频里形成动态效果。

[8:18] James Russo

So we took this insight of ours and basically applied it to our rendering of HTML as well. We freeze the clock in the browser and then we seek deterministically to every single moment in time or every single frame, uh wait for everything to load on the page, ensure that it's loaded and ready to go, and then we take a screenshot and move on to the next frame, and do that over and over again uh until we get all of the necessary frames to encode that into a video. So the same input that is previewed in the browser is also rendered into the video.

所以我们把这个洞察拿过来，同样应用到 HTML 的渲染上。我们把浏览器里的时钟冻住，然后确定性地跳转到每一个时间点、每一帧，等页面上所有东西加载完，确认它已经加载好、准备就绪，然后截一张图，再跳到下一帧，如此反复，直到拿到所有需要的帧，把它们编码成一支视频。这样一来，在浏览器里预览的那份输入，和渲染进视频里的是同一份。

[8:52] James Russo

Hence the name hyperframes. The power of this is that anything you can render in a browser essentially can now be in your video. Things like 3.js, charts, SVGs, shaders, WebGL, WebGPU, Lottie, all of these are renderable in the browser, and therefore all of them are renderable in hyperframes.

这也就是 hyperframes 这个名字的由来。它的威力在于，任何你能在浏览器里渲染出来的东西，现在基本上都能进到你的视频里。像 three.js、图表、SVG、shader、WebGL、WebGPU、Lottie，这些在浏览器里都能渲染，因此在 hyperframes 里也全都能渲染。

[9:13] James Russo

This is a big part of how we create a lot of our videos is finding inspiration on the internet, taking these as examples, tweaking them to our needs, and then putting them into our videos. The next part outside of the actual framework and rendering is how do you get great or good output out of the agents. And a big part of this is the skills that we couple with our framework.

我们做很多视频时，很大一部分就是靠这个：上网找灵感，把找到的东西当例子，按自己的需要改一改，然后放进我们的视频里。除了框架和渲染本身之外，下一个问题是：怎么让 agent 输出的东西够好、够棒？这里很大一部分靠的是我们跟框架配套的 skill。

[9:37] James Russo

Our skill is focused on taste and video aspects because the LLMs and agents already know how to write HTML and CSS and JavaScript, we don't have to teach them the language, we just teach them how to create good videos. This is a big difference between other frameworks where if you look at certain skills, it's really just like how do you write something in that framework?

我们的 skill 专注在审美和视频本身的门道上，因为 LLM 和 agent 本来就会写 HTML、CSS 和 JavaScript，我们不用教它们语言，只需要教它们怎么做出好视频。这是跟其他框架很大的一个区别——你去看某些框架的 skill，内容其实基本就是：在这个框架里东西该怎么写。

[9:57] James Russo

Um this allows us to focus on the important parts of what makes a great video and a big part of this is constantly evaling and using agents to improve them. Uh this allows us to raise the floor of videos and ensure that the base output of a single shot prompt gets you pretty good results.

这让我们能把精力集中在真正决定一支好视频的关键部分上，而其中很重要的一环，是持续做 eval，并且用 agent 来不断改进这些 skill。这样我们就能把视频的下限抬高，确保一次性 prompt 出来的基础产出就已经相当不错了。

[10:18] James Russo

Here's an example I'm going to show of a single shot utilizing our website to video skill. All you have to do is give it a website a We tell it how to get to that website, retrieve all the assets and information it needs uh to create that design and theming and branding using DesignMD or FrameMD in our case.

下面我要演示一个例子，一次性调用我们的 website to video skill。你要做的就是给它一个网址。我们会告诉它怎么访问那个网站，把它需要的所有素材和信息抓回来，用来还原那套设计、配色主题和品牌调性——在我们这里用的是 DesignMD，或者说 FrameMD。

[10:38] James Russo

Um and then from there we just teach the LLM basic motion examples and things like that of great videos that you can then go ahead and get good output like this. However, great output takes craft. Similar to AI coding, you can get decent output by just giving a single prompt and having something that works for your needs, but getting great output from agents requires craft, taste, the same principles of any uh software engineer before AI coding as well.

然后从这一步开始，我们只需要教 LLM 一些基础的动效示例之类的东西，让它知道好视频是什么样，接着你就能跑出像这样不错的产出。不过，真正出色的产出是需要功力的。这跟 AI 写代码很像：你随手给一个 prompt，也能拿到还过得去的结果，做出一个够你用的东西；但想从 agent 那里拿到出色的产出，就需要功力、需要品味，跟 AI 编程出现之前，任何一个软件工程师所需要的那套原则是一样的。

[11:14] James Russo

Breaking the problem up into individual pieces and working iteratively. The same is true for getting great output for HyperFrames videos. Um we want to make sure our power users have full control over this and the same way we created videos before HyperFrames is how we create them with HyperFrames.

把问题拆成一个个小块，然后一步步迭代着往前推。想用 Hyperframes 做出好视频，道理也一样。我们希望重度用户对这套流程有完全的掌控权——在有 Hyperframes 之前我们怎么做视频，用上 Hyperframes 之后还是怎么做。

[11:31] James Russo

We think about the narrative and the vision and the mission of this video. We storyboard it frame by frame and think about what each frame needs to do. We then move that into adding motion frame by frame utilizing HTML, CSS, and JavaScript, merging it into one cohesive video, um and then utilizing our studio that we released with Hyperframes for that last mile editing. Ensuring that humans are always in the loop and have access to do anything that they would do in their normal video editor within the open-source framework's editor so that you can manually drag, tweak, uh etc.

先想清楚这个视频的叙事、愿景和它要完成的使命。然后一帧一帧地做分镜，想清楚每一帧各自要承担什么。接着把这些搬进来，用 HTML、CSS 和 JavaScript 一帧一帧地加上动效，再合并成一个连贯完整的视频，最后用我们随 Hyperframes 一起发布的 studio 做最后一公里的精修。这样能保证人始终在流程里（human in the loop）：在这个开源框架的编辑器里，你平时在普通视频剪辑软件里会做的事，在这儿一样都能做——手动拖拽、微调，等等。

[12:05] James Russo

And that gets you output that looks like this. Uh we actually just released this today, keyframes in Hyperframes, uh which is a big aspect of what makes great videos for motion designers, and it allows you to basically coordinate all of this different motion uh frame by frame or keyframe by keyframe um in our studio and make sure that you can do anything that a professional motion designer might be able to do in After Effects.

做出来的效果就是这样。这个功能我们其实今天刚发布——Hyperframes 里的关键帧（keyframes）。对动效设计师来说，这是做出优秀视频非常关键的一环。有了它，你基本上就能在我们的 studio 里一帧一帧、或者说一个关键帧一个关键帧地去编排所有这些不同的动效，确保一个专业动效设计师在 After Effects 里能做的事，在这里也都能做。

[12:35] James Russo

So, it's not just a demo. We've been uh we released Hyperframes a couple months ago. It's been running at scale in our opinion. Over 1.3 million videos have been rendered by open-source users of Hyperframes in the last 90 days.

所以这不只是个 demo。Hyperframes 我们几个月前就发布了，在我们看来，它已经在大规模地跑了。过去 90 天里，Hyperframes 的开源用户一共渲染了超过 130 万个视频。

[12:47] James Russo

267,000 creators have tried it. We have about 15,000 videos every single day being rendered utilizing the open-source framework and 32,000 GitHub stars. Um but this is only just the start for us. As I mentioned already, it's open-source, free forever. Anyone can go ahead and use Hyperframes right now. It works with any coding agent that you have, Cloud Code, Codex, Cursor, any number of other ones up here, any ones I'm not aware of even. If your agent knows how to write HTML, CSS, and JavaScript, it knows how to create a Hyperframes video.

有 26.7 万名创作者试过它。每天大约有 1.5 万个视频是用这个开源框架渲染出来的，GitHub 上拿到了 3.2 万个 star。但对我们来说，这只是个开始。前面说过，它是开源的，永久免费，任何人现在就可以直接上手用 Hyperframes。它跟你手上任何一个 coding agent 都能配合：Claude Code、Codex、Cursor，还有屏幕上列的其他一堆，甚至包括我自己都还不知道的那些。只要你的 agent 会写 HTML、CSS 和 JavaScript，它就会做 Hyperframes 视频。

[13:19] James Russo

Now, with this, the same agent that is helping you create your product can also help you make your launch video. The one honest thing that we're going to say here is that the models still aren't good at creative work. We spend a lot of time evaling and trying to improve this in our skills and push it even further.

这样一来，那个帮你做产品的 agent，同时也能帮你做产品的发布视频。这里有句实话我们得说在前面：模型在创意类的活儿上还是不够好。我们花了大量时间做 eval，想在我们的 skills 里把这一块做得更好，再往前推一推。

[13:38] James Russo

But we think there's something at a higher level that needs to change here, which is why we started to work on a code to video benchmark where we are trying to work with the LLM labs, any creators who are working on video agents to ensure that we can raise the floor of videos for everyone. If anyone is interested in this space or working on this, we'd be happy to be collaborators, talk more about this. Feel free to find me or any member wearing a Hyperframes t-shirt in the crowd afterwards. Love to chat to you guys more.

但我们觉得，还有更高一层的东西需要改变，所以我们开始做一个 code-to-video 的 benchmark。我们想和各家 LLM 实验室、以及所有在做视频 agent 的创作者合作，一起把大家做视频的下限往上抬。如果有人对这个方向感兴趣，或者正在做这件事，我们很乐意一起合作、多聊聊。会后欢迎随时来找我，或者找现场任何一位穿 Hyperframes T 恤的同事。很期待跟大家多交流。

[14:09] James Russo

And one more thing, some of you may have seen the AI engineer Warfare showcase video. We collaborated heavily with the team to create this with them utilizing Hyperframes as a little surprise of what it can do.

还有一件事。你们中有些人可能看过 AI Engineer World's Fair 的宣传展示视频。那支片子是我们和大会团队深度合作、用 Hyperframes 一起做出来的，算是给大家留的一个小惊喜，让你们看看它到底能做到什么程度。

[14:22] James Russo

Here's a great example of the motion graphics and designing that you can do with HTML, CSS, and JavaScript. So yeah, agents are made building incredibly easy. Launching is still quite hard. We think HTML is all your agents need in order to make great videos and launch your product into the world.

这就是个很好的例子，说明用 HTML、CSS 和 JavaScript 能做出什么水准的动态图形和设计。所以说，agent 已经让「做产品」变得无比容易，但「把产品发出去」依然相当难。我们认为，要做出好视频、把你的产品推向世界，你的 agent 需要的就只有 HTML。

[14:42] James Russo

Here is a link to Hyperframes, the project, the open source repo. Feel free to check it out, star it, download the skills. You can reach me on X at Reems_Juso, and thank you for attending. I'll be outside if anyone wants to chat.

这是 Hyperframes 这个项目、也就是开源仓库的链接。欢迎去看看、点个 star、把 skills 下下来用。想找我的话，可以在 X 上私信 @Reems_Juso。谢谢大家来听我讲，我一会儿就在外面，想聊的随时来找我。