在油管上刷到一个很好的访谈。是对 OpenAI 的首席科学家 Ilya Sutskever 的采访，主要讲的通用人工智能（AGI）。是的，AGI，OpenAI 会走很远……
看着他的发量就让人觉得安心，于是我认真听了一下，确实很值得分享。我请 ChatGPT 帮忙整理成了文字然后摘要，然后翻译。
先放出 ChatGPT 的翻译版本，英文版本放在最后
(00:44) 今天我有幸采访OpenAI的联合创始人兼首席科学家Ilya Sutskever。Ilya，欢迎来到The Lunar Society。谢谢，很高兴来到这里。首先一个问题，不许谦虚。
(01:02) 并非所有科学家都会在自己的领域取得重大突破，而能够在整个职业生涯中取得多个独立重大突破的科学家更是少之又少，有什么区别？是什么让您与其他研究人员不同？为什么您能在您的领域取得多个突破？谢谢你的夸奖。这个问题很难回答。
(01:21) 我非常努力，全力以赴，到目前为止效果还不错。我认为就是这样而已。明白了。为什么GPT没有更多非法用途？为什么没有更多的外国政府利用它传播宣传或欺诈老年人？也许他们还没有真正做很多这样的事情。(01:49) 但同时，如果现在有这种情况发生，我也不会感到惊讶。我可以想象他们会利用一些开源模型，尝试将其用于这个目的。当然，我预计这将是他们未来感兴趣的事情。技术上可能他们还没有想得足够多？或者没有使用他们的技术大规模实施。或者可能正在发生，这是很烦人的。
(02:09) 如果发生了这种情况，你能追踪到吗？我认为大规模追踪是可能的，是的。这需要特殊操作，但是可能的。现在AI在经济上非常有价值，比如说在飞机的规模上，但我们还没有达到AGI。
(02:29) 这个窗口有多大？很难给出一个准确的答案，而且肯定会有一个好几年的窗口。这也是一个定义的问题。因为在AI成为AGI之前，它将逐年以指数方式增长价值。回顾过去，可能只有一两年，因为那两年比以前的年份更大。
(03:03) 但我会说，已经从去年开始，AI产生了相当多的经济价值。明年会更大，之后会更大。所以我认为从现在到AGI，这段时间会有一个很好的多年时期。好吧。因为我很好奇，如果有一个使用你们模型的创业公司，在某个时候，如果你们有了AGI，世界上就只有一个企业了，那就是OpenAI。
(03:31) 任何企业还有多少时间窗口，他们实际上正在生产AGI无法生产的东西？这和问距离AGI还有多久是同一个问题。回答这个问题很困难。我犹豫不决地给你一个数字。也因为有这样一个效应，那就是研究技术的乐观人士往往低估了到达那里所需的时间。
(03:55) 但我通过思考自动驾驶汽车来使自己脚踏实地。特别是，有一个类比，如果你看看特斯拉的大小，看看它的自动驾驶行为，它看起来像是什么都能做。但同时也很明显，就可靠性而言，还有很长的路要走。我们可能在我们的模型方面也处于类似的位置，它看起来也像是能做任何事情，同时，我们还需要做一些工作，直到我们真正解决所有问题，使其变得更好、更可靠、更稳健和表现良好。到2030年，AI占GDP的百分比是多少？
(04:32) 哦，天哪，这个问题很难回答。给我一个上下限。问题是我的误差范围是对数级别的。我可以想象一个巨大的百分比，我也可以想象一个令人失望的很小的百分比。
(04:50) 假设到2030年，这些LLM创造的经济价值并不多。尽管你可能认为这不太可能，但现在你认为这种情况发生的最佳解释是什么？我真的不认为这是一个可能的可能性，这是对问题前提的声明。但是，如果我接受你的问题前提，事情在现实影响方面令人失望的原因是什么？我的答案是可靠性。
(05:22) 如果结果是你真的希望它们是可靠的，但它们最终没有实现可靠性，或者可靠性比我们预期的要难。我真的不认为这会是这样的情况。但是，如果我必须选择一个，你告诉我——嘿，为什么事情没有成功？那就是可靠性。你仍然需要查看答案并仔细核对一切。
(05:43) 这确实会对这些系统能产生的经济价值产生很大的影响。明白了。它们在技术上是成熟的，问题只是它们是否足够可靠。嗯，在某种意义上，不可靠意味着技术上不成熟。是的，说得对。
(06:03) 在生成模型之后是什么？之前，你一直在研究强化学习。这基本上是我们通向AGI的范例吗？还是说在这之后还有什么？
我认为这个范例会走得非常非常远，而且我不会低估它。这个范例很可能不是AGI的形式因素。我不愿意说下一个范例会是什么，但它可能涉及到过去所有不同想法的整合。
(06:33) 你指的是某个具体的东西吗？很难具体说明。所以你可以说，下一个token预测只能帮助我们匹配人类的表现，而不能超越它？要超越人类的表现需要什么？我质疑下一个token预测不能超越人类表现的说法。表面上看，它不能。
(06:54) 看起来好像如果你只学会模仿，预测人们会做什么，那就意味着你只能复制人们。但这里有一个反驳这种观点的反论。如果你的基础神经网络足够智能，你只需问它——一个具有极大洞察力、智慧和能力的人会做什么？也许这样的人并不存在，但很有可能神经网络能够推断出这样一个人的行为。
(07:25) 你明白我的意思吗？是的，尽管这种洞察力是从哪里得来的？如果不是从…从普通人的数据中得来的。因为如果你想一想，预测下一个token意味着什么？实际上这是一个比看起来更深刻的问题。
(07:46) 预测下一个token意味着你理解导致创建那个token的底层现实。这不仅仅是统计。虽然它是统计，但统计又是什么？为了理解那些统计数据并将它们压缩，你需要了解世界上有什么是在创造这组统计数据的？所以你说——好吧，我有了所有这些人。
(08:15) 是什么让人们产生这种行为？那么他们有思想和感觉，他们有想法，他们以某种方式行事。所有这些都可以通过预测下一个token来推断出来。我会争辩说，这应该使得在很大程度上可能说——嗯，你能猜猜如果你拿一个具有这种特征和那种特征的人会怎么做吗？这样的人可能不存在，但因为你在预测下一个token方面如此优秀，你仍然应该能猜到那个人会做什么。
(08:43) 这个假想的、想象中的人，他的心智能力远远超过我们其他人。当我们对这些模型进行强化学习时，从AI而不是人类获得强化学习数据还需要多久？
已经有大部分默认的强化学习数据来自AI了。人类被用来训练奖励函数。但是，奖励函数与模型的交互是自动的，强化学习过程中产生的所有数据都是由AI创建的。如果你看看当前受到关注的技术/范例，比如因chatGPT而备受关注的强化学习，从人类反馈中学习（RLHF）。
(09:37) 人类反馈已被用来训练奖励函数，然后奖励函数被用来创建训练模型的数据。明白了。是否有可能完全摆脱人类的参与，让它像AlphaGo一样自我改进？是的，当然。你真正希望的是人类教师与AI合作教导AI。你可能想把这想象成一个世界，在这个世界里，人类教师做1%的工作，AI做99%的工作。你不希望它是100%的AI。但你确实希望它是一个人机协作，来教导下一代机器。
我有机会尝试这些模型，它们似乎在多步推理方面表现不佳。虽然它们一直在变得更好，但要真正突破这个障碍需要什么？我认为专门的训练将帮助我们实现这一目标。更多的基础模型改进也将帮助我们实现这一目标。但从根本上说，我觉得它们在多步推理方面并不那么糟糕。实际上，我认为它们在不允许大声思考时的心智多步推理能力是不足的。
(10:47) 但是当它们被允许大声思考时，它们的表现相当不错。我预计这方面的表现将随着更好的模型和特殊训练而显著提高。在互联网上的推理token是否不足？是否有足够的推理token？
对于这个问题的背景，有人声称在某个时候，我们将用完token，一般来说，用于训练这些模型。我不认为我们会在推理token方面面临短缺。在训练大型语言模型时，互联网上仍然有大量数据可以利用。我们可以从各种领域和类型的文本中获取推理和知识，这有助于我们的模型不断地学习和改进。
当然，随着技术的发展，我们可能需要更多的高质量、多样化的数据以继续推动AI的进步。这可能意味着我们需要关注那些尚未充分利用的数据来源或创造新的数据集。然而，就目前而言，我们并未看到推理token短缺的迹象。
作为研究人员和开发人员，我们的任务是确保我们的AI系统不断地学习、推理和进步，以便为全球各行各业创造价值。通过不断地改进我们的模型和训练技术，我们可以充分利用互联网上现有的大量数据，从而推动AI领域的持续发展。
(11:14) 是的，我认为总有一天这种情况会发生，到那时，我们需要其他方法来训练模型，提高它们的能力，改善它们的行为，确保它们能精确地做到我们想要的，而不需要更多的数据。你们还没有用完数据吗？还有更多？是的，我认为数据的情况还相当不错。
(11:34) 还有很多可以继续。但是某个时候数据会用尽的。最有价值的数据来源是什么？是 Reddit、Twitter 还是书籍？你们从哪里获取大量其他类型的 tokens？通常来说，你需要那些讨论更智能话题的 tokens，更有趣的 tokens。
(11:56) 你提到的所有来源都很有价值。也许 Twitter 除外。但是我们是否需要采用多模态来获取更多 tokens？还是我们仍然有足够的文本 tokens？我认为你仍然可以在仅使用文本的情况下走得很远，但采用多模态似乎是一个非常有成果的方向。
(12:12) 如果你愿意谈论这个问题，我们还没有抓取 tokens 的地方在哪里？显然，我不能回答这个问题，但我相信对于每个人来说，这个问题都有不同的答案。我们能从算法改进中获得多少数量级的提高，而不是从规模或数据方面？这个问题很难回答，但我相信肯定有一些。
(12:36) “有一些”是很多还是一点点？只有一种方法可以找到答案。好的。让我听听你对这些不同研究方向的快速看法。检索式变压器。就是以某种方式将数据存储在模型之外，并以某种方式检索它。看起来很有前途。
(12:52) 但你认为这是一条前进的道路吗？看起来很有前途。机器人。当时 OpenAI 放弃机器人领域是正确的决定吗？是的，那是正确的。那时候，继续研究机器人是不可能的，因为数据太少了。那时如果你想研究机器人，你需要成为一家机器人公司。
(13:12) 你需要有一个非常庞大的团队来研究机器人和维护它们。即使如此，如果你要拥有100台机器人，这已经是一个庞大的运作了，但你也不会得到太多数据。所以，在大部分进展来自计算能力和数据结合的世界中，机器人领域没有数据途径。
(13:47) 所以在当时，我们做出了停止研究机器人的决定，因为没有前进的道路。现在呢？我会说现在有可能创造一条前进的道路。但是人们需要真正致力于机器人任务。你真的需要说 - 我要制造成千上万，甚至数十万、数百万的机器人，并从它们那里收集数据，找到一条逐渐的道路，在这条道路上，机器人做得越来越有用。然后使用获得的数据来训练模型，它们会做得更好。
(14:22) 你可以想象这是一个逐步改进的过程，制造更多的机器人，它们做更多的事情，收集更多的数据，依此类推。但你真的需要致力于这条道路。如果你说，我想让机器人发生改变，那就是你需要做的事情。
(14:39) 我相信有一些公司正正在做这件事。但你需要真正热爱机器人，并且愿意解决处理它们的所有物理和物流问题。这和软件完全不一样。我认为如今，在足够的动力下，人们可以在机器人领域取得进展。有哪些想法让你兴奋，但由于目前的硬件无法很好地运行，所以你无法尝试？我不认为现有的硬件是一个限制。这不是情况。
(15:05) 明白了。但是你想尝试的任何东西都可以启动吗？当然。你可能希望现有的硬件更便宜，或者说它具有更高的内存处理带宽。但总的来说，硬件根本不是问题。让我们谈谈对齐。
(15:31) 你认为我们会有一个对齐的数学定义吗？一个数学定义是不太可能的。而不是获得一个数学定义，我认为我们将获得多个定义，从不同方面看待对齐。这就是我们想要的保证。我的意思是，你可以在各种测试中观察行为，一致性，在各种对抗压力情况下观察，你可以观察神经网络是如何从内部运行的。你必须同时观察这几个因素。在模型释放到野外之前，你需要多么确定？100%？95%？这取决于模型的能力。模型能力越强大，我们需要的信心就越大。好的，假设它几乎是AGI。AGI在哪里？这取决于你的AGI能做什么。请记住，AGI是一个含糊不清的术语。你的普通大学本科生就是一个AGI，对吧？关于AGI的含义存在很大的模糊性。根据你将这个标记放在哪里，你需要更多或更少的信心。你之前提到了几种对齐的路径，现在你认为哪种是最有前景的？我认为它将是一个组合。
(16:57) 我真的认为你不会只想要一个方法。人们希望有多种方法的组合。在这里，你花费大量计算力对抗地寻找你想要教授的行为与它所展示的行为之间的任何不匹配。我们使用另一个神经网络来观察神经网络的内部，了解它是如何运作的。
(17:22) 所有这些都是必要的。每种方法都会降低不对齐的概率。而且你还希望建立在一个世界中，你的对齐程度增加得比模型的能力更快。你认为我们今天为了理解模型所采取的方法对实际超级强大的模型适用吗？它们适用的程度如何？是同样的东西在它们上面也能很好地工作吗？这不能保证。
(17:39) 我会说目前我们对我们的模型的理解仍然相当粗浅。我们取得了一些进展，但还有更多的进展是可能的。所以我希望最终，真正成功的事情是当我们有一个小型神经网络，这个神经网络被很好地理解，并被赋予了研究大型神经网络行为的任务，以便验证。
(17:59) 在什么时候，大部分AI研究都是由AI完成的？今天当你使用Copilot时，你是如何划分的？所以我预计有一天你会问你的ChatGPT的后代，你会说 - 嘿，我正在考虑这个和那个。你能给我一些建议我应该尝试的有成效的想法吗？你实际上会得到有成效的想法。
(18:22) 我不认为这会让你能够解决以前无法解决的问题。明白了。但它只是以某种方式更快地告诉人们给他们想法。它本身并没有参与研究？这只是一个例子。你可以用各种方式切分这个问题。但瓶颈在于好的想法、好的见解，这是神经网络可以帮助我们的地方。如果你要为某种对齐研究结果或产品设计一个价值十亿美元的奖金，你会为这个十亿美元的奖金设定什么具体标准？有没有什么适合这个奖金的东西？
(18:55) 有趣的是，您问的这个问题，我最近正在考虑这个问题。我还没有想出确切的标准。也许是一个奖项，我们可以说两年后、三年后或五年后回头看，就像那是主要成果一样。所以，与其说有一个评奖委员会立即决定，不如说你等待五年，然后追溯地授予奖项。
(19:21) 但没有具体的事情，我们可以确定你解决了这个特定问题，并且取得了很多进展？取得了很多进展，是的。我不会说这是全部的东西。你认为端到端训练对于越来越大的模型是正确的架构吗？或者我们需要更好的方式来连接这些东西？端到端训练非常有前景。将事物连接在一起也非常有前景。
(19:44) 一切都充满希望。所以Open AI预计在2024年的收入为十亿美元。这个数字可能非常准确，但我只是好奇，当你谈论一种新的通用技术时，你如何估计它将带来多大的意外收获？为什么是那个特定的数字？我们已经有了一个产品，从GPT-3时代开始，从两年前通过API，我们看到了它是如何成长的。我们也看到了DALL-E的反应是如何增长的。
(20:18) 你看到了ChatGPT的反应是怎样的，所有这些都给了我们可以进行相对明智的推断的信息。也许这将是一个答案。你需要有数据，你不能凭空产生这些东西，因为否则你的误差范围将在每个方向上都是100倍。
(20:42) 但是，大多数指数级增长在进入越来越大的数量时不会保持指数级，对吗？那么在这种情况下如何确定呢？你会对AI下赌注吗？在与你交谈之后，我不会。让我们谈谈一个后人工智能（AGI）时代的未来是什么样子的。我猜你正在为这个宏伟目标每周工作80小时，你真的很迷恋这个目标。(21:02) 在一个你基本生活在AI退休社区的世界里，你会感到满足吗？在AGI出现后，你个人会做些什么？关于在AGI出现后我会做什么或者人们会做什么这个问题非常棘手。人们将从何寻找意义？但我认为AI可以帮助我们解决这个问题。(21:30) 我想象的是，我们将因与AGI互动而变得更加开明，这将帮助我们更正确地看待世界，并因此在内心变得更好。想象一下与历史上最好的冥想老师交谈，这将是一件有益的事情。但我也认为，由于世界将发生很大变化，人们很难确切地理解正在发生的事情以及如何真正作出贡献。(21:59) 我认为有些人会选择成为部分AI，以便真正扩展他们的思维和理解，并真正解决社会将面临的最艰难的问题。你会成为部分AI吗？这非常诱人。你认为到公元3000年还会有实体的人类存在吗？3000年？我怎么知道公元3000年会发生什么事？这看起来像什么？地球上还有人类在四处走动吗？或者你们考虑过这个世界实际上应该是什么样子吗？(22:26) 让我向你描述一下我认为这个问题不太正确的地方。它暗示我们可以决定我们希望世界看起来像什么。我认为这个观点是错误的。变化是唯一的常态。因此，即使在建立AGI之后，这并不意味着世界将是静止的。(22:50) 世界将继续发生变化，世界将继续发展。它将经历各种各样的转变。我认为没有人知道3000年的世界会是什么样子。但我确实希望有很多人类的后代将过上幸福、充实的生活，他们可以按照自己的意愿自由生活。(23:15) 他们将是那些解决自己问题的人。我认为一个非常乏味的世界是，我们构建了这个强大的工具，然后政府说 - 好吧，这个AGI说社会应该以这样的方式运行，现在我们应该以这样的方式运行社会。我更愿意看到一个世界，人们仍然可以自由地犯错误，承受后果，并逐渐在道义上进化，以他们自己的方式向前发展，而AGI更像是一个基本的安全网。(23:47) 你花多少时间思考这些问题，而不是只做研究？我确实会花很多时间思考这些问题。它们是非常有趣的问题。我们现在的能力，在哪些方面已经超过了我们在2015年的预期？又在哪些方面还没有达到你此时此刻的预期？实际上，这正是我在2015年所期望的。(24:13) 2015年，我的想法更多的是 - 我不想对深度学习下赌注。我想尽可能地押在深度学习上。我不知道怎么样，但它会弄清楚。但是有没有什么具体的方式，你觉得它比你预期的要多或者要少？比如说从2015年开始的一些具体预测？很遗憾，我不记得我在2015年做的具体预测了。(24:37) 但我确实认为，总的来说，在2015年，我只是想搬到一个可以尽可能大地押在深度学习上的地方，但我并不知道具体情况。我没有关于七年后事情会发展到什么程度的具体想法。不过，2015年，我确实与人们进行了所有这些最好的预测，在2016年，也许是2017年，事情会发展得非常远。但是具体细节。(25:03) 所以这既是令我惊讶的，又是我提出了这些激进的预测。但也许我内心只相信其中的50%。现在你相信什么，即使OpenAI的大多数人也觉得牵强？因为我们在OpenAI进行了大量沟通，人们对我的想法有很好的了解，我们已经真正达到了在OpenAI彼此看待这些问题的程度。(25:26) 谷歌有其定制的TPU硬件，拥有来自所有用户、Gmail等的大量数据。这是否让他们在训练更大、更好的模型方面具有优势？最初，当TPU问世时，我印象深刻，认为这很了不起。但那是因为当时我对硬件了解不足。
(25:46) 实际情况是，TPU和GPU几乎是一回事。它们非常相似。GPU芯片稍大一些，TPU芯片稍小一些，也许成本稍低。但后来GPU和TPU都生产得更多，所以GPU可能最终更便宜。
(26:11) 但从根本上讲，你有一个大型处理器，还有大量的内存，两者之间存在瓶颈。TPU和GPU要解决的问题是，从内存将一个浮点数移动到处理器所需的时间，你可以在处理器上执行几百个浮点操作，这意味着你必须进行某种批处理。
(26:31) 从这个意义上讲，这两种架构是相同的。所以我真的觉得从某种程度上说，关于硬件的唯一重要事项是每次浮点运算的成本和整体系统成本。它们之间没有太大差别吗？实际上，我不知道。我不知道TPU的成本是多少，但我怀疑，如果有的话，TPU可能更贵，因为它们数量较少。
(26:58) 当你进行工作时，花多少时间配置正确的初始化？确保训练运行顺利并获得正确的超参数，又花多少时间提出全新的想法？我会说这是一个组合。提出全新想法只占工作的一小部分。
(27:21) 当然，提出新想法很重要，但更重要的是理解结果，理解现有的想法，理解正在发生的事情。神经网络是一个非常复杂的系统，对吧？你运行它，得到一些行为，这很难理解。发生了什么？理解结果，弄清楚要运行什么样的下一个实验，大部分时间都花在这上面。
(27:42) 理解可能出了什么问题，可能导致神经网络产生意料之外的结果。我会说花很多时间提出新想法。我不太喜欢这个框架。这不是说它是错误的，但主要活动实际上是理解。你认为两者之间的区别是什么？
(28:15) 至少在我心中，当你说提出新想法时，我会想到——哦，如果做这样那样怎么样？然而，理解更像是——这整个事物是什么？其中的真正潜在现象是什么？发生了哪些潜在的影响？为什么我们要以这种方式而不是另一种方式行事？当然，这与可以描述为提出想法的事物非常相邻。但理解部分是真正的行动发生地。
(28:35) 如果你回想一下像ImageNet这样的事情，那是更多的新想法还是更多的理解？那肯定是理解。那是对非常古老的东西的新理解。在Azure上进行培训的经历是怎样的？
(28:57) 很棒。Microsoft一直是我们非常好的合作伙伴。他们真正帮助将Azure带到一个非常适合机器学习的地步，我们对此非常满意。整个AI生态系统对台湾可能发生的事情有多脆弱？比如说，如果台湾发生海啸，AI的发展会受到怎样的影响？
(29:24) 这绝对会是一个重大的挫折。几年内，没有人能获得更多的计算能力。但我预计计算能力会重新涌现。例如，我相信英特尔有像几代前那样的晶圆厂。所以这意味着如果英特尔愿意，他们可以生产出类似四年前的GPU。
(29:42) 但是，这并不是最好的选择，实际上，我不确定我关于英特尔的说法是否正确，但我确实知道台湾以外还有晶圆厂，只是没有那么好。但你仍然可以使用它们，仍然可以在这方面走得很远。这只是成本问题，只是一个挫折。随着这些模型变得越来越大，推理会变得成本过高吗？
我对这个问题有一个不同的看法。推理不是变得成本过高。实际上，随着技术的发展，我们可以期待推理成本会下降。但是，随着模型变得更大、更复杂，我们可能需要更多的计算资源来支持它们。这就需要在计算能力和模型大小之间找到合适的平衡。
(30:08) 更好模型的推理成本确实会变得更高。但是这是否成为禁忌呢？这取决于它的实用性。如果其实用性超过了其成本，则不是禁止性的。打个比方，假设你想找律师谈话。你有一些案子或需要一些建议，你很乐意花费每小时400美元。对吧？如果你的神经网络能够给你可靠的法律建议，你会说——我很乐意花400美元来得到这个建议。突然推理变得非常不是禁忌了。问题是，神经网络能够以这个成本提供足够好的答案吗？是的。
(30:29) 你只需提供不同模型的价格歧视吗？今天已经是这样了。在我们的产品上，API提供了多个不同大小的神经网络，并且不同的客户使用不同大小的神经网络，具体取决于他们的使用情况。如果有人可以使用小型模型进行微调并获得满意的结果，他们将使用该模型。
(31:17) 但是，如果有人想做更复杂和更有趣的事情，他们将使用最大的模型。您如何防止这些模型变成商品，以至于这些不同的公司只是互相出价，直到GPU运行的成本基本上为止？是的，毫无疑问有一股力量正在试图创造这种情况。答案是要继续取得进展。
(31:32) 您必须继续改进模型，必须继续提出新想法，使我们的模型变得更好、更可靠、更值得信赖，这样您就可以信任它们的答案。所有这些都是。是的。但假设到了2025年，有人以成本提供2024年的模型。而且它仍然很好。
(31:53) 如果比起一年前的模型，新模型更好，为什么人们还要使用旧的模型呢？有几个答案。对于某些用例，这可能是真的。将有一个2025年的新模型，将驱动更有趣的用例。还将有推理成本的问题。如果你可以研究如何以更少的成本提供相同的模型。
(32:19) 对于不同的公司，同一模型的成本会有所不同。我还可以想象一定程度的专业化，某些公司可能会在某些领域专业化，并比其他公司更强。对我来说，这可能是对商品化的回应。随着时间的推移，这些不同公司的研究方向会趋于收敛还是分歧？他们是否会越来越相似？还是分支到不同的领域？在短期内，我认为看起来存在收敛。我预计会出现收敛-分歧-收敛的行为，即在短期内，会有很多收敛，长期会有一些分歧，但一旦长期的工作开始实现，会再次出现收敛。
(32:51) 对了，当其中一家发现最有前途的领域时，每个人都会去做…是的。当然，现在发表的论文少了，所以需要更长时间才能重新发现这个有前途的方向。但这就是我想象的情况。收敛，分歧，收敛。
(33:10) 我们一开始谈到这个话题。当外国政府了解到这些模型的能力时，你担心间谍或某种攻击会获取你的权重，或者以某种方式滥用这些模型并了解它们吗？是的，您绝对不能排除这种可能性。这是我们尽力防范的事情，但对于每个正在构建这些模型的人来说都将是一个问题。
(33:45) 您如何防止您的权重泄露？您有非常好的安全人员。有多少人有能力SSH进入具有权重的计算机？安全人员做得很好，所以我真的不担心权重会被泄露。
(34:14) 在这种规模下，您预计会有哪些新的意外属性？有什么是全新的吗？我相信一些非常新奇的属性会出现，我不会感到惊讶。我非常兴奋的事情是——可靠性和可控性。我认为这将是一个非常非常重要的紧急属性。如果您拥有可靠性和可控性，这将有助于您解决很多问题。可靠性意味着您可以信任模型的输出，可控性意味着您可以控制它。我们将看到，但如果这些新的紧急属性确实存在，那将非常酷。
(34:50) 有没有一些方法可以提前预测？在这个参数计数中会发生什么？在那个参数计数中会发生什么？我认为可以对特定能力进行一些预测，虽然这绝对不是简单的，并且至少在今天无法进行非常精细的预测。但变得更好地预测是非常重要的，任何对此感兴趣且有研究想法的人都可以做出有价值的贡献。您对这些缩放定律有多认真？有一篇论文说——需要多少数量级的增加才能得到所有推理？您是否认真对待它？还是您认为在某些时候它会失效？缩放定律告诉您下一个单词预测准确率的对数会发生什么，对吗？还有一个完全不同的挑战，即如何将下一个单词预测准确率与推理能力联系起来。我确信它们之间存在联系，但这种联系是复杂的。我们可能会发现其他东西可以为我们提供更多的推理能力。您提到了推理令牌，我认为它们可能有帮助。可能还有其他一些有帮助的东西。您是否考虑只雇用人类为您生成令牌？还是所有东西都要来自已有的资料库？我认为依赖人来教我们的模型做事情，特别是确保它们行为良好并且不会产生虚假信息，是一件非常明智的事情。这是不是很奇怪，我们恰好在同一时间拥有所需的数据、转换器和GPU？就你而言，你认为这些事情同时发生是奇怪的吗？这绝对是一个有趣的情况。我会说这在某种程度上不那么奇怪。为什么不那么奇怪呢？是因为什么驱使了数据存在、GPU存在和转换器存在？数据存在是因为计算机变得更好、更便宜，我们拥有越来越小的晶体管。
(37:04) 突然，某一刻，每个人都可以负担得起个人电脑。一旦每个人都有了个人电脑，您真的想将它们连接到网络上，您获得了互联网。一旦您拥有互联网，您就会突然出现大量数据。GPU同时改进，因为您拥有越来越小的晶体管，您正在寻找可用的事情。游戏竟然是您可以做的事情。然后在某个时候，英伟达说——游戏GPU，我可以将其变成通用GPU计算机，也许有人会发现它有用。结果它对神经网络很有用。也许GPU会在五年后，十年后到达，这可能是可能的。
(38:09) 所有这些维度的进步都是相互关联的，这并非巧合。在哪些维度上进步并不是可以自由选择的。这种进步有多么不可避免？假设你和 Geoffrey Hinton 以及其他一些先驱从未出生，那么深度学习革命是否会在大致相同的时间发生？它会有多少延迟？也许会有一些延迟。
(38:35) 也许只有一年的延迟？真的吗？很难说。我不愿意给一个更长的答案，因为GPU将会不断改进。我无法想象会有人不会发现它。因为这里还有一件事情。假设没有人做过它，计算机会变得越来越快，越来越好。
(38:57) 训练这些神经网络变得越来越容易，因为你有更大的GPU，所以训练它只需要更少的工程工作。你不需要像以前那样优化你的代码。当 ImageNet 数据集出现时，它非常巨大，使用起来非常困难。现在想象一下，你等几年，下载变得非常容易，人们只需要调整一下就可以了。
(39:18) 我猜最多只需要几年的时间。不过我不太确定，因为你不能重新运行这个世界。让我们再回到对齐的问题。作为一个深度理解这些模型的人，你对对齐有什么直觉？在当前的能力水平下，我们已经有了一套相当好的想法来对齐它们。
(39:45) 但我不会低估对齐比我们聪明的模型的难度，比如有能力误导我们的模型。这是一个需要深入研究的问题。学术研究人员经常问我他们最好的贡献是在哪里。
(40:14) 对齐研究是学术研究人员可以做出非常有意义贡献的一个领域。除此之外，你认为学术界是否会提出关于实际能力的重要洞见，或者这些洞见只会出现在公司中？公司将意识到这些能力。学术研究人员有可能提出这些见解。
(40:30) 由于某种原因，这似乎并不经常发生，但我认为这并不是学术界本身的问题。并不是学术界做不到。也许他们只是没有考虑到正确的问题或者说因为在公司中更容易看到需要解决的问题。我明白了。
(40:51) 但有可能会有人会突然意识到……我完全赞同。我怎么可能排除这种可能性呢？这些语言模型开始真正影响原子世界而不仅仅是位世界的具体步骤是什么？我认为位世界和原子世界之间并没有明确的区别。
(41:10) 假设神经网络告诉你：“嘿，你应该这样做，这会改善你的生活。”但你需要以某种方式重新安排你的公寓。然后你按照神经网络的建议重新安排了你的公寓。神经网络影响了原子世界。好吧。
(41:31) 你认为需要几个像Transformer一样重要的突破才能实现超人类AI？或者你认为我们已经在书中得到了这些见解，我们只需要实现它们并将它们联系起来？我实际上并没有看到这两种情况之间有太大的区别，我来解释一下原因。过去取得进步的方式之一是我们认识到某些东西一直具有期望的特性，但我们没有意识到。
(42:03) 这是一个突破吗？你可以说是的。这是对书中某些内容的实现吗？同样是的。我的感觉是，这些情况很可能会发生。但事后看来，这将不会像是一次突破。每个人都会说：“哦，当然。显然这种事情可以做到。”
(42:24) Transformer之所以被提出作为一个特定的进步，是因为对于几乎任何人来说，这种事情都不是显而易见的。所以人们可以说这不是他们知道的事情。让我们考虑深度学习最基本的进步，大型神经网络在反向传播的训练中可以做很多事情。哪里是新颖之处？不在神经网络中。不在反向传播中。
(42:50) 但这绝对是一个巨大的概念突破，因为很长一段时间以来，人们没有意识到这一点。但是现在，每个人都看到了，每个人都会说：“噢，当然，大型神经网络。每个人都知道它们可以做到这一点。”你对你的前导师的新的前向前向算法有什么看法？
(43:15) 我认为这是一种试图在不使用反向传播的情况下训练神经网络的方法。如果你有一个神经科学的动机，并且你想说：“好的，我怎样才能想出一些试图近似反向传播的好性质的东西而不进行反向传播？”这就是前向前向算法要做的。但是如果你只是想设计一个好的系统，那么没有理由不使用反向传播。
(44:03) 这是唯一的算法。我在不同的背景下听过你谈论将人类作为现有例子的用法，以证明AGI的存在。在什么时候你会不那么严肃地接受这个隐喻，不再追求它作为研究的动力？因为这对你来说是一个存在的例子。
(44:26) 在什么时候我会不再关心人类作为智能存在的例子？或者说在追求模型智能的过程中不再追求它作为例子？我认为受人类启发是好的，受大脑启发是好的。正确受到人类和大脑的启发是一门艺术，因为很容易抓住人类或大脑的非本质特性。
(44:59) 许多研究人员的研究旨在受到人类和大脑的启发，往往变得太具体。人们变得太——好吧，应该遵循哪个认知科学模型？同时，考虑神经网络本身的概念，即人工神经元的概念。这也受到了大脑的启发，但它却非常有成果。
(45:20) 那么，他们是如何做到这一点的？人类的行为有哪些是基本的，你可以说这证明了我们能够做到这一点？什么是本质的？实际上，这是某些更基本事物的新兴现象，我们只需要集中精力，让自己的基本功做得更好。我们可以通过受到人类智能的启发来获得灵感，但要小心。最后一个问题：为什么在你的情况下，成为深度学习革命的先驱之一并仍然成为顶尖研究人员之一之间有如此强烈的相关性？你会认为这两件事情不会那么相关。但是为什么会有这种相关性？
(46:11) 我认为这些事情并不是非常相关。老实说，这个问题很难回答。我只是一直很努力，结果到目前为止足够了。因此，这是毅力。这是必要的，但不是充分条件。许多事情需要在一起才能真正弄清楚。你需要真正去尝试，并且需要有正确的看问题的方式。回答这个问题真的很难给出一个有意义的答案。
(46:37) Ilya，与你交谈真的很快乐。非常感谢您来到The Lunar Society。我很感激您把我们带到了办公室。谢谢。是的，我真的很喜欢。非常感谢。
请 Chatgpt 生成的摘要：
Title: Ilya Sutskever on Building AGI, Alignment, Spies, Microsoft, and Enlightenment
[Introduction]
- Interview with Ilya Sutskever, Co-founder and Chief Scientist of OpenAI.
- Discusses success in making breakthroughs, potential misuse of GPT, the window for AI value, and more.
[Key Insights]
- Ilya attributes his success in making multiple breakthroughs to perseverance and giving it everything he's got.
- Large-scale tracking of illicit GPT usage is possible but requires special operations.
- There is a multi-year window of increasing economic value before AI becomes AGI.
- Mathematical definition of alignment is unlikely, but multiple definitions can help achieve better alignment.
- Bottleneck in understanding more powerful models is good ideas and insights, which neural nets could help with.
- Ilya considers the possibility of becoming part AI to expand his mind and understanding.
- Training on Azure has been a fantastic experience; Microsoft is a great partner.
- Predicting specific capabilities in models at different parameter counts is possible but not simple.
- Progress in AI is intertwined with other dimensions of improvement, such as computers becoming better and cheaper.
- Alignment of models smarter than humans is a difficult and essential research area.
[Closing Thoughts]
- Ilya highlights the importance of perseverance and the right way of looking at things to achieve success in AI research.
- Being inspired by humans and the brain is important, but it's crucial to focus on essential qualities when pursuing intelligence in models.
英文全文：
(156) Ilya Sutskever (OpenAI Chief Scientist) - Building AGI, Alignment, Spies, Microsoft, & Enlightenment - YouTube
https://www.youtube.com/watch?v=Yf1o0TQzry8
Transcript:
(00:44) Today I have the pleasure of interviewing Ilya Sutskever, who is the Co-founder and Chief Scientist of OpenAI. Ilya, welcome to The Lunar Society. Thank you, happy to be here. First question and no humility allowed.
(01:02) There are not that many scientists who will make a big breakthrough in their field, there are far fewer scientists who will make multiple independent breakthroughs that define their field throughout their career, what is the difference? What distinguishes you from other researchers? Why have you been able to make multiple breakthroughs in your field? Thank you for the kind words. It's hard to answer that question.
(01:21) I try really hard, I give it everything I've got and that has worked so far. I think that's all there is to it. Got it. What's the explanation for why there aren't more illicit uses of GPT? Why aren't more foreign governments using it to spread propaganda or scam grandmothers? Maybe they haven't really gotten to do it a lot.
(01:49) But it also wouldn't surprise me if some of it was going on right now. I can certainly imagine they would be taking some of the open source models and trying to use them for that purpose. For sure I would expect this to be something they'd be interested in the future. It's technically possible they just haven't thought about it enough? Or haven't done it at scale using their technology. Or maybe it is happening, which is annoying.
(02:09) Would you be able to track it if it was happening? I think large-scale tracking is possible, yes. It requires special operations but it's possible. Now there's some window in which AI is very economically valuable, let’s say on the scale of airplanes, but we haven't reached AGI yet.
(02:29) How big is that window? It's hard to give a precise answer and it’s definitely going to be a good multi-year window. It's also a question of definition. Because AI, before it becomes AGI, is going to be increasingly more valuable year after year in an exponential way. In hindsight, it may feel like there was only one year or two years because those two years were larger than the previous years.
(03:03) But I would say that already, last year, there has been a fair amount of economic value produced by AI. Next year is going to be larger and larger after that. So I think it's going to be a good multi-year chunk of time where that’s going to be true, from now till AGI pretty much. Okay. Because I'm curious if there's a startup that's using your model, at some point if you have AGI there's only one business in the world, it's OpenAI.
(03:31) How much window does any business have where they're actually producing something that AGI can’t produce? It's the same question as asking how long until AGI. It's a hard question to answer. I hesitate to give you a number. Also because there is this effect where optimistic people who are working on the technology tend to underestimate the time it takes to get there.
(03:55) But the way I ground myself is by thinking about the self-driving car. In particular, there is an analogy where if you look at the size of a Tesla, and if you look at its self-driving behavior, it looks like it does everything. But it's also clear that there is still a long way to go in terms of reliability. And we might be in a similar place with respect to our models where it also looks like we can do everything, and at the same time, we will need to do some more work until we really iron out all the issues and make it really good and really reliable and robust and well behaved. By 2030, what percent of GDP is AI?
(04:32) Oh gosh, very hard to answer that question. Give me an over-under. The problem is that my error bars are in log scale. I could imagine a huge percentage, I could imagine a really disappointing small percentage at the same time. Okay, so let's take the counterfactual where it is a small percentage.
(04:50) Let's say it's 2030 and not that much economic value has been created by these LLMs. As unlikely as you think this might be, what would be your best explanation right now of why something like this might happen? I really don't think that's a likely possibility, that's the preface to the comment. But if I were to take the premise of your question, why were things disappointing in terms of real-world impact? My answer would be reliability.
(05:22) If it somehow ends up being the case that you really want them to be reliable and they ended up not being reliable, or if reliability turned out to be harder than we expect. I really don't think that will be the case. But if I had to pick one and you were telling me — hey, why didn't things work out? It would be reliability. That you still have to look over the answers and double-check everything.
(05:43) That just really puts a damper on the economic value that can be produced by those systems. Got it. They will be technologically mature, it’s just the question of whether they'll be reliable enough. Well, in some sense, not reliable means not technologically mature. Yeah, fair enough. What's after generative models? Before, you were working on reinforcement learning.
(06:03) Is this basically it? Is this the paradigm that gets us to AGI? Or is there something after this? I think this paradigm is gonna go really, really far and I would not underestimate it. It's quite likely that this exact paradigm is not quite going to be the AGI form factor. I hesitate to say precisely what the next paradigm will be but it will probably involve integration of all the different ideas that came in the past.
(06:33) Is there some specific one you're referring to? It's hard to be specific. So you could argue that next-token prediction can only help us match human performance and maybe not surpass it? What would it take to surpass human performance? I challenge the claim that next-token prediction cannot surpass human performance. On the surface, it looks like it cannot.
(06:54) It looks like if you just learn to imitate, to predict what people do, it means that you can only copy people. But here is a counter argument for why it might not be quite so. If your base neural net is smart enough, you just ask it — What would a person with great insight, wisdom, and capability do? Maybe such a person doesn't exist, but there's a pretty good chance that the neural net will be able to extrapolate how such a person would behave.
(07:25) Do you see what I mean? Yes, although where would it get that sort of insight about what that person would do? If not from… From the data of regular people. Because if you think about it, what does it mean to predict the next token well enough? It's actually a much deeper question than it seems.
(07:46) Predicting the next token well means that you understand the underlying reality that led to the creation of that token. It's not statistics. Like it is statistics but what is statistics? In order to understand those statistics to compress them, you need to understand what is it about the world that creates this set of statistics? And so then you say — Well, I have all those people.
(08:15) What is it about people that creates their behaviors? Well they have thoughts and their feelings, and they have ideas, and they do things in certain ways. All of those could be deduced from next-token prediction. And I'd argue that this should make it possible, not indefinitely but to a pretty decent degree to say — Well, can you guess what you'd do if you took a person with this characteristic and that characteristic? Like such a person doesn't exist but because you're so good at predicting the next token, you should still be able to guess what that person who would do.
(08:43) This hypothetical, imaginary person with far greater mental ability than the rest of us. When we're doing reinforcement learning on these models, how long before most of the data for the reinforcement learning is coming from AI and not humans? Already most of the default enforcement learning is coming from AIs.
(09:06) The humans are being used to train the reward function. But then the reward function and its interaction with the model is automatic and all the data that's generated during the process of reinforcement learning is created by AI. If you look at the current technique/paradigm, which is getting some significant attention because of chatGPT, Reinforcement Learning from Human Feedback (RLHF).
(09:37) The human feedback has been used to train the reward function and then the reward function is being used to create the data which trains the model. Got it. And is there any hope of just removing a human from the loop and have it improve itself in some sort of AlphaGo way? Yeah, definitely. The thing you really want is for the human teachers that teach the AI to collaborate with an AI.
(10:06) You might want to think of it as being in a world where the human teachers do 1% of the work and the AI does 99% of the work. You don't want it to be 100% AI. But you do want it to be a human-machine collaboration, which teaches the next machine. I've had a chance to play around these models and they seem bad at multi-step reasoning.
(10:25) While they have been getting better, what does it take to really surpass that barrier? I think dedicated training will get us there. More and more improvements to the base models will get us there. But fundamentally I also don't feel like they're that bad at multi-step reasoning. I actually think that they are bad at mental multistep reasoning when they are not allowed to think out loud.
(10:47) But when they are allowed to think out loud, they're quite good. And I expect this to improve significantly, both with better models and with special training. Are you running out of reasoning tokens on the internet? Are there enough of them? So for context on this question, there are claims that at some point we will run out of tokens, in general, to train those models.
(11:14) And yeah, I think this will happen one day and by the time that happens, we need to have other ways of training models, other ways of productively improving their capabilities and sharpening their behavior, making sure they're doing exactly, precisely what you want, without more data. You haven't run out of data yet? There's more? Yeah, I would say the data situation is still quite good.
(11:34) There's still lots to go. But at some point the data will run out. What is the most valuable source of data? Is it Reddit, Twitter, books? Where would you train many other tokens of other varieties for? Generally speaking, you'd like tokens which are speaking about smarter things, tokens which are more interesting.
(11:56) All the sources which you mentioned are valuable. So maybe not Twitter. But do we need to go multimodal to get more tokens? Or do we still have enough text tokens left? I think that you can still go very far in text only but going multimodal seems like a very fruitful direction.
(12:12) If you're comfortable talking about this, where is the place where we haven't scraped the tokens yet? Obviously I can't answer that question for us but I'm sure that for everyone there is a different answer to that question. How many orders of magnitude improvement can we get, not from scale or not from data, but just from algorithmic improvements? Hard to answer but I'm sure there is some.
(12:36) Is some a lot or some a little? There’s only one way to find out. Okay. Let me get your quickfire opinions about these different research directions. Retrieval transformers. So it’s just somehow storing the data outside of the model itself and retrieving it somehow. Seems promising.
(12:52) But do you see that as a path forward? It seems promising. Robotics. Was it the right step for Open AI to leave that behind? Yeah, it was. Back then it really wasn't possible to continue working in robotics because there was so little data. Back then if you wanted to work on robotics, you needed to become a robotics company.
(13:12) You needed to have a really giant group of people working on building robots and maintaining them. And even then, if you’re gonna have 100 robots, it's a giant operation already, but you're not going to get that much data. So in a world where most of the progress comes from the combination of compute and data, there was no path to data on robotics.
(13:47) So back in the day, when we made a decision to stop working in robotics, there was no path forward. Is there one now? I'd say that now it is possible to create a path forward. But one needs to really commit to the task of robotics. You really need to say — I'm going to build many thousands, tens of thousands, hundreds of thousands of robots, and somehow collect data from them and find a gradual path where the robots are doing something slightly more useful. And then the data that is obtained and used to train the models, and they do
(14:22) something that's slightly more useful. You could imagine it's this gradual path of improvement, where you build more robots, they do more things, you collect more data, and so on. But you really need to be committed to this path. If you say, I want to make robotics happen, that's what you need to do.
(14:39) I believe that there are companies who are doing exactly that. But you need to really love robots and need to be really willing to solve all the physical and logistical problems of dealing with them. It's not the same as software at all. I think one could make progress in robotics today, with enough motivation. What ideas are you excited to try but you can't because they don't work well on current hardware? I don't think current hardware is a limitation. It's just not the case.
(15:05) Got it. But anything you want to try you can just spin it up? Of course. You might wish that current hardware was cheaper or maybe it would be better if it had higher memory processing bandwidth let’s say. But by and large hardware is just not an issue. Let's talk about alignment.
(15:31) Do you think we'll ever have a mathematical definition of alignment? A mathematical definition is unlikely. Rather than achieving one mathematical definition, I think we will achieve multiple definitions that look at alignment from different aspects. And that this is how we will get the assurance that we want. By which I mean you can look at the behavior in various tests, congruence, in various adversarial stress situations, you can look at how the neural net operates from the inside. You have to look at
(16:10) several of these factors at the same time. And how sure do you have to be before you release a model in the wild? 100%? 95%? Depends on how capable the model is. The more capable the model, the more confident we need to be. Alright, so let's say it's something that's almost AGI.
(16:28) Where is AGI? Depends on what your AGI can do. Keep in mind that AGI is an ambiguous term. Your average college undergrad is an AGI, right? There's significant ambiguity in terms of what is meant by AGI. Depending on where you put this mark you need to be more or less confident. You mentioned a few of the paths toward alignment earlier, what is the one you think is most promising at this point? I think that it will be a combination.
(16:57) I really think that you will not want to have just one approach. People want to have a combination of approaches. Where you spend a lot of compute adversarially to find any mismatch between the behavior you want it to teach and the behavior that it exhibits.We look into the neural net using another neural net to understand how it operates on the inside.
(17:22) All of them will be necessary. Every approach like this reduces the probability of misalignment. And you also want to be in a world where your degree of alignment keeps increasing faster than the capability of the models. Do you think that the approaches we’ve taken to understand the model today will be applicable to the actual super-powerful models? Or how applicable will they be? Is it the same kind of thing that will work on them as well or? x It's not guaranteed.
(17:39) I would say that right now, our understanding of our models is still quite rudimentary. We’ve made some progress but much more progress is possible. And so I would expect that ultimately, the thing that will really succeed is when we will have a small neural net that is well understood that’s been given the task to study the behavior of a large neural net that is not understood, to verify.
(17:59) By what point is most of the AI research being done by AI? Today when you use Copilot, how do you divide it up? So I expect at some point you ask your descendant of ChatGPT, you say — Hey, I'm thinking about this and this. Can you suggest fruitful ideas I should try? And you would actually get fruitful ideas.
(18:22) I don't think that's gonna make it possible for you to solve problems you couldn't solve before. Got it. But it's somehow just telling the humans giving them ideas faster or something. It's not itself interacting with the research? That was one example. You could slice it in a variety of ways.
(18:38) But the bottleneck there is good ideas, good insights and that's something that the neural nets could help us with. If you're designing a billion-dollar prize for some sort of alignment research result or product, what is the concrete criterion you would set for that billion-dollar prize? Is there something that makes sense for such a prize? It's funny that you asked, I was actually thinking about this exact question.
(18:55) I haven't come up with the exact criterion yet. Maybe a prize where we could say that two years later, or three years or five years later, we look back and say like that was the main result. So rather than say that there is a prize committee that decides right away, you wait for five years and then award it retroactively.
(19:21) But there's no concrete thing we can identify as you solve this particular problem and you’ve made a lot of progress? A lot of progress, yes. I wouldn't say that this would be the full thing. Do you think end-to-end training is the right architecture for bigger and bigger models? Or do we need better ways of just connecting things together? End-to-end training is very promising. Connecting things together is very promising.
(19:44) Everything is promising. So Open AI is projecting revenues of a billion dollars in 2024. That might very well be correct but I'm just curious, when you're talking about a new general-purpose technology, how do you estimate how big a windfall it'll be? Why that particular number? We've had a product for quite a while now, back from the GPT-3 days, from two years ago through the API and we've seen how it grew. We've seen how the response to DALL-E has grown as well and you see how the
(20:18) response to ChatGPT is, and all of this gives us information that allows us to make relatively sensible extrapolations of anything. Maybe that would be one answer. You need to have data, you can’t come up with those things out of thin air because otherwise, your error bars are going to be like 100x in each direction.
(20:42) But most exponentials don't stay exponential especially when they get into bigger and bigger quantities, right? So how do you determine in this case? Would you bet against AI? Not after talking with you. Let's talk about what a post-AGI future looks like. I'm guessing you're working 80-hour weeks towards this grand goal that you're really obsessed with.
(21:02) Are you going to be satisfied in a world where you're basically living in an AI retirement home? What are you personally doing after AGI comes? The question of what I'll be doing or what people will be doing after AGI comes is a very tricky question. Where will people find meaning? But I think that that's something that AI could help us with.
(21:30) One thing I imagine is that we will be able to become more enlightened because we interact with an AGI which will help us see the world more correctly, and become better on the inside as a result of interacting. Imagine talking to the best meditation teacher in history, that will be a helpful thing. But I also think that because the world will change a lot, it will be very hard for people to understand what is happening precisely and how to really contribute.
(21:59) One thing that I think some people will choose to do is to become part AI. In order to really expand their minds and understanding and to really be able to solve the hardest problems that society will face then. Are you going to become part AI? It is very tempting. Do you think there'll be physically embodied humans in the year 3000? 3000? How do I know what’s gonna happen in 3000? Like what does it look like? Are there still humans walking around on Earth? Or have you guys thought concretely about what
(22:26) you actually want this world to look like? Let me describe to you what I think is not quite right about the question. It implies we get to decide how we want the world to look like. I don't think that picture is correct. Change is the only constant. And so of course, even after AGI is built, it doesn't mean that the world will be static.
(22:50) The world will continue to change, the world will continue to evolve. And it will go through all kinds of transformations. I don't think anyone has any idea of how the world will look like in 3000. But I do hope that there will be a lot of descendants of human beings who will live happy, fulfilled lives where they're free to do as they see fit.
(23:15) Or they are the ones who are solving their own problems. One world which I would find very unexciting is one where we build this powerful tool, and then the government said — Okay, so the AGI said that society should be run in such a way and now we should run society in such a way. I'd much rather have a world where people are still free to make their own mistakes and suffer their consequences and gradually evolve morally and progress forward on their own, with the AGI providing more like a base safety net.
(23:47) How much time do you spend thinking about these kinds of things versus just doing the research? I do think about those things a fair bit. They are very interesting questions. The capabilities we have today, in what ways have they surpassed where we expected them to be in 2015? And in what ways are they still not where you'd expected them to be by this point? In fairness, it's sort of what I expected in 2015.
(24:13) In 2015, my thinking was a lot more — I just don't want to bet against deep learning. I want to make the biggest possible bet on deep learning. I don't know how, but it will figure it out. But is there any specific way in which it's been more than you expected or less than you expected? Like some concrete prediction out of 2015 that's been bounced? Unfortunately, I don't remember concrete predictions I made in 2015.
(24:37) But I definitely think that overall, in 2015, I just wanted to move to make the biggest bet possible on deep learning, but I didn't know exactly. I didn't have a specific idea of how far things will go in seven years. Well, no in 2015, I did have all these best with people in 2016, maybe 2017, that things will go really far. But specifics.
(25:03) So it's like, it's both, it's both the case that it surprised me and I was making these aggressive predictions. But maybe I believed them only 50% on the inside. What do you believe now that even most people at OpenAI would find far fetched? Because we communicate a lot at OpenAI people have a pretty good sense of what I think and we've really reached the point at OpenAI where we see eye to eye on all these questions.
(25:26) Google has its custom TPU hardware, it has all this data from all its users, Gmail, and so on. Does it give them an advantage in terms of training bigger models and better models than you? At first, when the TPU came out I was really impressed and I thought — wow, this is amazing. But that's because I didn't quite understand hardware back then.
(25:46) What really turned out to be the case is that TPUs and GPUs are almost the same thing. They are very, very similar. The GPU chip is a little bit bigger, the TPU chip is a little bit smaller, maybe a little bit cheaper. But then they make more GPUs and TPUs so the GPUs might be cheaper after all.
(26:11) But fundamentally, you have a big processor, and you have a lot of memory and there is a bottleneck between those two. And the problem that both the TPU and the GPU are trying to solve is that the amount of time it takes you to move one floating point from the memory to the processor, you can do several hundred floating point operations on the processor, which means that you have to do some kind of batch processing.
(26:31) And in this sense, both of these architectures are the same. So I really feel like in some sense, the only thing that matters about hardware is cost per flop and overall systems cost. There isn't that much difference? Actually, I don't know. I don't know what the TPU costs are but I would suspect that if anything, TPUs are probably more expensive because there are less of them.
(26:58) When you are doing your work, how much of the time is spent configuring the right initializations? Making sure the training run goes well and getting the right hyperparameters, and how much is it just coming up with whole new ideas? I would say it's a combination. Coming up with whole new ideas is a modest part of the work.
(27:21) Certainly coming up with new ideas is important but even more important is to understand the results, to understand the existing ideas, to understand what's going on. A neural net is a very complicated system, right? And you ran it, and you get some behavior, which is hard to understand. What's going on? Understanding the results, figuring out what next experiment to run, a lot of the time is spent on that.
(27:42) Understanding what could be wrong, what could have caused the neural net to produce a result which was not expected. I'd say a lot of time is spent coming up with new ideas as well. I don't like this framing as much. It's not that it's false but the main activity is actually understanding. What do you see as the difference between the two? At least in my mind, when you say come up with new ideas, I'm like — Oh, what happens if it did such and such? Whereas understanding it's more like — What is this whole thing? What
(28:15) are the real underlying phenomena that are going on? What are the underlying effects? Why are we doing things this way and not another way? And of course, this is very adjacent to what can be described as coming up with ideas. But the understanding part is where the real action takes place.
(28:35) Does that describe your entire career? If you think back on something like ImageNet, was that more new idea or was that more understanding? Well, that was definitely understanding. It was a new understanding of very old things. What has the experience of training on Azure been like? Fantastic. Microsoft has been a very, very good partner for us.
(28:57) They've really helped take Azure and bring it to a point where it's really good for ML and we’re super happy with it. How vulnerable is the whole AI ecosystem to something that might happen in Taiwan? So let's say there's a tsunami in Taiwan or something, what happens to AI in general? It's definitely going to be a significant setback.
(29:24) No one will be able to get more compute for a few years. But I expect compute will spring up. For example, I believe that Intel has fabs just like a few generations ago. So that means that if Intel wanted to they could produce something GPU-like from four years ago.
(29:42) But yeah, it's not the best, I'm actually not sure if my statement about Intel is correct, but I do know that there are fabs outside of Taiwan, they're just not as good. But you can still use them and still go very far with them. It's just cost, it’s just a setback. Would inference get cost prohibitive as these models get bigger and bigger? I have a different way of looking at this question. It's not that inference will become cost prohibitive.
(30:08) Inference of better models will indeed become more expensive. But is it prohibitive? That depends on how useful it is. If it is more useful than it is expensive then it is not prohibitive. To give you an analogy, suppose you want to talk to a lawyer. You have some case or need some advice or something, you're perfectly happy to spend $400 an hour.
(30:29) Right? So if your neural net could give you really reliable legal advice, you'd say — I'm happy to spend $400 for that advice. And suddenly inference becomes very much non-prohibitive. The question is, can a neural net produce an answer good enough at this cost? Yes.
(30:54) And you will just have price discrimination in different models? It's already the case today. On our product, the API serves multiple neural nets of different sizes and different customers use different neural nets of different sizes depending on their use case. If someone can take a small model and fine-tune it and get something that's satisfactory for them, they'll use that.
(31:17) But if someone wants to do something more complicated and more interesting, they’ll use the biggest model. How do you prevent these models from just becoming commodities where these different companies just bid each other's prices down until it's basically the cost of the GPU run? Yeah, there's without question a force that's trying to create that. And the answer is you got to keep on making progress.
(31:32) You got to keep improving the models, you gotta keep on coming up with new ideas and making our models better and more reliable, more trustworthy, so you can trust their answers. All those things. Yeah. But let's say it's 2025 and somebody is offering the model from 2024 at cost. And it's still pretty good.
(31:53) Why would people use a new one from 2025 if the one from just a year older is even better? There are several answers there. For some use cases that may be true. There will be a new model for 2025, which will be driving the more interesting use cases. There is also going to be a question of inference cost. If you can do research to serve the same model at less cost.
(32:19) The same model will cost different amounts to serve for different companies. I can also imagine some degree of specialization where some companies may try to specialize in some area and be stronger compared to other companies. And to me that may be a response to commoditization to some degree. Over time do the research directions of these different companies converge or diverge? Are they doing similar and similar things over time? Or are they branching off into different areas? I’d say in the near term, it looks like there is convergence. I expect there's going to be a convergence-divergence-convergence
(32:51) behavior, where there is a lot of convergence on the near term work, there's going to be some divergence on the longer term work. But then once the longer term work starts to fruit, there will be convergence again, Got it. When one of them finds the most promising area, everybody just… That's right.
(33:10) There is obviously less publishing now so it will take longer before this promising direction gets rediscovered. But that's how I would imagine the thing is going to be. Convergence, divergence, convergence. Yeah. We talked about this a little bit at the beginning. But as foreign governments learn about how capable these models are, are you worried about spies or some sort of attack to get your weights or somehow abuse these models and learn about them? Yeah, you absolutely can't discount that. Something that we try to guard against to the
(33:45) best of our ability, but it's going to be a problem for everyone who's building this. How do you prevent your weights from leaking? You have really good security people. How many people have the ability to SSH into the machine with the weights? The security people have done a really good job so I'm really not worried about the weights being leaked.
(34:14) What kinds of emergent properties are you expecting from these models at this scale? Is there something that just comes about de novo? I'm sure really new surprising properties will come up, I would not be surprised. The thing which I'm really excited about, the things which I’d like to see is — reliability and controllability.
(34:29) I think that this will be a very, very important class of emergent properties. If you have reliability and controllability that helps you solve a lot of problems. Reliability means you can trust the model's output, controllability means you can control it. And we'll see but it will be very cool if those emergent properties did exist.
(34:50) Is there some way you can predict that in advance? What will happen in this parameter count, what will happen in that parameter count? I think it's possible to make some predictions about specific capabilities though it's definitely not simple and you can’t do it in a super fine-grained way, at least today. But getting better at that is really important.
(35:09) And anyone who is interested and who has research ideas on how to do that, that can be a valuable contribution. How seriously do you take these scaling laws? There's a paper that says — You need this many orders of magnitude more to get all the reasoning out? Do you take that seriously or do you think it breaks down at some point? The thing is that the scaling law tells you what happens to your log of your next word prediction accuracy, right? There is a whole separate challenge of linking next-word prediction accuracy to reasoning capability. I do believe that there is a link but this link is complicated.
(35:45) And we may find that there are other things that can give us more reasoning per unit effort. You mentioned reasoning tokens, I think they can be helpful. There can probably be some things that help. Are you considering just hiring humans to generate tokens for you? Or is it all going to come from stuff that already exists out there? I think that relying on people to teach our models to do things, especially to make sure that they are well-behaved and they don't produce false things is an extremely sensible thing to do.
(36:24) Isn't it odd that we have the data we needed exactly at the same time as we have the transformer at the exact same time that we have these GPUs? Like is it odd to you that all these things happened at the same time or do you not see it that way? It is definitely an interesting situation that is the case.
(36:43) I will say that it is odd and it is less odd on some level. Here's why it's less odd — what is the driving force behind the fact that the data exists, that the GPUs exist, and that the transformers exist? The data exists because computers became better and cheaper, we've got smaller and smaller transistors.
(37:04) And suddenly, at some point, it became economical for every person to have a personal computer. Once everyone has a personal computer, you really want to connect them to the network, you get the internet. Once you have the internet, you suddenly have data appearing in great quantities. The GPUs were improving concurrently because you have smaller and smaller transistors and you're looking for things to do with them.
(37:21) Gaming turned out to be a thing that you could do. And then at some point, Nvidia said — the gaming GPU, I might turn it into a general purpose GPU computer, maybe someone will find it useful. It turns out it's good for neural nets. It could have been the case that maybe the GPU would have arrived five years later, ten years later.
(37:48) Let's suppose gaming wasn't the thing. It's kind of hard to imagine, what does it mean if gaming isn't a thing? But maybe there was a counterfactual world where GPUs arrived five years after the data or five years before the data, in which case maybe things wouldn’t have been as ready to go as they are now. But that's the picture which I imagine.
(38:09) All this progress in all these dimensions is very intertwined. It's not a coincidence. You don't get to pick and choose in which dimensions things improve. How inevitable is this kind of progress? Let's say you and Geoffrey Hinton and a few other pioneers were never born. Does the deep learning revolution happen around the same time? How much is it delayed? Maybe there would have been some delay.
(38:35) Maybe like a year delayed? Really? That’s it? It's really hard to tell. I hesitate to give a longer answer because — GPUs will keep on improving. I cannot see how someone would not have discovered it. Because here's the other thing. Let's suppose no one has done it, computers keep getting faster and better.
(38:57) It becomes easier and easier to train these neural nets because you have bigger GPUs, so it takes less engineering effort to train one. You don't need to optimize your code as much. When the ImageNet data set came out, it was huge and it was very, very difficult to use. Now imagine you wait for a few years, and it becomes very easy to download and people can just tinker.
(39:18) A modest number of years maximum would be my guess. I hesitate to give a lot longer answer though. You can’t re-run the world you don’t know. Let's go back to alignment for a second. As somebody who deeply understands these models, what is your intuition of how hard alignment will be? At the current level of capabilities, we have a pretty good set of ideas for how to align them.
(39:45) But I would not underestimate the difficulty of alignment of models that are actually smarter than us, of models that are capable of misrepresenting their intentions. It's something to think about a lot and do research. Oftentimes academic researchers ask me what’s the best place where they can contribute.
(40:14) And alignment research is one place where academic researchers can make very meaningful contributions. Other than that, do you think academia will come up with important insights about actual capabilities or is that going to be just the companies at this point? The companies will realize the capabilities. It's very possible for academic research to come up with those insights.
(40:30) It doesn't seem to happen that much for some reason but I don't think there's anything fundamental about academia. It's not like academia can't. Maybe they're just not thinking about the right problems or something because maybe it's just easier to see what needs to be done inside these companies. I see.
(40:51) But there's a possibility that somebody could just realize… I totally think so. Why would I possibly rule this out? What are the concrete steps by which these language models start actually impacting the world of atoms and not just the world of bits? I don't think that there is a clean distinction between the world of bits and the world of atoms.
(41:10) Suppose the neural net tells you — hey here's something that you should do, and it's going to improve your life. But you need to rearrange your apartment in a certain way. And then you go and rearrange your apartment as a result. The neural net impacted the world of atoms. Fair enough.
(41:31) Do you think it'll take a couple of additional breakthroughs as important as the Transformer to get to superhuman AI? Or do you think we basically got the insights in the books somewhere, and we just need to implement them and connect them? I don't really see such a big distinction between those two cases and let me explain why. One of the ways in which progress is taking place in the past is that we've understood that something had a desirable property all along but we didn't realize.
(42:03) Is that a breakthrough? You can say yes, it is. Is that an implementation of something in the books? Also, yes. My feeling is that a few of those are quite likely to happen. But in hindsight, it will not feel like a breakthrough. Everybody's gonna say — Oh, well, of course. It's totally obvious that such and such a thing can work.
(42:24) The reason the Transformer has been brought up as a specific advance is because it's the kind of thing that was not obvious for almost anyone. So people can say it's not something which they knew about. Let's consider the most fundamental advance of deep learning, that a big neural network trained in backpropagation can do a lot of things. Where's the novelty? Not in the neural network. It's not in the backpropagation.
(42:50) But it was most definitely a giant conceptual breakthrough because for the longest time, people just didn't see that. But then now that everyone sees, everyone’s gonna say — Well, of course, it's totally obvious. Big neural network. Everyone knows that they can do it. What is your opinion of your former advisor’s new forward forward algorithm? I think that it's an attempt to train a neural network without backpropagation.
(43:15) And that this is especially interesting if you are motivated to try to understand how the brain might be learning its connections. The reason for that is that, as far as I know, neuroscientists are really convinced that the brain cannot implement backpropagation because the signals in the synapses only move in one direction.
(43:37) And so if you have a neuroscience motivation, and you want to say — okay, how can I come up with something that tries to approximate the good properties of backpropagation without doing backpropagation? That's what the forward forward algorithm is trying to do. But if you are trying to just engineer a good system there is no reason to not use backpropagation.
(44:03) It's the only algorithm. I guess I've heard you in different contexts talk about using humans as the existing example case that AGI exists. At what point do you take the metaphor less seriously and don't feel the need to pursue it in terms of the research? Because it is important to you as a sort of existence case.
(44:26) At what point do I stop caring about humans as an existence case of intelligence? Or as an example you want to follow in terms of pursuing intelligence in models. I think it's good to be inspired by humans, it's good to be inspired by the brain. There is an art into being inspired by humans in the brain correctly, because it's very easy to latch on to a non-essential quality of humans or of the brain.
(44:59) And many people whose research is trying to be inspired by humans and by the brain often get a little bit specific. People get a little bit too — Okay, what cognitive science model should be followed? At the same time, consider the idea of the neural network itself, the idea of the artificial neuron. This too is inspired by the brain but it turned out to be extremely fruitful.
(45:20) So how do they do this? What behaviors of human beings are essential that you say this is something that proves to us that it's possible? What is an essential? No this is actually some emergent phenomenon of something more basic, and we just need to focus on getting our own basics right. One can and should be inspired by human intelligence with care.
(45:48) Final question. Why is there, in your case, such a strong correlation between being first to the deep learning revolution and still being one of the top researchers? You would think that these two things wouldn't be that correlated. But why is there that correlation? I don't think those things are super correlated. Honestly, it's hard to answer the question.
(46:11) I just kept trying really hard and it turned out to have sufficed thus far. So it's perseverance. It's a necessary but not a sufficient condition. Many things need to come together in order to really figure something out. You need to really go for it and also need to have the right way of looking at things. It's hard to give a really meaningful answer to this question.
(46:37) Ilya, it has been a true pleasure. Thank you so much for coming to The Lunar Society. I appreciate you bringing us to the offices. Thank you. Yeah, I really enjoyed it. Thank you very much.