更快的模型，真的让任务更快完成了吗？MiniMax Morgan Suo 谈完整任务评估

简要说明Brief Description

Morgan Suo 探讨了稀疏注意力、低精度计算、服务负载和推理预算如何影响生成一份分析师能够核验和使用的研究报告所需的时间和成本。Morgan Suo examines how sparse attention, lower-precision computation, serving load and reasoning budgets affect the time and cost of producing a research report that an analyst can verify and use.

目录Table of Contents

• 引言
  Intro
• 研究工作
  The research job
• 利润与现金流量
  Profit vs. cash flow
• 模型内部发生了哪些变化
  What changed inside the model
• 长输入上的稀疏注意力
  Sparse attention on long inputs
• 4 位 KV 缓存
  4-bit KV cache
• 低精度专家
  Low-precision experts
• 推测解码
  Speculative decoding
• 吞吐量与等待时间
  Throughput vs. wait time
• 推理预算
  Reasoning budgets
• 如何评估模型切换效果
  How to evaluate a model switch
• 回顾
  Recap

简介Intro

我是 MiniMax 的 Morgan。之前有多少人听说过 MiniMax？哇，真不错。我喜欢这样。是的，因为我加入这家 M 公司快要三年了。但我们刚进入美国市场时，几乎没人认识我们。当时我们已经推出了第一个视频生成模型。所以我会在演讲最后给大家展示一些相关内容。不过首先，我想给大家介绍一下我们最新的语言模型，它将于下周发布。今天我想讲的主题是：更快的模型所带来的隐性成本。人们会想，如果你有一个 AI 应用，产品已经融入自己的生产工作流中，那么当一个新模型出现时，我说：“嘿，我们有新模型了，要不要试试？”它能更快地完成任务，但我觉得你真正想问的是：我该不该切换到这个模型？I'm Morgan from MiniMax. how many of you knows MiniMax before? Oh wow. I like that. Yeah. Because I joined this M company for three almost three years. But when we first like step into the American market like almost no one knows us. But at that time we have the first video generation model. So I will show you something of that at the end of this speech. But first I will show you like about our the latest language model which will be released next week. So today I would like to my topic is the hidden cost of a faster model. So when people thinking about like if you have a AI application your product built in in your own pro workflow. So when a new model arrives when I say oh please we have a new model do you want to try it? It can generate a task faster but your I think your question is should I switch to that model?

于是你尝试完成一项研究任务，报告很快就出来了，但却漏掉了一个非常重要的来源。所以，必须有人找到那个来源并修正报告。我觉得这就是我们每天都在做的事。那么，使用这个更快的模型，工作真的变快了吗？今天我们将探讨如何减少模型的计算量，然后再看看如何测试这些改动对模型的价值。So you try a research task and the report just comes back very soon and but it misses a very important source. So someone has to find that source and fix that report. I think this is the thing we did every every day. So did the job really get faster with this faster model. Today we will look at how we reduce the model computation and then we will ask how to test the value of those changes of this model.

研究工作The research job

让我给你们展示一下我们将用来回答这个问题的任务。在 MiniMax，我们构建不同的多模态模型和专业产品，供人们使用，而快速聊天回复和研究报告实际上有不同的需求。因此，任务会告诉我们什么才算表现良好。今天我将重点介绍我们最新的大语言模型 M3.1 Flash，以及一项研究任务。分析师提出的问题是：随着 AI 服务器需求不断增长，供应电路板的公司是否正将这波增长转化为现金？这里的 PCB 指的是印刷电路板。我想大家都知道这一点。它连接设备内部的电子元件，例如 AI 服务器中的电子元件。因此，我们的研究案例关注这条供应链上的公司。内部评审描述了对 14 份公司报告的分析工作，而模型承担着多项任务。Let me show you the job that we will use to answer this question. So at MiniMax we build different multimodel models and pro products that people use and a quick chat reply and research report actually have different needs. So the task tells us what good performance means. Today I will focus on our latest large language model M3.1 Flash and one research job. So here's the analyst question as AI server demand is growing are the companies supplying the circuit boards turning this growth into cash? The PCB there means printed circuit board. I think you guys know that. and unc it connects electronic components that inside devices that such as the AI servers. So our research case looks like companies in this supply chain. The internal review describes work across 14 company reports and the model has several jobs.

它必须找到正确的报告，汇总相同的报告期，并将不同的业务分开。然后，它必须解释这些数字意味着什么。想想收到报告的分析师。他们需要打开重要论断背后的来源，并核查计算过程。他们需要了解哪些证据可能会改变结论。因此，一份有用的报告会提供可供分析师核查的内容，而这就是我们对一项研究工作完成时的标准。It must find the right reports. It must compile the same reporting periods and it must keep different businesses separate. Then it must explain what the numbers mean. Think about the analyst who receives the report. They need to open the source behind an important claim and they need to check a calculation. They need to see what evidence could change the conclusion. So a useful report gives the analyst something that they can check and that is our standard for a finished research job.

利润与现金流量Profit vs. cash flow

有一项发现说明了这些检查为何重要。这次内部审查揭示了一个非常有用的区别：利润可能增长，但现金流反映的却是另一种情况。公司销量增加，因此报告中的利润上升，但客户尚未付款，或者公司购买的材料仍在仓库中。因此，分析师需要知道为什么模型应指向现金流量表及相关披露。报告只有在其主张可以核查时才能赢得信任，而这里的内部审查描述了研究成果。因此，我们仍需核实其来源和计算结果，也需要衡量这项工作的耗时和成本。但技术任务要使用独立的任务。这样有助于我们理解具体的权衡取舍。One finding shows why those checks matter. This internal review highlighted a very useful distinction. So profit can grow but cash flow tells a different story. A company sells more so is reported pro profit rises but customers have not paid yet or it has bought materials that are still in storage. So the anal the analyst needs to know why the model should point to the cash flow statement and the relevant is disclosures. So a report earns trust when it claims can be checked and the internal review here describes the research output. So we still need to verify it sources and calculations and also we also need to measure the time and cost on this same job. But the technical task a heart use separate tasks. So they help us to understand specific trade-offs.

模型内部发生了哪些变化What changed inside the model

明确了这项任务后，我们来看看 M3.1 Flash 内部有哪些变化。这张表展示了 M3.1 Flash 的主要变化。我会用几个简单的问题来说明这些变化。首先，哪些地方可以减少注意力计算？其次，数据可以存储在哪里，又该如何移动？第三，推理服务配置会如何影响用户的等待时间？第四，我们还可以控制模型进行多少推理。这些选择会影响任务的不同部分，而且每一项都需要根据我们想要的结果进行检验。因此，每项优化都必须有助于完成最终任务。With this job clear, let's look at what changes inside M3.1 Flash. This table shows the main changes inside M3.1 Flash. So I will connect them to a few simple questions. First, where can we reduce the attention work? And second, where can we store and move data? And third, how does the serving setup affect the time a user waits? And fourth, we also control how much reasoning the model does. These choices affect different parts of the job and each knows a check against the result we want. So every optimization has to help the finished job.

长输入上的稀疏注意力Sparse attention on long inputs

我们先从模型如何处理长输入说起。如今我觉得每个模型都会说：“哦，我有一个非常大的上下文窗口，正如我们所说，有 100 万。”那么，这 100 万的上下文能如何帮助你更快、更准确地找到输出和结果呢？在这份报告中，有些段落对这个问题的重要性远高于其他段落。稀疏注意力本身指的是，首先进行一个很小范围的筛选步骤，就像那里的蓝色方框。它会为文本块打分，然后主注意力计算会使用选中的文本块。模型也会关注附近的文本。实际上，筛选结果会随着查询而变化，不同的问题也可能需要上下文中的不同段落。底部的数字显示了示例中处理的文本量，而完整输入可能要长得多。Let's begin with how the model uses a long input. So nowadays I think every model says oh I have a very big context window like we said we have 1 million. So among this one million how it can help you to find the output find your result quickly faster and correct. So some passenger passages like in this report like miter just much more than others for this question. So sparse attention itself means like we uses a very small selection step first like the blue box there. So it scores the blocks of the tax and the main attention then calculation calculate the uses of the selected blocks and the model also keeps the nearby tax in view and this selection actually they can change with the query and different questions can also needs different passengers of the context and the number on at the bottom show how much text is published examples lacks and the full input can be much longer.

对于我们的研究任务，目标是减少处理大量这类文档所需的注意力工作。但即便只看这份文档，其中一条很短的脚注也可能改变整个结论。因此，对于现金流问题，这条脚注可能比数页行业展望更重要。所以我们希望模型也能纳入这段内容。减少注意力工作后，仍然必须找到重要证据。接下来的问题很简单：我们还能找到所需的事实吗？M3.1 Flash 在最初三层中也采用了类似这个条形图的注意力机制，这使得这种检索成为一项重要的检验。因此，这项独立任务会在长输入中的不同 OP 位置进行测试。在 V 轮测试中，模型处理了全部 39 个请求，最长的输入约为 830k 个 token。因此，结果涵盖了这项测试中的各种情况；而对于我们的研究任务，我们需要使用真实报告。For our research job, the goal is to reduce this attention work across many of these documents. But just within this document, a very short footnote there may change the entire conclusion. So for a cash flow question, the footnote may matter more than the pages of industry outlook. So we want the model to include that passage too. less attention work still has to find the important evidence. So the next question is simple. Can we still find the facts we need M3.1 Flash also uses like the this bar's attention in its first three layers that makes this retrieval an important check. So this separate task places at different OP positions in long inputs. So in v run the model passes all 39 requests and the longest inputs were about 830k tokens. So the result covers the cases in this test and for our research job we need with real reports.

模型能找到现金流量脚注吗？它会使用正确的报告期吗？它能发现不同文档之间的冲突吗？因此，把这段重要内容放到输入中的不同位置，然后再次提出同一个问题。只有当模型能够找到关键事实时，长输入才有用。因此，能否找到证据也是一个评估方向。Can the model find a cash flow footnote? Does it use the right reporting period? And can it spot a conflict between different documents? So move this important passage to different places in the input and then ask the same question again. A long input is useful when the model can find a key fact. So finding the evidence is one track too.

4 位 KV 缓存4-bit KV cache

接下来我们看看这个模型内部使用的数字。模型会保留先前文本中的内部信息，我们称之为 KV 缓存。M3.1 Flash 将键和值从 8 位改为 4 位，但查询仍保持 8 位。这可以减少我们存储和传输的数据量，同时也会改变这些内部信息的精度。看看这张表。平均分实际上几乎相同，但其中的符号分数下降了约 6.6 个百分点。与此同时，其他一些分数有所上升。因此，平均分实际上可能会掩盖用户在意的错误。这里比较的是两种完整配置，但并没有告诉我们哪些改动导致了各项分数的变化。因此，在研究工作中，我们应该再次检查输出：来源是否正确？计算是否正确？解释是否与证据相符？Next we look at the numbers we used inside this model. So the model keeps the internal information from earlier text which we call this KV cache. And M3.1 Flash changes its keys and values from eight bits to four bits, but the query still stays at eight bits. And this can reduce the data we store and move. It also changes the precision of that internal information. Look at that table. So the average scores actually is almost the same but the sign score there it drops like 6.6 percentage. So some but some other scores rises. So the average score actually can hide a mistake we your user cares about. This compares that two full setups but it does not tell us which changes caused each score. So for our research job we should check the output out output again. Are the sources right? Are the calculations right? And does the explanation match the evidence?

低精度专家Low-precision experts

同样的思路也适用于模型的另一个部分：专家。我们的模型采用 MoE 架构，因此包含我们称为“专家”的计算模块，也就是 MoE 中的 E（Experts）。在 M3.1 Flash 中，模型会为每个 token 选取其中的一部分。这些被选中的专家使用四位权重和激活值。共享专家不受这项改动影响。权重是模型中存储的数值，激活值则是模型计算出的数值；较低的精度可以减少数据移动，并支持更高效的计算。不过，完整的计算包含多个步骤：我们先缩放数值，执行计算，再恢复缩放。步骤的顺序至关重要。表格展示了缩放恢复发生的位置，而确保顺序正确，正是确保结果正确的关键。因此，快速计算仍然需要校正。The same idea applies to another part of the model: its experts. our model is the MoE architecture. So it has the computing parts that we called experts which is the M of E experts. So it selects some of them for each token in M3.1 Flash. These selected experts use four-bit weights and act activations. The shared expert is outside this change and weights are stored model values and activations are values the model calculates and lower precision can reduce data movement and support more efficient computation. But the full calculation has several steps. So we scale the numbers, we run the calculation and restore the scale. The position of these steps really matters and the table shows like where scale restore happens and getting that order right is the part of getting the answer right. So fast math still has to be corrected.

检查完整结果时，我们测量的是完整的 imp 实现。对于这项研究工作，我们会再次提出相同的问题，检查答案为何仍未通过我们的检验。现在，我们来看看洞察。And when we check the complete result, we measure the complete imp implementation. And for this research job, we run the same questions again. We inspect why the answers still miss our checks. And now we look at insight.

推测解码Speculative decoding

我们已经深入了解了模型。现在，我们可以看看它是如何为你生成答案的。一种称为推测解码的服务方法分为两个阶段。首先是更快的草稿生成过程。它可以帮助你提出一小段 token 序列，然后目标模型可以对其进行跟踪，系统会保留目标模型接受的第一部分，并在需要时进行修正，然后继续这一整个过程。有时它会保留多个提议的 token，有时草稿确实会生成一些它无法保留的内容。因此，这里的机会在于通过一次目标模型检查获取多个 token；但起草需要时间，检查也需要时间。拒绝这些 token 同样需要时间。因此，也要把检查草稿所花的时间计算在内。这些检查会提出 token，但财务声明和来源也需要各自经过核查。We have already looked inside the model. And now we can look at how it generates an answer for you. So a serving method called speculative decoding works in two stages. A faster draft process. It can help you to propose a short sequence of tokens and the then the targeting model can tracks it and the system keeps this first part that the target accepts and makes a correction when it's needed and then continues this whole process. And sometimes it keeps several proposed tokens and sometimes the draft does work what it cannot keep. So the opportunity here is to get several tokens from one target check and but drafting takes time checking takes time. Rejecting this tokens also takes time. So count the time spent checking the draft too. These checks propose tokens and but financial claims and sources need their own checks.

因此，对于我们的研究任务，也应将来源检索、工具调用和重试纳入考量。最终衡量标准是生成一份分析师在实际工作中真正能用得上的报告所需的时间。So for our research job we should also include the source retrieval the tool calls and retries. The final measure is the time to a report that an analyst can really use in their home own work.

吞吐量与等待时间Throughput vs. wait time

即使单个请求运行良好，很多用户也会改变这种体验。下面是另一个服务测试。它使用存储生成，推理强度较低，但并发请求数现在从 32 增加到了 64。因此，每秒总输出量增长了约 1.9 倍。不过，看看等待时间：首个 token 的 P95 等待时间实际上上升了，从约 1.5 秒增至 11.3 秒，呃，6 秒。也就是说，95% 的请求会在这个时间内收到首个 token，现在等待时间变得长了约 8 倍。因此，中位数仍然是我们能够满足要求的时间指标。但我们不能只看中位数。只看这一个指标会让一切都变样。用户能感受到等待，即使你看到吞吐量超过每秒 300，也仍然需要关注最终的等待时间。Even when one request runs well, many users can change this experience. Here's an separate serving test. So it's uses the storage generation with low reasoning effort but the concurrence requests now like we rise from the 32s to 64. So the total output per second grows about 1.9 times but look at the wait time actually it the P95 wait for the first token rises. It goes from about 1.5 seconds to 11.3 seconds uh6 seconds. So for 95% of this request the first token arrives by this time and the wait now is becomes like eight times longer. So the medium stay as the time that we can satisfy it. But we cannot only look at the median time. So it changed everything that if we only like look at that one. So the users feel the wait and even when the throughput you see like oh we have through like more than 300 per second but still you need to care about the final wait time.

因此，第一个 token 可能是推理内容，我们仍需要测量可用报告何时到达。所以要为该报告设定一个截止时间。So the first token may be reasoning and we will still need to measure when the usable report arrives. So set a deadline for that report.

推理预算Reasoning budgets

还有一个设置会改变我们要求模型完成的工作量。这是我们第一次使用这种不同的推理控制方式。从这里开始，介绍我们在请求中使用的默认最大值设置，它可以控制模型思考时使用多少个 token。文档中的设置显示，思考功能保持开启；但如果我们将推理力度留空，默认值就是 max。默认设置会改变我们要求模型完成的工作量，而一项编码测试展示了其中的权衡。测试比较了 high 和 max 两种推理力度。high 使用的输出 token 比 max 少，但准确率实际上从 80% 降到了 68%。也就是说，在这些任务上，token 用得更少，准确率也更低。因此，应根据任务的实际需要为其分配思考预算。对于我们的研究工作，可以用同一批报告比较这些设置，检查每种设置是否都能找到证据并支撑计算；而对报告进行快速初审、准备交付审核时，可能需要不同的预算。There is one more setting that changes the work we ask the model to do. So this is the first time we just use this different reasoning control. So from this part so this is the default max changes that we use in the request and it can controls how many tokens we use for this model thinking. So in the document setup here so thinking stays on but if we leave the effort blank the default is max. So a default can change the amount of work we request and one coding test there shows the tradeoff. So high reasoning and max reasoning. So the high uses fewer up output tokens than the max but the accuracy actually fell from 80% to 68%. So the fewer tokens came with lower accuracy on those task items. So give this task the thinking budget it really needs. So for our research job compare these settings on the same reports and check whether each setting finds the evidence and supports the calculations and the quick first pass on the report ready for review may need different budgets.

所以，我们需要记录这一设置，或者记录需要采用哪种推理控制，以便在后续工作中使用。So we need to record this setting or which reasoning control we need to put and then used in our later work.

如何评估模型切换How to evaluate a model switch

好的。那么，在我们使用这个模型的所有新更新来发布这套研究工作流时，首先可以解决源报告中的这个问题，采用相同的信息截止时间，并在进行比较之前就报告达到什么标准才算合格达成一致。然后我们可以检查三件事。 第一，分析师能否信任这份报告，能否打开重要论断背后的资料，核对报告期、重新计算财务数据，并查找可能改变结论的证据。 第二，完成这项任务花了多少钱？用了多长时间？还要计入检索工具的费用和重试成本。因此，如果有人必须先修正报告才能使用，也要把这部分工作计入成本。所有失败的尝试都应计入生成这些合格报告的成本。Okay. Then when we ship this research workflow with all these new updates we have with this model. First we can fix this question on the source reports and use the same information cutoff and agree on what makes a report acceptable before running this comparison and then we can check three things. The first is can the analysts trust the report and open the resources behind the important claims and check the reporting periods recalculate the financial numbers and look for the evidence that could change the conclusion. And second, what did the finished job cost? How long did they take? And include retrieval tool costs and retries. So if a person must fix the report before use, count that work too. So include all the failed attempts in the cost of producing of producing this acceptable reports.

因此，要按照你预期的用户数量进行测试，检查报告是否能在承诺的时间内送达，并衡量实际可用报告的成本。第三个问题是：我们能否复现这一结果？要保存模型和运行时版本，也要保存提示设置和来源。保留这些失败案例，这样我们就可以从小规模推出开始，并保留回退的方式。So test are the number of these users you expect and check whether reports arrive within the promised time and measure the cost of report you can actually use. And the third one is can we repeat this result save this model and runtime versions and also save the prompt settings and sources. So keep these examples of failures and we can start with a small roll out and keep a way to switch back.

总结Recap

在结束之前，先简短回顾一下今天的分享。这段内容由我们的 AI 视频模型 Miniax A3 制作，你可以在这里查看。好，这是我们正在开发的另一个模型，而且它现在已经开放使用了。所以，所有想要构建这类模型的人，实际上只需要解决一个问题：文本转视频。好，那我们回到今天开场时提出的问题：我们是否应该切换到更快的模型，看看我们过去 18 分钟一直在讨论的这项任务能否顺利完成？对我们的分析师来说，这意味着一份附有可打开的资料、包含可核查数据的报告，必须在他们需要时送达。而 M3.1 Flash 改变了我们分配计算资源的方式，完整任务的演示则展示了这些变化对用户意味着什么。因此，我们要衡量的是完成任务的成本，以及能否按时完成。我认为，更快的模型确实很重要，但最重要的是我们能用它们做什么。So before we finish, here's a short recap of today's talk. It's created with Miniax A3, our AI video model, and you can check here. Okay. So this is another model that we are working on and it's opened way. So everyone that who wants to build this actually it's only a one problem with the text to video. Okay. So let's return to this our opening question today. Should we switch to a faster model and look at this finish job that we talking about for the last 18 minutes. So for our analyst that means a report with resources they can open and it has numbers they can check it must arrive when they need it and for this M3.1 Flash changes how we spend computation and the complete task shows that those changes means for the user so measure what we cost to finish job and on time. So faster models I think like faster mo models really matter but what matter most is what we can do with them.

谢谢大家。Thank you guys.