How fast is The Box?

The speed panel on the Box page shows four numbers per model. This page shows everything behind them: how speed changes as a chat gets longer, how long you wait for the first token, how we measured, and where we have no number yet.

Every figure here was measured on a Box running the models exactly as they ship: the same model files, the same engine build, the same settings. That means a 64 GB Box for every model except Qwen 3.8 Flash-Next, which only runs on 128 GB. Where a number is our estimate rather than a measurement, it says so and shows the arithmetic.

The short version

"~" marks an estimate. The two fast models are mixture-of-experts models: only 3–4 billion of their parameters work on each token, so they write quickly. The two 27B models are dense: all 27 billion parameters work on every token. They are slower and more thorough, and they are the ones to pick when the answer matters more than the wait.

Qwen 3.8 Flash-Next sits between them. It is the largest model on the Box, 125 billion parameters, but it is also a mixture of experts: 6 billion work on each token. It needs 128 GB of memory, so it is not available on the 64 GB Box.

Output speed as a chat grows

Output speed is how fast the model writes its reply. It drops as the conversation gets longer, because the model has more context to look back over for every new token.

Tokens per second. The default model, Qwen 3.6 35B-A3B, still writes at about 47 tokens a second 128K tokens into a conversation [1]. Gemma is close to it in a new chat but falls off faster, so it is the better pick for short and mid-length work, not for very long conversations [2]. Flash-Next has only been measured at the start of a chat so far [8].

Input speed as a prompt grows

Input speed is how fast the model reads what you give it: your message, a pasted document, the results of a tool it ran. Longer prompts read more slowly per token.

Tokens per second. To read a full 128K-token prompt from scratch, that works out to about 6 minutes on the default model, 7 on Gemma and 16 on Qwen 3.6 27B [1][2][3]. Flash-Next reads fast for its size: 1,012 tokens a second on a 16K-token prompt, on the new engine it runs on [8].

Time to first token

This is the wait between sending a message and seeing the first word of the reply. On the Box it depends mostly on one thing: how much text the model has to read first.

The Box runs the Hermes agent. Before you type anything, every chat already holds about 18,000 tokens of instructions, tool descriptions and memory. On the first message of every new chat the model has to read all of it. After that it is cached for the rest of the chat, and each later message only has to read what is new.

Measured: the Qwen 3.6 35B-A3B and Qwen 3.8 27B rows come from a real 18K-token chat with five follow-up turns [4][5].

Estimated: the wait is the prompt length divided by input speed. That formula reproduces both measurements to within a tenth of a second (17,900 ÷ 786 = 22.8 s against 22.7 s measured; 17,900 ÷ 272 = 65.8 s against 65.7 s). For Gemma and Qwen 3.6 27B we have input speeds at 8K and 32K but not at 18K, so we interpolated between them, then took off 7%, the amount by which the same interpolation overstates the default model's measured figure. Flash-Next was measured on a 16,132-token prompt: 15.9 seconds to the first token. Scaled to an 18K chat, that is about 18 seconds [8].

If you paste something in, add its length divided by the input speed. A 4,000-word report is about 5,000 tokens, which the default model reads in about 6 seconds before it starts answering.

The models think before they answer. Hermes runs the models with thinking on, so the first token is the start of the model's reasoning, not its answer. Reasoning is written at output speed, and how long it runs depends on the question: every 500 tokens of it adds about 7 seconds on the default model, 13 on Flash-Next and about 30 on a 27B. We have not yet measured how long a typical answer's reasoning is.

How long a whole reply takes

The total is time to first token plus the reply length divided by output speed. For a 300-word reply (about 400 tokens) in an ongoing chat:

These are arithmetic from the tables above, not separate measurements, and they leave out thinking time.

More than one chat at once

The Box runs several chats side by side, each with its own full context: two on the 64 GB Box and four on the 128 GB Box, for every model except Qwen 3.6 27B, which runs one 128K chat on 64 GB and four on 128 GB, and Flash-Next, which runs one 256K chat at a time on 128 GB [6][9]. A second Flash-Next chat waits for the first to finish.

We have proven that these fit in memory. We have not measured how fast each chat runs when several are busy at the same time, so we don't quote a per-chat speed under load. Every figure on this page is one chat at a time.

64 GB or 128 GB

Both use the same chip with the same memory bandwidth, and speed on these models is limited by memory bandwidth. So a single chat runs at the same speed on either. One reading of the default model on a 128 GB machine with the Box's chip matched the 64 GB Box in the same real-chat test [7]. It was a single run, so we print no separate 128 GB numbers for the models both sizes run.

What 128 GB buys is more chats at once, and bigger models. The first is Qwen 3.8 Flash-Next: a 97 GB download from the Model Store that will not fit on 64 GB [9].

How we measured

• Hardware: a 64 GB Box (AMD Ryzen AI Max+ 395, Radeon 8060S graphics, memory shared by both). Flash-Next on a 128 GB machine with the same chip.
• Engine: llama.cpp on Vulkan, build 4fc4ec5, the build the Box ships, served through llama-server the way the Box serves it. Flash-Next runs on a different engine: llama.cpp on AMD's ROCm (HIP), which the 0.2.8 update installs for it alone.
• Model files: the exact quantizations that ship: UD-Q5_K_XL for Qwen 3.6 35B-A3B, UD-Q6_K_XL for both 27B models, UD-IQ4_XS for Flash-Next. Gemma is the exception (see below).
• Speed-up drafter: each model runs with its multi-token-prediction drafter switched on, as it does on the Box. The drafter guesses a few tokens ahead and the model checks every guess, so it changes how fast the model writes, not what it writes.
• Prompts: ordinary prose. We never use counting or repetition prompts, which make drafters look a third faster than real use.
• Runs: the default model's figures are the average of 2 runs. Everything else is 1 run. We will replace these with 3-run averages as we re-measure.

Where the numbers are not exact

• Gemma 4 26B-A4B was measured on Q8_0, the file the shipping UD-Q8_K_XL replaced on 22 July. A later test without the drafter had the shipping file writing about 4–6% slower [2].
• Qwen 3.8 27B was measured on newer engine builds than the Box ships (07822bd and f5e85d4). Its first-reply time was measured with a different drafter loaded; input speed barely depends on the drafter [5].
• Qwen 3.8 Flash-Next was measured once, on 5 October 2026, with a 16K-token test prompt sent straight to the engine rather than a real agent chat. Its first-reply and later-reply times are estimates from that run [8].
• Not measured yet: Flash-Next output past the start of a chat, Qwen 3.8 27B output past 32K, Gemma and Qwen 3.6 27B time to first token, any per-chat speed with several chats running, and code prompts on the default model.

---

Sources

Arx benchmark runs, from our engineering benchmark log. Sources 1–5 are on the 64 GB Box.

1. Qwen 3.6 35B-A3B, UD-Q5_K_XL, multi-token prediction 2 ahead: depth sweep, 15 July 2026, llama.cpp build 4fc4ec5, 2 runs per figure.
2. Gemma 4 26B-A4B, Q8_0, multi-token prediction 2 ahead: depth sweep, 15 July 2026, build 4fc4ec5, 1 run per figure. Shipping UD-Q8_K_XL without drafter: 28 August 2026, build f5e85d4.
3. Qwen 3.6 27B, UD-Q6_K_XL, multi-token prediction 3 ahead: depth sweep, 15 July 2026, build 4fc4ec5, 1 run per figure.
4. Qwen 3.6 35B-A3B in a real 17.9K-token agent chat with five follow-ups: 31 August 2026, build f5e85d4, 1 run.
5. Qwen 3.8 27B, UD-Q6_K_XL: output with multi-token prediction 3 ahead, 20 August 2026, build 07822bd; real 17.9K-token chat, 31 August 2026, build f5e85d4; 1 run each.
6. Box serving configuration as shipped, factory model list, and the Model Store catalog, checked 2 October 2026.
7. Qwen 3.6 35B-A3B on a 128 GB machine with the Box's chip, real 17.9K-token chat, 15 September 2026, build 4fc4ec5, 1 run.
8. Qwen 3.8 Flash-Next, UD-IQ4_XS, multi-token prediction 3 ahead, KV cache f16, on a 128 GB machine with the Box's chip: the 0.2.8 engine (llama.cpp fork 40a9f4d on ROCm with the PM4 runtime 7dda3ac), 5 October 2026, 1 run. 16,132-token prompt: 15.9 s to first token, 1,012 tokens/s input; output 38.0–40.5 tokens/s at the start of a chat.
9. Model Store catalog for the 0.2.8 update (version 2026-10-02c), checked 5 October 2026: Flash-Next requires 128 GB, runs one 262,144-token chat, and downloads 97.4 GB.