Kimi K3 (2.8T) and six other open models on a single 8×B300 node — full vLLM numbers

Before opening our own B300 cloud, we wanted first-hand answers to the question every serving team asks: what do current open models actually do on this hardware? Not vendor slides, not extrapolations — measured numbers, on the exact node class we'll be renting out.

So we rented an 8× B300 SXM6 node (spot instance from a public GPU cloud), took out-of-box vLLM — deliberately no tuning — and ran seven of the most-used open models through the same matrix. This is the summary; per-model deep dives follow in the coming weeks.

Setup, briefly

Hardware: single node, 8× NVIDIA B300 SXM6 — 288 GB HBM3e per GPU (268.6 GiB visible to the driver; ~2.1 TiB across the node), NVLink NV18, driver 580.126.09, CUDA 13.0. Software: vLLM 0.26.0 (Kimi K3 — nightly 0.26.1rc1), default flags unless a model refused to start without one. Scored matrix: concurrency {1, 4, 16, 64} × prompt {512, 4k, 32k} tokens, 512 output tokens. On top of the matrix, a separate ramp pass (prompt 4k, concurrency doubling per step up to a hard cap of C=512) finds the node's plateau, followed by a 240s sustained hold. Zero errors across all completed scored runs (the Mistral crash is a separate story — Finding 3). Definitions: plateau = max aggregate output tok/s across ramp steps (closed-loop, 3×C requests per step); sustained = total output tokens of requests completing inside the steady window ÷ window length (225s of the 240s hold). Single-user numbers: C=1, prompt 512 — TTFT is p50 of 8 sequential requests; tok/s is median per-request decode rate, output tokens only (prefill excluded). Honesty flags: rented node, single run per cell, one client process (matters — see below). Our own rack lands in Q4; we'll republish on it with the same harness, side by side.

The numbers

† Gemma's TTFT is reported at C64 (its C=1 TTFT is 0.028s); it runs as 8 independent replicas (DP8) since 31B fits on a single GPU — that's how you'd actually serve it on this node. * Sustained window (225s) is short relative to ~150s end-to-end requests at C512 — treat as a lower bound. ** Reproducible engine crash under sustained saturation — details below.

Finding 1: on B300, the giants are interactive

To be clear about what's news here: trillion-parameter MoEs being interactive on 2 TiB of HBM is the Blackwell spec sheet doing its job — verification, not surprise. The interesting part is what the spec sheet won't tell you, below. Still, the pattern deserves stating: the responsiveness gap between small and huge models has nearly collapsed. Time-to-first-token runs 23–89ms across the entire lineup — a 2.78-trillion-parameter Kimi K3 answers a single user at 77 tok/s with a sub-100ms first token. Two years ago, "frontier-scale open model" implied seconds of TTFT and single-digit decode rates without a cluster. On one B300 node, trillion-parameter MoEs behave like chat-grade models.

The mechanism is no mystery — ~2.1 TiB of aggregate HBM keeps weights and KV cache resident, and MoE architectures activate a fraction of parameters per token — but seeing 0.040s TTFT on a 1.6T model is still a recalibration. For single-user interactive inference with weights resident in HBM, model size has stopped being a latency argument. It remains a throughput argument: plateau numbers scale roughly with active-parameter count, from 5.8k tok/s (Gemma) down to 1.1k (Kimi).

Where the bandwidth goes (and why batching exists)

A quick roofline sanity-check makes the latency result less mysterious. At C=1, decode reads each model's active parameters once per token. Multiply by measured tok/s and you get the bandwidth a single user actually harvests — against the node's ~64 TB/s aggregate:

Qwen (17B active, FP8): ~2.7 TB/s — 4% of the node. GLM-5.2 (~40B): ~4.4 TB/s — 7%. V4-Flash (~14B, FP4): ~1.3 TB/s — 2%. V4-Pro (49B): ~2.8 TB/s — 4%. Mistral (39B): ~3.5 TB/s — 5%. Kimi (104B): ~4.4 TB/s — 7%. Single-stream decode on a TP8 giant leaves >90% of the node's memory bandwidth idle — the bottleneck at C=1 is kernel-launch and sync latency, not HBM. That's precisely why the plateau lives at C=256–512: batching is how this node earns its 64 TB/s. And it's why Gemma runs DP8 — one 31B replica per GPU harvests ~4.4 TB/s of its own 8 TB/s (~55%), which is what a properly bandwidth-fed dense model looks like. (Active-parameter counts: vendor cards where declared, config.json reconstruction where not — GLM and V4-Flash are estimates. FP4 weights counted at ~0.55 bytes/param — NVFP4 block scales add ~10% over the raw 0.5. MTP modules excluded, speculative decoding off.)

Finding 2: defaults are a sane baseline — and the knobs we turned made things worse

We tested the classic capacity knobs (--gpu-memory-utilization 0.9→0.95, --max-num-seqs →1024, --max-num-batched-tokens →16384) on two models. Peak throughput: Gemma −4.2%, DeepSeek V4-Flash −2.0%. The interactive cell was the real casualty: −28.6% throughput with TTFT going 0.51s → 16.4s — large prefill chunks let prefill starve decode, a known throughput↔latency tradeoff we cranked in the wrong direction ourselves.

Two honest caveats on scope. First, these are capacity knobs, not the interesting tuning surface: expert parallelism for MoE, speculative decoding (DeepSeek V4 ships MTP), attention backend choice, CUDA graphs — none of that is explored here; that's harness-v2 territory, and we'd expect real gains to live there. Second, one flag was mandatory: DeepSeek V4 needs --kv-cache-dtype fp8 (its native fp8_ds_mla format) — vLLM's auto-detection doesn't pick it, and without it the model runs silently slower. So the accurate summary isn't "tuning is dead" — it's: vLLM 0.26 defaults are a production-worthy starting point on this hardware, the folklore capacity knobs from the H100 era can actively hurt interactive traffic, and the model-specific flags you actually need are documented per model, not discovered by knob-turning.

Finding 3: what didn't launch is data too

Benchmark posts usually hide the failures. Ours were instructive enough to publish:

• Qwen3-Coder-Next is incompatible out-of-box — FP8 checkpoint fails block-quant validation (64∤128), BF16 hits a Blackwell MoE-kernel shape assert, nightly included. Three checkpoints, two images, no dice. If your roadmap includes it, budget integration time.
• Kimi K3 didn't fit on our boot disk — 1,454GB of weights vs the 968GB drive on our (deliberately cheap) spot config. Classic workaround, worked fine: tmpfs out of the node's 2TB RAM, stream weights straight there — loads faster than NVMe anyway. Lesson for anyone benchmarking frontier-size models on rented spot capacity: check the disk line before you check the GPU line. 1.5TB of weights is a logistics problem before it's a serving problem.
• Mistral Large 3 (NVFP4) crashes reproducibly under sustained saturation — clean hardware (Xid/ECC zero across the series), reproduced twice, looks like an inference-engine path issue rather than silicon. We're isolating a minimal repro before filing upstream; until resolved, we'd treat NVFP4 Mistral serving at high sustained load as not production-ready.

Finding 4: measure your client, not just your server

Our sustained numbers cluster suspiciously around ~4.7k tok/s for the three fastest models. That's not the node's ceiling — it's our load client's: a single event-loop process saturates around there. The fastest models need a multi-process client to show their real plateau; harness v2 will have one, and we'll re-score the leaders. If you benchmark inference yourself: profile the measuring instrument before trusting the measurement. A surprising share of published "model X does N tok/s" numbers are client-bound.

What this means if you're choosing hardware

Three practical takeaways from two days of runs:

1. One node now covers model classes that used to demand clusters. Kimi K3, DeepSeek V4-Pro, GLM-5.2 — resident and interactive on a single box. For a lot of serving roadmaps, "when do we need multi-node" just moved out by a generation.
2. FP4/FP8 checkpoints are the difference between fitting and not. The table's weight column is the real capacity planner: 288GB per GPU is generous, but a 2.78T model still eats 1.4TB.
3. Out-of-box is a valid production baseline on this generation — and the real tuning upside (expert parallelism, speculative decoding, attention backends) is still on the table. Budget engineering time for model-specific flags and quant compatibility, not folklore capacity knobs.

Reproducibility

Model revisions (HF commit hashes, reconstructed from commit history against download dates), launch commands and raw per-run JSON are available on request — capacity@theai.com. Revisions weren't pinned at download time in harness v1 — v2 pins them; reconstruction was verified against repo histories (no commits landed between the referenced revision and our download for any of the seven models).

What's next

Per-model deep dives (Kimi K3 first — 2.78T on one node deserves its own numbers), a multi-process harness v2, and the same suite re-run on our own rack when it lands in Q4 — same methodology, published side by side, so you can see whether "our" numbers hold on the hardware we actually rent out.

The node class in this post — whole 8× B300 SXM6, 288GB per GPU — is exactly what TheAI Cloud rents: per-minute billing, 1–12 month terms, from $5.30/GPU·hr reserved. Reservations are open at cloud.theai.com.

Questions about the methodology or a model you'd like benchmarked next — capacity@theai.com.