Whole-node vs fractional GPU rental — how to choose

The GPU rental market sells the same silicon in very different units. You can rent a slice of a card, a whole card on a shared host, or an entire bare-metal node with every GPU and its interconnect. The per-hour numbers make fractions look obviously cheaper. Whether they are cheaper depends entirely on what you are running.

The ways a GPU gets shared

"Fractional" covers several mechanisms with very different isolation guarantees:

• Hardware partitioning (MIG). Some data-center GPUs can be split into isolated instances with dedicated slices of compute and memory. Isolation is strong, but each instance is small, and partitions cannot be joined by the high-speed intra-GPU fabric.
• Time-slicing / vGPU. The scheduler hands the whole GPU to different tenants in turns. You get the full card briefly, but latency becomes bursty and unpredictable under contention.
• Container on a shared host. You get a GPU, but CPU, system memory, PCIe lanes, local disk and network are shared with strangers. The GPU is yours; the box around it is not.

Each step trades away isolation for a lower headline price.

Noisy neighbors and tail latency

The failure mode of shared infrastructure is rarely the average — it is the tail. A co-tenant saturating PCIe during a checkpoint load, or hammering host memory bandwidth, shows up as sporadic latency spikes in your service that you cannot reproduce or explain. For batch jobs that is an annoyance; for a latency-SLO inference endpoint it is a production incident with someone else's name on the root cause.

Bare metal removes the variable. When the node is yours, the only workload contending for its bandwidth, its lanes and its caches is your own — so a benchmark you run on Monday still holds on Friday.

When you actually need the whole node

Some workloads simply do not fit the fractional model:

• Models larger than one GPU. Tensor and pipeline parallelism split a model across GPUs and rely on the fast intra-node link (NVLink) to exchange activations every layer. A fraction gives you part of one GPU — there is no fast path to the others.
• Multi-GPU training. Gradient all-reduce across 8 GPUs wants the full NVLink domain; routing it over a shared PCIe bus throttles the whole job to the slowest link.
• High-throughput serving. As covered in the economics of inference, throughput comes from batching, and batching wants VRAM headroom and bandwidth that a slice does not have.

If your job touches more than one GPU at a time, the interconnect is the product, and you can only get the interconnect by holding the node.

Bare metal vs virtualized

Even at "whole card" granularity, there is a difference between a virtualized host and bare metal. A hypervisor adds a layer between you and the hardware: overhead on I/O, constraints on driver and kernel versions, and no true root. Bare metal gives you the physical machine — install your own driver stack, pin kernel versions, profile at the hardware counters, and run privileged tooling that a shared tenancy would never allow.

TheAI Cloud rents whole NVIDIA B300 nodes — all eight GPUs, the full NVLink 5 domain, a non-blocking 400G RDMA fabric (RoCE v2), root on the box, no virtualization. Nothing sits between your code and the silicon.

The cost intuition

Fractions optimize for occupancy of a shared asset; whole nodes optimize for predictable performance of a dedicated one. The honest trade-off:

• A fraction can be cheaper for a small, steady, single-GPU workload that never needs a neighbor's bandwidth.
• A whole node is usually cheaper per token or per training step for anything multi-GPU or throughput-oriented, because you get the interconnect and the full memory system — and because predictable performance means you are not over-provisioning to cover the tail.

The one catch with a dedicated node: utilization is now your responsibility. An idle whole node bills for silicon you are not using. Which unit wins comes down to whether you can keep the node busy.

Matching the unit to the job

A quick rule of thumb:

• Single small model, low traffic, no latency SLO → a fraction may be the economical choice.
• Model spanning multiple GPUs, any training, or a throughput/latency target → rent the whole node and keep it fed.

When you do want a whole node, live rates are in the pricing section and setup is in the docs.

FAQ

Is a fractional GPU always cheaper than a whole node?

Only per hour, and only for workloads that fit in one fraction. For multi-GPU or high-throughput jobs, a whole node is frequently cheaper per token or per training step because you get the full interconnect and memory system and predictable performance. Compare on cost per unit of work, not cost per hour.

What is a noisy neighbor?

Another tenant on the same physical host whose activity — saturating PCIe, host memory bandwidth or network — degrades your performance unpredictably. It mainly shows up as latency spikes you cannot reproduce. Dedicated bare metal eliminates it because you are the only tenant on the machine.

Can I run multi-GPU training on a fractional GPU?

Not effectively. Multi-GPU training depends on fast inter-GPU communication over NVLink for gradient exchange. A fraction is a slice of a single GPU with no fast path to others, so collective operations fall back to slow shared links and throttle the whole job. Multi-GPU work wants a whole node.

Why choose bare metal over a virtualized GPU host?

Bare metal removes the hypervisor layer: you get root, direct hardware access, your own driver and kernel versions, and access to hardware performance counters for profiling. There is no virtualization overhead on I/O and no restriction on privileged tooling. For performance-sensitive or systems-level work, that direct access matters.