When several models share one machine and each one needs memory of its own, the number that decides the build is not peak throughput but the price of a gigabyte of video memory. On that measure the RTX PRO 6000 Blackwell Server Edition beats the server cards in its class.

What production inference is, and how it differs from training

Training is a single run: the machine spends a month on one task, and the gradients set the memory budget. Production inference looks different. A chat model, a document search model, a recognition model and a couple of smaller services all run at the same time. Each keeps its own weights resident around the clock.

So the question is not which card is fastest, but how many models fit at once and what each gigabyte costs. Four RTX PRO 6000 Server Edition cards give 384 GB, and that is the cheapest video memory among NVIDIA server cards.

Configuration

Platform 4U rack chassis, up to 8 GPUs
Processors 2× AMD EPYC 9355
Memory 768 GB DDR5-6400 ECC Registered, 12× 64 GB
Accelerators 4× NVIDIA RTX PRO 6000 Blackwell Server Edition 96 GB, 384 GB in total
Storage 2× 960 GB M.2 for the system in RAID1, 2× 7.68 TB U.2 NVMe for data
Network 2× 25GbE, 200G optional
Power 4× 3200 W with 2+2 redundancy

The main advantage: MIG instead of queueing for a card

The classic trouble with a shared GPU server: two teams, one card, and somebody is always waiting. Blackwell can do the partitioning in hardware. MIG cuts the physical accelerator into isolated slices, each with its own memory and compute units, and the system sees every slice as a separate card.

In practice four physical cards turn into a dozen logical ones. The development team gets its slice for experiments, the production service gets its own, and neither disturbs the other even when someone launches a heavy script by mistake.

The isolation here is hardware, not software: a failure in one slice does not bring down its neighbour. That is the difference between agreeing to stay out of the way and being physically unable to interfere.

How many models fit

Scenario What 384 GB holds
Several mid-sized models four 32B-class models at half precision, plus headroom for the context cache
One large model a 70B model in FP16 across two cards, with the other two left for other services
Many small ones a dozen 7-13B services, each in its own MIG slice
Expansion to 8 cards 768 GB, so twice as many services without changing the platform

Why the Server Edition and not the ordinary RTX PRO 6000

The same architecture ships in two forms. The workstation card sits in a tower under a desk and carries a fan of its own. The server card is built for front-to-back airflow through the rack: it has no fan at all, and the chassis fans move the air.

Fitting a workstation card into a dense 4U means cooking it: the rack leaves no room for its fan, and it never catches the chassis airflow. A server build therefore takes the Server Edition only, and the reverse holds as well.

Compared with an H200 machine

Criterion RTX PRO, 4 cards H200 NVL, 2 cards
Video memory 384 GB 282 GB shared
Memory as one pool no, four separate pools of 96 GB yes, over an NVLink bridge
Partitioning between teams MIG, up to a dozen slices available, but only two cards
Strong point many services at once one very large model

Room to grow

  • Eight cards instead of four: 768 GB of video memory in the same chassis.
  • System memory up to 1.5 TB, once there are more services and each holds its own cache.
  • A 200G port, if the machine becomes part of a cluster rather than a standalone node.

Who it suits and who it does not

A good fit

  • Production where several models serve users around the clock.
  • Companies where several teams compete for the same cards.
  • Workloads where memory per unit of budget counts for more than a record in a synthetic benchmark.

Better to look elsewhere

  • Training large models from scratch: that calls for a shared memory pool over NVLink, which means an H200 machine.
  • One person and one model: a tower under the desk costs less and fits in an office.

Common questions

Can these cards be bridged into a single memory pool?

No, and that is the main difference from the H200 NVL. There are four independent 96 GB cards here: a model larger than 96 GB has to be split in software, at the usual cost in speed.

How many MIG slices does one card give?

It depends on the profile: memory and compute units are divided in proportion. In practice a card is split into two to four slices, and finer partitioning rarely pays off.

Why 768 GB of system memory rather than a terabyte, as in the H200 machine?

Inference keeps no optimiser state or gradients in system memory. The weights load once, video memory does the work from then on, so less headroom is needed.

What happens if the cards run out under load?

The platform holds eight, so the next step is buying more cards rather than replacing the server. Power and cooling in this configuration are specified with that in mind.

What we have of this

Author: ETE Engineer