GPU server for LLMs: what fits in 96 GB of VRAM
One RTX PRO 6000 Blackwell with 96 GB of memory runs a 70B model in 8-bit or 4-bit, any 30B-class model at full precision, and gpt-oss-120b. Here is the math, the setup, and when a monthly GPU server is the wrong choice.
The card
All our GPU servers (DS-G1, DS-G2, DS-G3) carry one NVIDIA RTX PRO 6000 Blackwell Max-Q:
- 96 GB GDDR7 with ECC, 512-bit bus, about 1.8 TB/s of memory bandwidth;
- 5th-generation Tensor Cores with FP8 and FP4 support;
- 300 W, the Max-Q version built for multi-GPU servers;
- MIG: the card can be split into up to four isolated GPU instances.
For LLM inference, memory size and bandwidth decide almost everything. Compute rarely is the limit on a single user.
Will my model fit?
Weights take roughly parameters × bytes per parameter: 2 bytes at FP16/BF16, 1 byte at FP8, about 0.55 bytes in 4-bit formats (AWQ, GPTQ, GGUF Q4). On top of that you need room for the KV cache, which grows with context length and the number of parallel requests. Keep 10–25 GB free for it.
| Model size | FP16 / BF16 | FP8 | 4-bit | On 96 GB |
|---|---|---|---|---|
| 7–8B | ~16 GB | ~8 GB | ~5 GB | Easily, with many parallel users |
| 14B | ~28 GB | ~14 GB | ~8 GB | Easily |
| 32B | ~64 GB | ~32 GB | ~18 GB | Yes, even at BF16 |
| 70–72B | ~140 GB | ~70 GB | ~40 GB | FP8 or 4-bit only |
| gpt-oss-120b (MoE) | — | — | ~65 GB (MXFP4) | Yes |
| 200B+ | — | — | 110 GB+ | No, needs several GPUs |
A 70B model in FP8 fits, but leaves around 20 GB for the KV cache: fine for a few users with 8–16k context. If you need long context or a team hitting it at once, take the 4-bit version. The quality drop is small for chat and RAG, and you get 50 GB of headroom.
Setup in about 15 minutes
The server comes with a clean OS image. Install the NVIDIA driver first (Blackwell needs a recent 570+ driver with the open kernel modules), reboot, and check:
nvidia-smi
Ollama is the quickest way to try models:
curl -fsSL https://ollama.com/install.sh | sh
ollama run gpt-oss:120b
ollama run llama3.3:70b # 4-bit by default, about 43 GB
vLLM is what you want for an API that several people or services use. It batches requests and exposes an OpenAI-compatible endpoint:
python3 -m venv ~/vllm && . ~/vllm/bin/activate
pip install vllm
vllm serve Qwen/Qwen2.5-32B-Instruct --max-model-len 16384
# or a 72B model in 4-bit:
vllm serve Qwen/Qwen2.5-72B-Instruct-AWQ --max-model-len 32768
The API listens on port 8000 at /v1. Don't expose it to the internet as is: there is no authentication. Put it behind nginx with a token, or reach it over an SSH tunnel or WireGuard.
Which of the three servers
The GPU is the same in all three. The difference is everything around it.
- DS-G1 (Intel Core Ultra 7 265, 128 GB RAM, 2 × 1.92 TB NVMe): one model, one API. The right choice for most inference setups, from $1550/month.
- DS-G2 (Xeon Gold 5412U, 24 cores, 512 GB ECC RAM): when the GPU is only part of the job. A vector database in memory for RAG, CPU offloading of a model that doesn't quite fit, preprocessing pipelines next to the model. From $2900/month.
- DS-G3 (same CPU, 768 GB RAM, 4 × 3.84 TB NVMe): fine-tuning on large datasets and keeping many checkpoints and model versions on local disk. From $3600/month.
When this is not for you
Be honest about the usage pattern. A dedicated GPU server bills by the month. If you need a GPU for a few hours a week, hourly cloud GPUs come out cheaper. It pays off when the model runs all day: a production API, an internal assistant for a team, or regular fine-tuning jobs.
Also note: it's one GPU. Training a large model from scratch or serving a 400B model needs several GPUs with a fast interconnect, and that's a different class of hardware.
How to get one
GPU servers are assembled to order. Send a request with the configuration and your workload. We confirm availability and the final price, and the server is ready 1–3 business days after prepayment. Payment is in USDC, USDT, ETH, POL or Monero, the same as for a VPS.