gpt-oss-120b and gpt-oss-20b hardware requirements: VRAM and GPUs

gpt-oss-120b’s weights take about 61 GiB (65 GB) of GPU memory, and OpenAI states that the model runs on a single 80 GB GPU such as an H100; gpt-oss-20b runs within 16 GB. Context adds about 37 KB per token for the 120b in BF16, so a 96 GB card leaves room for several long conversations at once.

The short version

What gpt-oss is, in hardware terms

gpt-oss-120b and gpt-oss-20b are open-weight reasoning models that OpenAI released in August 2025 under the Apache 2.0 license; the repository adds a short usage policy asking users to comply with applicable law. Three design choices set their hardware needs.

• Mixture of experts. Each MoE block holds 128 experts in the 120b and 32 in the 20b, and a router picks 4 per token. Only 5.1B of the 120b’s 116.8B parameters are active for each token, but all of them must sit in memory.
• MXFP4 weights. OpenAI post-trained the models with the MoE weights quantized to MXFP4, about 4.25 bits per parameter. Those weights are more than 90% of the parameters, which is what brings the 120b down to one 80 GB GPU. Attention, the router and the embeddings are not quantized, and OpenAI ran all its evaluations with the same MXFP4 weights.
• Lean attention. Layers alternate between dense attention and a banded window of 128 tokens, and use grouped-query attention with 64 query heads and 8 key-value heads of dimension 64. Context reaches 131,072 tokens, using YaRN on the dense layers.

Both models expect OpenAI’s harmony response format and, according to OpenAI, will not work correctly without it. Transformers’ chat template applies it automatically, and vLLM’s Responses endpoint renders and parses it with the openai-harmony library. Reasoning effort, low, medium or high, is set in the system prompt, for example “Reasoning: high”.

How much VRAM does gpt-oss-120b need?

Start with the weights. OpenAI’s model card puts the 120b checkpoint at 60.8 GiB; the safetensors files on Hugging Face add up to 65.25 GB, the same amount in decimal units, and vLLM’s launch post gives 63 GB. A single GPU serving the model needs that much memory before the first token of context.

The rest of the card goes to the KV cache, activations and the runtime. That is why 80 GB is the practical floor and why it is tight. OpenAI’s vLLM guide says the 120b is “best with ≥60GB VRAM” and “can fit on a single H100 or multi-GPU setups”. vLLM’s recipe says it fits on a single 80 GB A100, but warns that on an H100 with one GPU, default settings can run out of memory; its suggested command raises --gpu-memory-utilization to 0.95 and lowers --max-num-batched-tokens to 1024.

Two things don’t work as a shortcut:

• Unquantized weights. OpenAI’s reference PyTorch implementation upcasts every weight to BF16 and needs at least four H100 GPUs. Use the MXFP4 checkpoint.
• A 48 GB card. The weights alone are larger than an L40S’s memory, so part of the model has to live in system RAM, as explained below.

Plan disk space as well. The Hugging Face repository holds the weights twice: once in the format that vLLM and Transformers load, and once in an original folder for OpenAI’s reference code, about 130 GB in total if you download everything.

gpt-oss-20b requirements

gpt-oss-20b has 20.9B parameters, 3.6B of them active per token, in 24 layers with 32 experts. Its checkpoint is 12.8 GiB, or 13.8 GB of files on Hugging Face. OpenAI says it runs within 16 GB of memory; its cookbook says it “only requires about 16GB of VRAM” with vLLM and is “best with ≥16GB VRAM or unified memory” with Ollama.

On a 16 GB GPU, only about 2 GB remain for context and the runtime, which limits you to short conversations. A 24 GB card such as the RTX PRO 4000 leaves about 8 GB at vLLM’s default memory budget, and a 48 GB L40S about 30 GB. According to OpenAI, the 20b can even be fine-tuned on consumer hardware, while the 120b can be fine-tuned on a single H100 node.

KV cache: how much memory context takes

During generation, each attention layer stores a key and a value vector for every token in the context. With 8 key-value heads of dimension 64, that is 2,048 bytes per token per layer in BF16. gpt-oss saves half of it: only the dense layers keep every token, while the banded-window layers keep the last 128. vLLM’s hybrid KV cache manager, which names gpt-oss among the models that mix the two layer types, frees cache blocks that fall outside the window.

• gpt-oss-120b: 18 dense layers × 2,048 bytes = 36,864 bytes, about 37 KB per token.
• gpt-oss-20b: 12 dense layers × 2,048 bytes = 24,576 bytes, about 25 KB per token.
• Window layers: a fixed 4.7 MB per conversation for the 120b and 3.1 MB for the 20b, whatever its length.
• FP8 KV cache: one byte per value instead of two, so half the figures above. vLLM’s Blackwell configuration for gpt-oss turns it on with kv-cache-dtype: fp8.

Multiply by the number of conversations you serve at once. vLLM sizes the cache automatically from the memory left within its budget, set by --gpu-memory-utilization (0.92 by default), unless you fix it with --kv-cache-memory-bytes. --max-model-len caps the context per request, and vLLM’s recipe notes that the largest batch you can run is limited by the memory left for the KV cache once the weights are loaded.

Can you run gpt-oss-120b on one GPU?

Yes, on a GPU with 80 GB of memory or more. One GPU is also what vLLM recommends for the best throughput per GPU, with 2, 4 or 8 GPUs for lower per-user latency. The table below estimates what is left for the KV cache on single cards, using the marketed memory size, decimal gigabytes and vLLM’s default 92% budget.

These are upper bounds: activations and CUDA graphs come out of the same budget, so the real cache is smaller. The thin margin in the H100 row is consistent with the out-of-memory warning in vLLM’s recipe. On a 96 GB RTX PRO 6000, the same weights leave nearly three times as much room: about four full-length conversations in BF16, nine in FP8, or dozens of 8,192-token ones, before runtime overhead.

On a 48 GB L40S, the 120b only runs with part of the weights in system RAM. vLLM’s --cpu-offload-gb option treats CPU memory as extra GPU memory, but its documentation warns that this needs a fast CPU-GPU interconnect, because offloaded weights move to the GPU on every forward pass. Ollama splits a model between CPU and GPU on its own. OpenAI’s cookbook sums it up: you can offload to CPU if you’re short on VRAM, “but expect it to run slower”.

GPU support: what the documentation says

MXFP4 is a recent format, and fast kernels for it depend on both the inference engine and the GPU generation. This is what the documentation said in October 2026:

• vLLM’s launch post (5 August 2025) announced support on NVIDIA Blackwell and Hopper GPUs and on AMD MI300X and MI355X, with a FlashInfer MoE kernel that uses Blackwell’s native MXFP4 tensor cores and a Triton kernel optimized for Hopper.
• vLLM’s gpt-oss recipe lists NVIDIA H100, H200 and B200, AMD MI300X, MI325X and MI355X, and the Radeon AI PRO R9700, and says work is under way to make the model run on Ampere, Ada Lovelace and the RTX 5090. It documents an Ampere path for the A100, with Triton attention and the Marlin MXFP4 MoE kernel. Its example launch script applies the Blackwell settings, FlashInfer MXFP4+MXFP8 MoE and an FP8 KV cache, only when the GPU reports compute capability 10.0, as the B200 does; every other GPU gets the Hopper configuration.
• Compute capability. NVIDIA lists 12.0 for the RTX PRO 6000 Blackwell, the RTX PRO 4000 Blackwell and the GeForce RTX 5090, and 8.9 for the L40S, an Ada Lovelace card.
• vLLM’s quantization table marks the Marlin kernels (GPTQ, AWQ, FP8 and FP4) as supported on Ampere, Ada and Hopper. The table has no Blackwell column.
• NVIDIA NIM, which serves both models with vLLM, lists the L40S, the RTX PRO 4500 Blackwell Server Edition and the RTX PRO 6000 Blackwell Server Edition among the verified GPUs for both models, and includes the RTX PRO 6000 Blackwell Server Edition in its optimized single-GPU MXFP4 profile for gpt-oss-120b.
• Ollama runs MXFP4 natively, with kernels written for its new engine, and announced work with NVIDIA on performance for GeForce RTX and RTX PRO GPUs.

The practical reading: single-GPU gpt-oss-120b on the RTX PRO 6000 Blackwell Server Edition is documented by NVIDIA’s NIM profile, while vLLM’s own tuned Blackwell path targets the B200. On the L40S, vLLM’s quantization table covers Ada through Marlin even though the gpt-oss recipe still lists Ada as work in progress. Whatever engine you choose, test the exact version on the exact card before you plan capacity, and don’t rely on published speed figures measured on other GPUs.

Serving gpt-oss with vLLM

vLLM exposes an OpenAI-compatible API. These commands follow its gpt-oss recipe:

# Python environment and the official vLLM release
uv venv
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

# gpt-oss-120b on one GPU, API on port 8000
vllm serve openai/gpt-oss-120b

# Shorter context limit, plus function calling
vllm serve openai/gpt-oss-120b --max-model-len 32768 --tool-call-parser openai --enable-auto-tool-choice

# gpt-oss-20b in the official container
docker run --gpus all -p 8000:8000 --ipc=host vllm/vllm-openai --model openai/gpt-oss-20b

• Endpoints. /v1/responses supports tool use between reasoning steps and is the interface OpenAI recommends for this model; /v1/chat/completions returns reasoning and final text without built-in tools.
• Memory flags. --max-model-len caps the context per request, and the recipe notes that a lower value can improve performance when inputs and outputs are short; --max-num-seqs caps the batch; --kv-cache-dtype fp8 halves the cache per token.
• Known gaps. The recipe lists usage accounting that returns zeros, basic streaming in the Responses API, and function calling limited to tool_choice="auto".

Running gpt-oss with Ollama

Ollama handles the download and, when memory runs short, the split between GPU and CPU:

ollama pull gpt-oss:120b
ollama run gpt-oss:120b

# Larger default context for the models this server loads
OLLAMA_CONTEXT_LENGTH=64000 ollama serve

# Check where the model is loaded, GPU or CPU
ollama ps

Ollama picks a default context length from the GPU’s memory: 4k tokens below 24 GiB, 32k from 24 to 48 GiB and 256k at 48 GiB or more. It recommends at least 64,000 tokens for agents, coding tools and web search, and warns that a longer context needs more memory. On a 24 GB card, set the context length explicitly instead of relying on the default, and use ollama ps to confirm that nothing spilled to the CPU.

Which PowerDedi server fits which model

For gpt-oss-120b, the RTX PRO 6000 is the one that fits on a single card with headroom, and its 384 GB of host memory and Gen5 NVMe leave plenty of space for the weights and a model cache. For gpt-oss-20b, the RTX PRO 4000 covers a single application with moderate context, and the L40S serves many more parallel conversations. Every server has its own 10 Gbps unmetered port, IPMI/KVM access and one-click Ubuntu or Debian; our getting-started guide covers the first steps after delivery. See also our AI and GPU compute overview.

Where we are not the right fit:

• Fine-tuning gpt-oss-120b. OpenAI points to a single H100 node. We sell one GPU per server, with no NVLink and no multi-GPU clusters.
• Lowest latency per user. vLLM reaches it with tensor parallelism across 2, 4 or 8 GPUs, which needs a multi-GPU machine.
• A few hours of testing. Hourly clouds cost less for short use; our RTX PRO 6000 rental comparison shows where the break-even point lies.

Sources

• gpt-oss-120b & gpt-oss-20b Model Card — OpenAI, arXiv:2508.10925, August 2025
• openai/gpt-oss-120b — OpenAI on Hugging Face
• openai/gpt-oss — OpenAI on GitHub
• How to run gpt-oss with vLLM — OpenAI Cookbook
• How to run gpt-oss locally with Ollama — OpenAI Cookbook
• GPT OSS — vLLM Recipes
• vLLM Now Supports gpt-oss — vLLM Blog, 5 August 2025
• Hybrid KV Cache Manager — vLLM documentation
• Engine Arguments — vLLM documentation
• Quantization — vLLM documentation
• OpenAI gpt-oss — Ollama Blog, 5 August 2025
• Context length — Ollama documentation
• Support Matrix for NIMs — NVIDIA NIM for LLM and VLM
• CUDA GPU Compute Capability — NVIDIA Developer