--- title: "gpt-oss-120b Hardware Requirements: VRAM and GPUs" description: "gpt-oss-120b needs about 65 GB for its weights and runs on one 80 GB GPU; gpt-oss-20b fits in 16 GB. KV-cache math, vLLM and Ollama setup, and GPU choice." url: "https://powerdedi.com/blog/gpt-oss-120b-hardware-requirements" last_updated: "2026-10-15" publisher: "PowerDedi (https://powerdedi.com)" --- # gpt-oss-120b and gpt-oss-20b hardware requirements: VRAM and GPUs What OpenAI’s gpt-oss-120b and gpt-oss-20b need to run: weight sizes from the model card, the memory each token of context takes, single-GPU serving with vLLM and Ollama, documented GPU support, and which 24, 48 and 96 GB cards fit which model. By PowerDedi Engineering · October 15, 2026 · 10 min read **Share this article** gpt-oss-120b’s weights take about 61 GiB (65 GB) of GPU memory, and OpenAI states that the model runs on a single 80 GB GPU such as an H100; gpt-oss-20b runs within 16 GB. Context adds about 37 KB per token for the 120b in BF16, so a 96 GB card leaves room for several long conversations at once. ## The short version | | gpt-oss-120b | gpt-oss-20b | | --- | --- | --- | | Parameters, total / active per token | 116.8B / 5.1B | 20.9B / 3.6B | | Architecture | Mixture of experts, 36 layers, 128 experts, 4 active | Mixture of experts, 24 layers, 32 experts, 4 active | | Weights (MoE layers in MXFP4) | 60.8 GiB (65.3 GB) | 12.8 GiB (13.8 GB) | | OpenAI’s memory guidance | A single 80 GB GPU | Within 16 GB of memory | | Context window | 131,072 tokens | 131,072 tokens | | KV cache per token of context, BF16 / FP8 | About 37 / 18 KB | About 25 / 12 KB | | Fits on one | RTX PRO 6000 (96 GB), not on the L40S (48 GB) without offloading | RTX PRO 4000 (24 GB), L40S, RTX PRO 6000 | | License | Apache 2.0 | Apache 2.0 | ## What gpt-oss is, in hardware terms gpt-oss-120b and gpt-oss-20b are open-weight reasoning models that OpenAI released in August 2025 under the Apache 2.0 license; the repository adds a short usage policy asking users to comply with applicable law. Three design choices set their hardware needs. - **Mixture of experts.** Each MoE block holds 128 experts in the 120b and 32 in the 20b, and a router picks 4 per token. Only 5.1B of the 120b’s 116.8B parameters are active for each token, but all of them must sit in memory. - **MXFP4 weights.** OpenAI post-trained the models with the MoE weights quantized to MXFP4, about 4.25 bits per parameter. Those weights are more than 90% of the parameters, which is what brings the 120b down to one 80 GB GPU. Attention, the router and the embeddings are not quantized, and OpenAI ran all its evaluations with the same MXFP4 weights. - **Lean attention.** Layers alternate between dense attention and a banded window of 128 tokens, and use grouped-query attention with 64 query heads and 8 key-value heads of dimension 64. Context reaches 131,072 tokens, using YaRN on the dense layers. Both models expect OpenAI’s harmony response format and, according to OpenAI, will not work correctly without it. Transformers’ chat template applies it automatically, and vLLM’s Responses endpoint renders and parses it with the openai-harmony library. Reasoning effort, low, medium or high, is set in the system prompt, for example “Reasoning: high”. ## How much VRAM does gpt-oss-120b need? Start with the weights. OpenAI’s model card puts the 120b checkpoint at 60.8 GiB; the safetensors files on Hugging Face add up to 65.25 GB, the same amount in decimal units, and vLLM’s launch post gives 63 GB. A single GPU serving the model needs that much memory before the first token of context. The rest of the card goes to the KV cache, activations and the runtime. That is why 80 GB is the practical floor and why it is tight. OpenAI’s vLLM guide says the 120b is “best with ≥60GB VRAM” and “can fit on a single H100 or multi-GPU setups”. vLLM’s recipe says it fits on a single 80 GB A100, but warns that on an H100 with one GPU, default settings can run out of memory; its suggested command raises `--gpu-memory-utilization` to 0.95 and lowers `--max-num-batched-tokens` to 1024. Two things don’t work as a shortcut: - **Unquantized weights.** OpenAI’s reference PyTorch implementation upcasts every weight to BF16 and needs at least four H100 GPUs. Use the MXFP4 checkpoint. - **A 48 GB card.** The weights alone are larger than an L40S’s memory, so part of the model has to live in system RAM, as explained below. Plan disk space as well. The Hugging Face repository holds the weights twice: once in the format that vLLM and Transformers load, and once in an `original` folder for OpenAI’s reference code, about 130 GB in total if you download everything. ## gpt-oss-20b requirements gpt-oss-20b has 20.9B parameters, 3.6B of them active per token, in 24 layers with 32 experts. Its checkpoint is 12.8 GiB, or 13.8 GB of files on Hugging Face. OpenAI says it runs within 16 GB of memory; its cookbook says it “only requires about 16GB of VRAM” with vLLM and is “best with ≥16GB VRAM or unified memory” with Ollama. On a 16 GB GPU, only about 2 GB remain for context and the runtime, which limits you to short conversations. A 24 GB card such as the RTX PRO 4000 leaves about 8 GB at vLLM’s default memory budget, and a 48 GB L40S about 30 GB. According to OpenAI, the 20b can even be fine-tuned on consumer hardware, while the 120b can be fine-tuned on a single H100 node. ## KV cache: how much memory context takes During generation, each attention layer stores a key and a value vector for every token in the context. With 8 key-value heads of dimension 64, that is 2,048 bytes per token per layer in BF16. gpt-oss saves half of it: only the dense layers keep every token, while the banded-window layers keep the last 128. vLLM’s hybrid KV cache manager, which names gpt-oss among the models that mix the two layer types, frees cache blocks that fall outside the window. - **gpt-oss-120b:** 18 dense layers × 2,048 bytes = 36,864 bytes, about 37 KB per token. - **gpt-oss-20b:** 12 dense layers × 2,048 bytes = 24,576 bytes, about 25 KB per token. - **Window layers:** a fixed 4.7 MB per conversation for the 120b and 3.1 MB for the 20b, whatever its length. - **FP8 KV cache:** one byte per value instead of two, so half the figures above. vLLM’s Blackwell configuration for gpt-oss turns it on with `kv-cache-dtype: fp8`. | Context per conversation | gpt-oss-120b, BF16 | gpt-oss-120b, FP8 | gpt-oss-20b, BF16 | gpt-oss-20b, FP8 | | --- | --- | --- | --- | --- | | 8,192 tokens | 0.30 GB | 0.15 GB | 0.20 GB | 0.10 GB | | 32,768 tokens | 1.21 GB | 0.60 GB | 0.81 GB | 0.40 GB | | 131,072 tokens (maximum) | 4.83 GB | 2.42 GB | 3.22 GB | 1.61 GB | Multiply by the number of conversations you serve at once. vLLM sizes the cache automatically from the memory left within its budget, set by `--gpu-memory-utilization` (0.92 by default), unless you fix it with `--kv-cache-memory-bytes`. `--max-model-len` caps the context per request, and vLLM’s recipe notes that the largest batch you can run is limited by the memory left for the KV cache once the weights are loaded. ## Can you run gpt-oss-120b on one GPU? Yes, on a GPU with 80 GB of memory or more. One GPU is also what vLLM recommends for the best throughput per GPU, with 2, 4 or 8 GPUs for lower per-user latency. The table below estimates what is left for the KV cache on single cards, using the marketed memory size, decimal gigabytes and vLLM’s default 92% budget. | GPU | Model | Weights | Left within vLLM’s budget | Context that fits, BF16 (upper bound) | Full 131,072-token conversations, BF16 / FP8 | | --- | --- | --- | --- | --- | --- | | H100, 80 GB (reference) | 120b | 65.3 GB | About 8 GB | About 225,000 tokens | 1 / 3 | | RTX PRO 6000, 96 GB | 120b | 65.3 GB | About 23 GB | About 625,000 tokens | 4 / 9 | | L40S, 48 GB | 120b | 65.3 GB | Does not fit | — | Only with CPU offloading | | L40S, 48 GB | 20b | 13.8 GB | About 30 GB | About 1.2 million tokens | 9 / 18 | | RTX PRO 4000, 24 GB | 20b | 13.8 GB | About 8 GB | About 340,000 tokens | 2 / 5 | These are upper bounds: activations and CUDA graphs come out of the same budget, so the real cache is smaller. The thin margin in the H100 row is consistent with the out-of-memory warning in vLLM’s recipe. On a 96 GB RTX PRO 6000, the same weights leave nearly three times as much room: about four full-length conversations in BF16, nine in FP8, or dozens of 8,192-token ones, before runtime overhead. On a 48 GB L40S, the 120b only runs with part of the weights in system RAM. vLLM’s `--cpu-offload-gb` option treats CPU memory as extra GPU memory, but its documentation warns that this needs a fast CPU-GPU interconnect, because offloaded weights move to the GPU on every forward pass. Ollama splits a model between CPU and GPU on its own. OpenAI’s cookbook sums it up: you can offload to CPU if you’re short on VRAM, “but expect it to run slower”. ## GPU support: what the documentation says MXFP4 is a recent format, and fast kernels for it depend on both the inference engine and the GPU generation. This is what the documentation said in October 2026: - **vLLM’s launch post** (5 August 2025) announced support on NVIDIA Blackwell and Hopper GPUs and on AMD MI300X and MI355X, with a FlashInfer MoE kernel that uses Blackwell’s native MXFP4 tensor cores and a Triton kernel optimized for Hopper. - **vLLM’s gpt-oss recipe** lists NVIDIA H100, H200 and B200, AMD MI300X, MI325X and MI355X, and the Radeon AI PRO R9700, and says work is under way to make the model run on Ampere, Ada Lovelace and the RTX 5090. It documents an Ampere path for the A100, with Triton attention and the Marlin MXFP4 MoE kernel. Its example launch script applies the Blackwell settings, FlashInfer MXFP4+MXFP8 MoE and an FP8 KV cache, only when the GPU reports compute capability 10.0, as the B200 does; every other GPU gets the Hopper configuration. - **Compute capability.** NVIDIA lists 12.0 for the RTX PRO 6000 Blackwell, the RTX PRO 4000 Blackwell and the GeForce RTX 5090, and 8.9 for the L40S, an Ada Lovelace card. - **vLLM’s quantization table** marks the Marlin kernels (GPTQ, AWQ, FP8 and FP4) as supported on Ampere, Ada and Hopper. The table has no Blackwell column. - **NVIDIA NIM**, which serves both models with vLLM, lists the L40S, the RTX PRO 4500 Blackwell Server Edition and the RTX PRO 6000 Blackwell Server Edition among the verified GPUs for both models, and includes the RTX PRO 6000 Blackwell Server Edition in its optimized single-GPU MXFP4 profile for gpt-oss-120b. - **Ollama** runs MXFP4 natively, with kernels written for its new engine, and announced work with NVIDIA on performance for GeForce RTX and RTX PRO GPUs. The practical reading: single-GPU gpt-oss-120b on the RTX PRO 6000 Blackwell Server Edition is documented by NVIDIA’s NIM profile, while vLLM’s own tuned Blackwell path targets the B200. On the L40S, vLLM’s quantization table covers Ada through Marlin even though the gpt-oss recipe still lists Ada as work in progress. Whatever engine you choose, test the exact version on the exact card before you plan capacity, and don’t rely on published speed figures measured on other GPUs. ## Serving gpt-oss with vLLM vLLM exposes an OpenAI-compatible API. These commands follow its gpt-oss recipe: ```bash # Python environment and the official vLLM release uv venv source .venv/bin/activate uv pip install vllm --torch-backend=auto # gpt-oss-120b on one GPU, API on port 8000 vllm serve openai/gpt-oss-120b # Shorter context limit, plus function calling vllm serve openai/gpt-oss-120b --max-model-len 32768 --tool-call-parser openai --enable-auto-tool-choice # gpt-oss-20b in the official container docker run --gpus all -p 8000:8000 --ipc=host vllm/vllm-openai --model openai/gpt-oss-20b ``` - **Endpoints.** `/v1/responses` supports tool use between reasoning steps and is the interface OpenAI recommends for this model; `/v1/chat/completions` returns reasoning and final text without built-in tools. - **Memory flags.** `--max-model-len` caps the context per request, and the recipe notes that a lower value can improve performance when inputs and outputs are short; `--max-num-seqs` caps the batch; `--kv-cache-dtype fp8` halves the cache per token. - **Known gaps.** The recipe lists usage accounting that returns zeros, basic streaming in the Responses API, and function calling limited to `tool_choice="auto"`. ## Running gpt-oss with Ollama Ollama handles the download and, when memory runs short, the split between GPU and CPU: ```bash ollama pull gpt-oss:120b ollama run gpt-oss:120b # Larger default context for the models this server loads OLLAMA_CONTEXT_LENGTH=64000 ollama serve # Check where the model is loaded, GPU or CPU ollama ps ``` Ollama picks a default context length from the GPU’s memory: 4k tokens below 24 GiB, 32k from 24 to 48 GiB and 256k at 48 GiB or more. It recommends at least 64,000 tokens for agents, coding tools and web search, and warns that a longer context needs more memory. On a 24 GB card, set the context length explicitly instead of relying on the default, and use `ollama ps` to confirm that nothing spilled to the CPU. ## Which PowerDedi server fits which model | Server | Host | gpt-oss-20b | gpt-oss-120b | Price, delivery, cities | | --- | --- | --- | --- | --- | | [RTX PRO 6000 Blackwell](https://powerdedi.com/dedicated-servers/gpu/rtx-pro-6000-blackwell), 96 GB GDDR7 | EPYC 9355P, 384 GB DDR5 ECC, 2 × 3.84 TB NVMe Gen5 | Yes, with a very large KV cache | Yes, on one GPU, with room for long contexts | $899 a month; 72 hours; Amsterdam, Frankfurt | | [L40S](https://powerdedi.com/dedicated-servers/gpu/l40s), 48 GB GDDR6 ECC | EPYC 9355P, 256 GB DDR5 ECC, 2 × 3.84 TB NVMe | Yes, comfortably | Only with CPU offloading, and slower | $549 a month; 72 hours; Amsterdam, Frankfurt | | [RTX PRO 4000 Blackwell](https://powerdedi.com/dedicated-servers/gpu/rtx-pro-4000-blackwell), 24 GB GDDR7 | Ryzen 9 9900X, 64 GB DDR5, 2 × 1.92 TB NVMe | Yes | No | $169 a month; 24 hours; Amsterdam, Frankfurt, New York | For gpt-oss-120b, the RTX PRO 6000 is the one that fits on a single card with headroom, and its 384 GB of host memory and Gen5 NVMe leave plenty of space for the weights and a model cache. For gpt-oss-20b, the RTX PRO 4000 covers a single application with moderate context, and the L40S serves many more parallel conversations. Every server has its own 10 Gbps unmetered port, IPMI/KVM access and one-click Ubuntu or Debian; our [getting-started guide](https://powerdedi.com/docs/getting-started) covers the first steps after delivery. See also our [AI and GPU compute](https://powerdedi.com/solutions/ai-gpu-compute) overview. Where we are not the right fit: - **Fine-tuning gpt-oss-120b.** OpenAI points to a single H100 node. We sell one GPU per server, with no NVLink and no multi-GPU clusters. - **Lowest latency per user.** vLLM reaches it with tensor parallelism across 2, 4 or 8 GPUs, which needs a multi-GPU machine. - **A few hours of testing.** Hourly clouds cost less for short use; our [RTX PRO 6000 rental comparison](https://powerdedi.com/blog/rtx-pro-6000-rental-monthly-vs-hourly) shows where the break-even point lies. ## Sources - [gpt-oss-120b & gpt-oss-20b Model Card](https://arxiv.org/abs/2508.10925) — OpenAI, arXiv:2508.10925, August 2025 - [openai/gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) — OpenAI on Hugging Face - [openai/gpt-oss](https://github.com/openai/gpt-oss) — OpenAI on GitHub - [How to run gpt-oss with vLLM](https://developers.openai.com/cookbook/articles/gpt-oss/run-vllm) — OpenAI Cookbook - [How to run gpt-oss locally with Ollama](https://developers.openai.com/cookbook/articles/gpt-oss/run-locally-ollama) — OpenAI Cookbook - [GPT OSS](https://docs.vllm.ai/projects/recipes/en/latest/OpenAI/GPT-OSS.html) — vLLM Recipes - [vLLM Now Supports gpt-oss](https://vllm.ai/blog/2025-08-05-gpt-oss) — vLLM Blog, 5 August 2025 - [Hybrid KV Cache Manager](https://docs.vllm.ai/en/latest/design/hybrid_kv_cache_manager.html) — vLLM documentation - [Engine Arguments](https://docs.vllm.ai/en/latest/configuration/engine_args.html) — vLLM documentation - [Quantization](https://docs.vllm.ai/en/stable/features/quantization/) — vLLM documentation - [OpenAI gpt-oss](https://ollama.com/blog/gpt-oss) — Ollama Blog, 5 August 2025 - [Context length](https://docs.ollama.com/context-length) — Ollama documentation - [Support Matrix for NIMs](https://docs.nvidia.com/nim/large-language-models/latest/reference/support-matrix.html) — NVIDIA NIM for LLM and VLM - [CUDA GPU Compute Capability](https://developer.nvidia.com/cuda-gpus) — NVIDIA Developer **Share this article** PowerDedi Engineering The team that builds, racks and runs every PowerDedi server. We write about the hardware, networks and costs we deal with every day, and we cite our sources. **Dedicated servers** $55/mo and up About 40% below the market, with a 10 Gbps unmetered port on every server. Paid in crypto. [Browse servers](https://powerdedi.com/dedicated-servers) **More from the blog** - [Guides](https://powerdedi.com/blog/ton-node-requirements) October 21, 2026 **TON node requirements in 2026: full node, liteserver, validator** - [Comparisons](https://powerdedi.com/blog/crypto-dedicated-server-providers-compared) October 15, 2026 **Dedicated servers paid in crypto: 14 providers compared (October 2026)** - [Comparisons](https://powerdedi.com/blog/hetzner-alternatives-2026) October 15, 2026 **Hetzner alternatives in 2026: crypto payment, 10 Gbps, no setup fees** - [Comparisons](https://powerdedi.com/blog/ovhcloud-alternatives-2026) October 15, 2026 **Does OVHcloud accept crypto? OVHcloud alternatives in 2026** ## Related articles [All articles](https://powerdedi.com/blog) [Market](https://powerdedi.com/blog/category/market) October 15, 2026 ### [RTX PRO 6000 rental in 2026: hourly cloud prices vs a monthly server](https://powerdedi.com/blog/rtx-pro-6000-rental-monthly-vs-hourly) What an NVIDIA RTX PRO 6000 costs by the hour at the main GPU clouds and marketplaces in October 2026, the break-even point between hourly billing and a monthly server, the costs the hourly rate leaves out, and when hourly still wins. 10 min read [Hardware](https://powerdedi.com/blog/category/hardware) October 15, 2026 ### [EPYC 4585PX vs Ryzen 9 9950X3D vs 9950X: which AM5 server CPU?](https://powerdedi.com/blog/epyc-4585px-vs-ryzen-9950x3d-vs-9950x) The EPYC 4585PX is AMD’s server-validated twin of the Ryzen 9 9950X3D. How it compares with the 9950X3D and 9950X on ECC, memory capacity and speed, PCIe lanes, published Phoronix benchmarks and price, and when a 12-channel EPYC makes more sense. 10 min read [Hardware](https://powerdedi.com/blog/category/hardware) October 15, 2026 ### [Ryzen 9 9950X vs 9950X3D for game servers: which to rent in 2026](https://powerdedi.com/blog/ryzen-9-9950x-vs-9950x3d-game-servers) The Ryzen 9 9950X and 9950X3D share 16 Zen 5 cores and a 5.7 GHz boost; the X3D adds 64 MB of L3 to one chiplet. What published Linux benchmarks show, how to keep a game server on the cache chiplet, how to measure TPS and MSPT, and which to rent. 12 min read ## Your next server is 15 minutes away. From $55/month with 10 Gbps unmetered, no setup fee and no contract. [Browse servers](https://powerdedi.com/dedicated-servers) [All articles](https://powerdedi.com/blog) Live in ~15 min · Cancel anytime · BTC, ETH, XMR, USDT or SOL --- PowerDedi — Dedicated servers with 10 Gbps unmetered, paid in crypto. Page: https://powerdedi.com/blog/gpt-oss-120b-hardware-requirements. Index of the site for AI assistants: https://powerdedi.com/llms.txt