Model Deployment Guide
openai/gpt-oss-20b Hardware, Architecture, and Deployment Guide
GPT-OSS is OpenAI's open-weight release, shipped as mixture-of-experts models (roughly 20B and 120B total parameters) under a permissive Apache-2.0 license, aimed at teams that want an OpenAI-lineage model they can self-host. At roughly 22B parameters this is a large model: expect a 24–48 GB GPU in quantized form, or a data-center card for FP16 serving. This page covers what gpt-oss-20b is for, what its architecture implies for memory, how much VRAM to budget across precisions, and when quantization or an alternative model makes more sense.
Overview
GPT-OSS is OpenAI's open-weight release, shipped as mixture-of-experts models (roughly 20B and 120B total parameters) under a permissive Apache-2.0 license, aimed at teams that want an OpenAI-lineage model they can self-host. At roughly 22B parameters this is a large model: expect a 24–48 GB GPU in quantized form, or a data-center card for FP16 serving. This page covers what gpt-oss-20b is for, what its architecture implies for memory, how much VRAM to budget across precisions, and when quantization or an alternative model makes more sense.
Architecture
The detected architecture is Gpt Oss, reporting 24 layers, 64 attention heads, 8 key-value heads, and a context window of 131,072 tokens. It uses grouped-query attention (64 attention heads sharing 8 key-value heads), which shrinks the KV cache substantially versus full multi-head attention and helps long-context serving. The config indicates a mixture-of-experts layout with 32 experts and 4 active experts per token, so all experts occupy memory even though only a few are active per token.
Hardware Requirements
Budget about 50 GB for FP16/BF16, 25 GB for 8-bit, and 12 GB for 4-bit weights. Its 131,072-token context enables long-document and repository-scale workloads, but note that filling that window makes the KV cache the dominant memory cost — often larger than the weights. These are weight-plus-overhead planning numbers; add the KV cache for your real context length, since it is stored in FP16 even when the weights are quantized.
Deployment Advice
The 20B variant is designed to fit a single high-memory consumer or workstation GPU, while the 120B targets a data-center card or multi-GPU serving; both are MoE, so only a subset of experts activates per token but every expert must be resident in memory. For a large-tier model like this, a single consumer GPU is practical only when the chosen precision plus the KV cache fits with safety margin. If the FP16 estimate exceeds your GPU by more than a small margin, plan for quantization, CPU offload, or tensor-parallel serving before committing.
Quantization Guidance
GPT-OSS ships in a native low-precision (MXFP4) expert format, so much of the memory saving is already built in — when quantizing further, confirm your serving stack supports that format and validate tool-calling and reasoning output rather than assuming parity with the released weights. GGUF suits llama.cpp and local desktop workflows, AWQ is common for efficient GPU serving, and GPTQ remains useful when prebuilt kernels and model availability match your stack. Mixture-of-experts models can be more sensitive to aggressive quantization of the router, so validate outputs after quantizing.
Comparison Notes
Compare gpt-oss-20b against nearby sizes in the GPT-OSS family and against adjacent open families before committing: DeepSeek R1 for reasoning-heavy workloads, Qwen for multilingual and coding breadth, Gemma for compact deployment, and Llama for the broadest ecosystem support. The right choice depends on whether your constraint is quality, latency, license, or GPU budget.
| Deployment Question | Practical Answer |
|---|---|
| Best first hardware check | Compare FP16, INT8, and INT4 estimates against available VRAM with room for KV cache. |
| When to use tensor parallelism | Use it when the model plus runtime overhead does not fit one GPU or latency improves with sharding. |
| When to quantize | Quantize after creating a full-precision quality baseline and rerunning representative prompts. |