Serve NVIDIA Nemotron 3 Ultra yourself for high-throughput agents (vLLM)

The Stack

Tested Against

Prerequisites

• A multi-GPU node (8x B200 or 8x H100 NVFP4) or a rented GPU instance, to actually serve it

Steps

1. 1

   Write the serve command + agent endpoint, and validate them

   This is NVIDIA's own 8x B200 NVFP4 example; the BF16 path and full flag set are in the Nemotron vLLM cookbook. Note --served-model-name, so the agent targets that name (not the checkpoint path). CI checks the serve command pins the NVFP4 checkpoint with the documented parsers and mamba backend, and the endpoint is well-formed.

   cat > serve.sh <<'SH'
   docker pull vllm/vllm-openai:v0.22.0 && VLLM_USE_FLASHINFER_MOE_FP4=1 vllm serve nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 --served-model-name nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len 262144 --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --speculative_config.method mtp --speculative_config.num_speculative_tokens 5 --mamba-backend triton
   SH
   cat > endpoint.json <<'JSON'
   {
     "base_url": "http://localhost:8000/v1",
     "api_key": "EMPTY",
     "model": "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B"
   }
   JSON
   node -e 'const fs=require("fs");const s=fs.readFileSync("serve.sh","utf8");const e=JSON.parse(fs.readFileSync("endpoint.json","utf8"));const need=["nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4","--reasoning-parser nemotron_v3","--tool-call-parser qwen3_coder","--mamba-backend triton"];for(const t of need){if(!s.includes(t)){console.log("BAD: serve command missing "+t);process.exit(1)}}if(!String(e.base_url).includes("localhost:8000/v1")){console.log("BAD: endpoint base_url");process.exit(1)}if(e.api_key!=="EMPTY"){console.log("BAD: endpoint api_key should be EMPTY");process.exit(1)}if(e.model!=="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B"){console.log("BAD: endpoint model should match --served-model-name");process.exit(1)}console.log("config OK: serve pins the NVFP4 checkpoint + nemotron_v3/qwen3_coder parsers + triton mamba backend, endpoint targets the served model name")'

2. 2

   Point your agent at it (the model step, not checked by CI)

   Use base_url http://localhost:8000/v1, key EMPTY, and model nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B (the --served-model-name, not the checkpoint path). The serving and agent loops are fenced.