BlackwellQwen38
Qwen 3.8 27B at 1M context on one Blackwell GPU, with Claude Code, WebSearch, and hour-scale video working end to end. Six compose variants from 96 GB down to 24 GB cards, zero monkeypatches.
Highlights
- Six
docker composevariants: FP8 (84 tok/s, recommended), BF16 (quality reference), NVFP4 (113 tok/s), FP8-video (long-video understanding at 500K context), plus sized-down tiers for a 32 GB RTX 5090 (128K context) and a 24 GB 3090/4090 (64K context). - Claude Code works against the local model end to end: tool calls, reasoning effort, mid-turn system messages, and the WebSearch tool all handled.
- The Qwen model card's long-video recipe expressed as pure serve config instead of editing files inside the checkpoint; a 90 s clip costs ~53K tokens instead of ~184K.
- Images build in CI as pure-Python overlays on pinned official images (no kernel compilation) and publish to ghcr; compose files pull them or rebuild locally from pinned fork tags.
- Every patch the stack carries is under review upstream: three vLLM PRs, four LiteLLM PRs, one transformers PR, and two vLLM issues with fix branches attached.
Claude Code assumes an Anthropic backend. Point it at a local model behind a proxy and you find every place that assumption leaks: reasoning_effort values the Qwen chat template rejects, mid-conversation system messages the template refuses outright, a WebSearch tool that expects the server to run searches, and streaming shapes the proxy has to fake. BlackwellQwen38 is the stack where all of those seams are closed on one RTX PRO 6000 (96 GB) at the model’s full 1M-token context.
What it is
Six self-contained compose files, each bundling a vLLM worker and a LiteLLM proxy with a claude-* wildcard route, so any Anthropic-style client lands on local Qwen. The FP8 variant is the daily driver: 84 tok/s single-stream decode with MTP speculative decoding, ~1.7M-token KV pool. NVFP4 trades some accuracy for 113 tok/s. BF16 is the full-precision reference. The video variant runs the model card’s “Long Video Understanding” recipe (a 224K-token video budget for hour-scale frame sampling) at 500K context, ingesting file:// videos on the direct engine port; two clips per prompt, a 1080p feature film at ~225K prompt tokens. Two sized-down tiers bring the same stack to consumer cards, each sized empirically by capping vLLM’s memory budget to the target card’s envelope: a 32 GB RTX 5090 runs NVFP4 at 128K context with speculative decoding kept, and a 24 GB 3090/4090 runs a W4A16 AWQ quant at 64K.
Why the images are forks
Making the seams close required patches in three upstream projects, and the stack refuses to carry anything that isn’t also on its way upstream. The images are built by the repo’s own CI as pure-Python overlays on pinned official images from immutable tags on the vLLM, LiteLLM, and transformers forks. The README carries the full table of what each patch does and which PR tracks it; when they merge, the overlays collapse back into stock images. The WebSearch piece lives in its own package, litellm-claude-code-websearch.
Status
Running as my daily Claude Code backend. Secrets stay in a gitignored .env; the compose files pull the published images or rebuild them locally from nothing but the repo. The write-up of how it came together (and why the first attempt on Qwen 3.6 was a folder of monkeypatches instead) is on the blog.