Running Claude Code against a local model is not a GPU problem. The GPU part is a config file. The real problem is the seams: every place where Claude Code assumes an Anthropic backend and your local stack quietly disagrees. I’ve built this stack twice now, once on Qwen 3.6 and once on Qwen 3.8, and the difference between the two attempts taught me the rule this post argues for: a local agentic stack is done when the patches run out, and until then every patch deserves an upstream address.

A fair question is: why fight these seams at all instead of picking a harness built for open backends? Aider, Pi, and Roo Code all speak OpenAI natively and would have run against this stack on day one. The answer is that Claude Code is where I do my professional work, and the harness matters more than the backend. I’d rather carry one set of muscle memory, one config, and one set of habits across every model I touch than keep a second-choice environment around because it’s easier to wire up. The best harness is worth using everywhere, even when everywhere takes more work to set up. That premise is what makes the seams worth closing instead of avoiding.

The result is BlackwellQwen38: six docker compose variants of Qwen 3.8 27B, with Claude Code working end to end. Four run at the model’s full 1M-token context on a single RTX PRO 6000 (96 GB); two more scale the same stack down to a 32 GB RTX 5090 and a 24 GB 3090 or 4090. Tool calls, web search, and hour-scale video ingest included. Nothing in it is a monkeypatch. Everything it changes about vLLM, LiteLLM, or transformers is a pinned fork commit with a PR under review.

Why Qwen 3.8

The stack is only worth building if the model clears the bar for real work, and this is the first open-weights single-GPU release where the answer was obvious. On Artificial Analysis, Qwen 3.8 27B benchmarks on par with ChatGPT Luna and ahead of Opus 4.6. Benchmarks are benchmarks, so here’s the framing I actually trust: its coding ability today is where the frontier models were at the beginning of the year, and nobody called start-of-year frontier models unusable for serious agentic work. That capability on my own card, with a 1M context and no meter running, is past the point where “local” means “toy”. It is a monumental step up from Qwen 3.6 in both performance and usability.

Attempt one: paving the seams

The Qwen 3.6 version ran the way a machine held together with zip ties and hot glue runs: fine until you touch it, but I was the only one who knew where the zip ties were. Getting agentic loops stable took weeks of tracing, and the failure modes were the kind that don’t show up in a chat demo.

Turns would end abruptly. The model would take a tool result, emit a token or two, and stop, leaving Claude Code staring at an empty turn mid-task. The fix I landed on was a minimum-token floor injected into every request, which is exactly as crude as it sounds.

Tool calls would fragment. Under streaming, the model’s tool-call tags could split across deltas, and the parser would occasionally drop or mangle the call. Enable speculative decoding and the draft boundaries added a second way to split them. I ended up bind-mounting an instrumented copy of vLLM’s parser into the container just to log every delta to a firehose file, because there was no other way to see where a tool call died.

By the end, the stack ran a custom chat template, a logits processor that forced reasoning blocks closed (and made speculative decoding impossible), patched tool parsers, and a stack of sitecustomize shims, all mounted over files inside the container image. It worked on exactly one machine. Every image upgrade meant re-porting the patches, and none of it was shareable, because “mount these dozen files over your vLLM install” is not a stack, it’s a crime scene.

The rebuild rule

Qwen 3.8 shipped with better templates, better parsers, and MTP speculative decoding heads in the checkpoint, so the rebuild started from a stack instead of rubble. The rule was: just run the stock model with the settings the model card recommends, and when something still breaks, fix it as a real commit that can go upstream. No mounted files, no shims, no exceptions.

Then I walked the seams. There were more than I expected, and each one is a small story about what “Anthropic-shaped client” means.

Reasoning effort. Claude Code sends reasoning_effort values like high, max, and minimal. Qwen’s chat template accepts low, medium, and xhigh, and returns a 400 for anything else, which kills every request with thinking enabled. Fix: a vLLM flag that remaps unsupported efforts to the nearest supported level (vllm#52739).

Mid-turn system messages. Recent Claude Code sends system-role reminder messages in the middle of the conversation. OpenAI’s API accepts system messages anywhere; Qwen’s template rejects any that aren’t first, with "System message must be at the beginning." The practical effect is brutal: the first turn works, and every session dies on the second. Fix: a LiteLLM option to rewrite those reminders as user turns, or drop them entirely (litellm#37351). I default to drop. It costs the reminder content, and it buys something worth having on a local backend: the conversation prefix stops churning between turns, so vLLM’s prefix cache keeps hitting.

WebSearch. Claude Code runs a web search by sending a request whose only tool is Anthropic’s server-side search tool. A local model can’t search anything, so the request just errors. I built litellm-claude-code-websearch, a LiteLLM plugin that intercepts the standalone search request, runs the query through Brave, and answers in the native block format, so Claude Code renders “Did 1 search” and result links exactly as it does against the real API. Streaming those blocks needed one more LiteLLM fix (litellm#37318).

The video rabbit hole

Qwen 3.8 is vision-native, and its model card has a “Long Video Understanding” recipe: raise the video processor’s pixel budget to about 224K tokens so hour-scale videos sample enough frames. The card’s instructions for applying it are to edit a JSON file inside the downloaded checkpoint. That’s not configuration, that’s surgery. It isn’t reproducible from your serve command, and it silently reverts the next time the model re-downloads.

Doing it properly turned into a three-project chain:

  • vLLM’s --mm-processor-kwargs couldn’t express the override safely. A flat size override leaks into the image processor too, inflating the maximum image from 16K tokens to 448K, and memory profiling grinds on the giant dummy (vllm#52834). Worse, the officially documented recipe is exactly this flat form.
  • With the budget raised, the profiling dummy item exceeds the multimodal processor cache and the engine refuses to boot with ValueError("value too large") from deep inside cachetools (vllm#52835).
  • Short clips still cost as much as long ones, because the processor spends the whole budget on whatever frames it sampled. A 90-second clip cost 184K tokens at near-native frame resolution. I first fixed this as a vLLM flag (vllm#52754); the maintainers pointed out the knob belongs in the HF processor, so it moved to transformers (transformers#48071). The reference implementation in qwen-vl-utils has capped per-frame cost this way since Qwen2-VL, which makes the uncapped behavior the odd one out.

With scoped videos_kwargs, the cache guard, and the per-frame cap, the entire recipe collapses into one serve flag. The 90-second clip drops from 184K to 53K tokens, the video budget no longer touches images, and the stack takes two videos per prompt for clip comparison. A 1080p feature film comes in around 225K prompt tokens with a first answer in about two minutes. The cost is context: profiling reserves ~26 GB for the worst-case video, so the video variant runs 500K context instead of 1M.

Shipping it

The fork deltas are all frontend Python, and that’s the trick that makes the whole thing shippable: CI builds the images as pure-Python overlays on pinned official images. No kernel compilation, so the builds run on free GitHub runners in minutes and publish to ghcr. When a fork base has no matching nightly image, vLLM’s per-commit wheel index fills the gap. The compose files pull the published images, and each carries a build: block that reproduces them locally from pinned fork tags, so the repo works with nothing but itself.

The lineup, tok/s measured single-stream on the blackwell card:

Variant Card Context Decode The pitch
FP8 96 GB RTX PRO 6000 1M 84 tok/s The daily driver: official quant, near-lossless
BF16 96 GB 1M 56.5 tok/s The quality reference
NVFP4 96 GB 1M 113 tok/s The speed pick, a few accuracy points down
FP8 video 96 GB 500K 84 tok/s The long-video recipe above
NVFP4 small 32 GB (RTX 5090) 128K 105 tok/s Same checkpoint on one consumer card, speculative decoding kept
AWQ micro 24 GB (RTX 3090/4090) 64K 73 tok/s W4A16 quant, runs on Ampere; performance will be lower with 3090’s bandwidth

All six share the same two images and the same LiteLLM claude-* wildcard route, so claude pointed at the proxy just works.

The two small tiers were also tested on my Blackwell card, sized by capping vLLM’s memory budget on my card to the target card’s exact envelope. The NVFP4 checkpoint’s weights alone load at 22 GiB, so the 24 GB tier needed a leaner checkpoint entirely. And on a small card the KV cache is the scarce resource, at roughly 40 KiB per token. Every 26K tokens of context is another GiB of VRAM, and the compose headers price the features in exactly those terms: the speculative-decoding head costs another 20K tokens.

What the second attempt actually bought

The 3.6 stack carried roughly a dozen mounted patches, and every one of them was my problem forever. The 3.8 stack carries zero. Its behavior changes live in three vLLM contributions, two LiteLLM contributions, one transformers contribution, and two vLLM issues with fix branches attached, all linked from the repo README. Each merge shrinks the fork. The endgame is six compose files pulling stock images, and the repo is built to shrink into exactly that.

Monkeypatches are loans against every future upgrade, and agentic workloads are where the interest compounds, because Claude Code exercises the ugly paths: streaming tool calls, giant contexts, weird message shapes, ten-hour sessions. If a fix is worth mounting over a file in a container, it’s worth submitting as a commit upstream. When the last one merges, this stack stops being a project. That’s the goal.