Goal
Check whether non-llama.cpp runtimes can beat the current llama.cpp GGUF results on a10g-small.
Hypothesis
llama.cpp is likely best for single-GPU 24GB with this 35B-A3B MoE because GGUF dynamic quants fit comfortably. vLLM/Transformers may fail to fit or lack efficient support for the hybrid Qwen3.6 MoE architecture, but a quick smoke benchmark can rule them out.
Method
Run minimal vLLM/Transformers smoke tests with quantized variants. Record whether model loads, memory use, and any decode throughput.
Running
- vLLM GGUF IQ1 smoke test using
vllm/vllm-openai:v0.11.0, local GGUF file, Qwen tokenizer,max_model_len=2048: https://huggingface.co/jobs/abidlabs/6a2fb800871c005b5352d3f5