Goal
Maximize inference decode throughput (tokens/sec) for Qwen/Qwen3.6-35B-A3B on Hugging Face Jobs a10g-small (single A10G 24GB).
Hypothesis
Because the dense BF16 model cannot fit in 24GB, fastest practical paths will be quantized runtimes: llama.cpp CUDA with GGUF quants and/or Transformers/vLLM 4-bit/FP8 variants. Start with cheap single-job benchmarks over candidate runtimes/settings, then record the best median decode tok/s.
Method
Use fixed prompt length and generation length, warmup once, then time several decode runs. Record model variant, runtime, quantization, GPU offload settings, memory, and tok/s.