llama.cpp CUDA GGUF quant/offload sweep

Goal

Benchmark llama.cpp CUDA on a10g-small using Unsloth GGUF quants.

Hypothesis

A mid-size GGUF quant that fully offloads to the 24GB A10G (e.g. IQ3/Q4/MXFP4_MOE) will outperform Transformers 4-bit due to lower runtime overhead and compact weights. Very large Q4/Q5 quants may leave too little VRAM for KV/cache and slow/OOM.

Method

Download one GGUF at a time, run llama-bench/llama-cli with full GPU offload (-ngl 999) and small prompt + fixed decode, sweeping quant and KV cache type where available. Compare decode tokens/sec.

Findings 2026-06-15

Setup notes

• Correct llama.cpp image: ghcr.io/ggml-org/llama.cpp:full-cuda (the older ghcr.io/ggerganov/... tag failed to pull).
• Must set LD_LIBRARY_PATH=/app:$LD_LIBRARY_PATH; otherwise /app/llama-bench cannot find libllama-bench-impl.so.
• Mounting the full GGUF repo as a Jobs volume stayed in scheduling for several minutes; direct hf_hub_download of one GGUF at a time worked (~188 MB/s for 11.5 GB IQ2_M).

Completed result: UD-IQ2_M full GPU offload

Job: https://huggingface.co/jobs/abidlabs/6a2f465b871c005b5352cd21

Runtime: llama.cpp build c2ba3e47a (9628), CUDA backend, A10G 24GB, -ngl 999 -fa 1 -p 512 -n 256 -b 1024 -ub 512 -r 3.

Initial best: 118.42 decode tok/s with F16 KV. Quantized KV did not help at this short context; it slightly reduced decode throughput.

Running next

• UD-Q3_K_M full-offload KV sweep: https://huggingface.co/jobs/abidlabs/6a2f47b6871c005b5352cd4b
• UD-Q4_K_S full-offload attempt with fallback ngl=36: https://huggingface.co/jobs/abidlabs/6a2f47c9234ca64b60123ea4

Completed result: UD-Q3_K_M full GPU offload

Job: https://huggingface.co/jobs/abidlabs/6a2f47b6871c005b5352cd4b

Same runtime/settings as IQ2_M (-ngl 999 -fa 1 -p 512 -n 256 -b 1024 -ub 512 -r 3).

Q3_K_M is slower than IQ2_M for decode, and F16 KV is again best at this context.

Failed: UD-Q4_K_S via hf_hub_download

Job: https://huggingface.co/jobs/abidlabs/6a2f47c9234ca64b60123ea4

The first Q4_K_S attempt was killed with exit 137 (OOMKilled) during hf_hub_download at ~4.85/20.9 GB on a10g-small, likely CPU RAM pressure from the downloader/Xet path rather than GPU OOM. Retrying with plain curl streaming: https://huggingface.co/jobs/abidlabs/6a2f49a1234ca64b60123ecb

Running next

• IQ2_M runtime settings sweep (fa, poll, n_batch): https://huggingface.co/jobs/abidlabs/6a2f49b5234ca64b60123ecf

Completed result: UD-Q4_K_S full GPU offload

Job: https://huggingface.co/jobs/abidlabs/6a2f49a1234ca64b60123ecb

Using plain curl avoided the CPU OOM seen with hf_hub_download; a10g-small has ~15 GiB RAM and ~233 GB disk. Q4_K_S (19.45 GiB in llama.cpp) fully offloaded successfully.

Q4_K_S fits but does not beat IQ2_M/Q3 for throughput; F16 KV remains best.

Completed result: IQ2_M runtime settings sweep

Job: https://huggingface.co/jobs/abidlabs/6a2f49b5234ca64b60123ecf

Swept fa ∈ {1,0}, poll ∈ {0,50,100}, n_batch ∈ {512,1024,2048} for IQ2_M + F16 KV. Best decode result was 118.80 ± 0.57 tok/s at -fa 1 --poll 0 -b 1024 -ub 512.

Observations:

• Decode is insensitive to batch/poll in this range (~118 tok/s throughout).
• Flash attention slightly improves prompt processing and very slightly helps decode (fa=1 ≈ 118.6–118.8 vs fa=0 ≈ 118.0).
• Prompt processing best seen: ~2761 pp tok/s with fa=1, b>=1024.

Running next

• Lower-bit GGUF sweep (UD-IQ1_M, UD-IQ2_XXS, UD-Q2_K_XL) with current best runtime flags: https://huggingface.co/jobs/abidlabs/6a2f4dc9871c005b5352cdac

Completed result: lower-bit GGUF sweep

Job: https://huggingface.co/jobs/abidlabs/6a2f4dc9871c005b5352cdac

Settings: -ngl 999 -fa 1 --poll 0 -ctk f16 -ctv f16 -p 512 -n 256 -b 1024 -ub 512 -r 3.

New best throughput is UD-IQ1_M at 122.57 tok/s, about +3.2% over the prior IQ2_M best. If quality budget allows IQ1_M, this is the current fastest measured config.

Running next

• IQ1_M KV/settings sweep to confirm best flags: https://huggingface.co/jobs/abidlabs/6a2f4f36871c005b5352cdc6

Completed result: IQ1_M KV/settings sweep

Job: https://huggingface.co/jobs/abidlabs/6a2f4f36871c005b5352cdc6

Best single-sequence decode observed: 122.65 ± 0.55 tok/s with UD-IQ1_M, F16 KV, -ngl 999 -fa 1 --poll 0 -b 1024 -ub 512.

Batch/poll/flash sweep again showed decode nearly flat; F16 KV remains best. fa=1 improves prompt processing (2770 vs ~2725 pp tok/s) and gives a small decode edge over fa=0 (122.6 vs ~122.1).

Running next

• Batched/parallel throughput with IQ1_M (npl=1,2,4,8,16): https://huggingface.co/jobs/abidlabs/6a2f5074234ca64b60123f77

Completed result: IQ1_M batched throughput (npl sweep)

Job: https://huggingface.co/jobs/abidlabs/6a2f5074234ca64b60123f77

llama-batched-bench with UD-IQ1_M, F16 KV, -ngl 999 -fa 1 --poll 0 -c 16384 -b 2048 -ub 512 -npp 512 -ntg 256.

For serving throughput, batching/parallel sequences is the biggest win so far: ~385 aggregate decode tok/s at 16 parallel sequences, vs ~122 single-sequence tok/s. Running higher npl sweep next: https://huggingface.co/jobs/abidlabs/6a2f515c871c005b5352cdf0

Completed result: IQ1_M higher batched throughput (npl 16–64)

Job: https://huggingface.co/jobs/abidlabs/6a2f515c871c005b5352cdf0

llama-batched-bench with UD-IQ1_M, F16 KV, -ngl 999 -fa 1 --poll 0 -c 65536 -b 4096 -ub 512 -npp 512 -ntg 256.

New best aggregate decode throughput: 497.44 tok/s at 64 parallel sequences. Gains are tapering but still positive through 64.

Completed result: high concurrency KV quantization sweep (npl 64–160)

Job: https://huggingface.co/jobs/abidlabs/6a2f52a2234ca64b60123fa9

At very high concurrency, the F16 KV cache context allocation failed for npl=64,96,128,160 with -c 196608 (CUDA OOM allocating ~10.05 GiB rs cache). Quantized KV enabled the run.

Settings: UD-IQ1_M, -ngl 999 -fa 1 --poll 0 -c 196608 -b 4096 -ub 512 -npp 512 -ntg 256.

New best aggregate decode throughput: 554.90 tok/s with UD-IQ1_M, q8_0 KV, 160 parallel sequences. For single-sequence latency, F16 KV remains best; for high concurrency, q8/q4 KV are necessary to fit larger contexts and improve total throughput.

Failed/negative: exact-context attempts above npl=160

Jobs:

• High-npl overlarge context attempt: https://huggingface.co/jobs/abidlabs/6a2f559d234ca64b60123fdd
• Exact c=147456 attempt: https://huggingface.co/jobs/abidlabs/6a2f56d1234ca64b60124007

Findings:

• -c 196608 with npl=160,192,224,256 failed context creation even with q8/q4 KV due ~16.08 GiB CUDA allocation for the rs cache.
• -c 147456 with npl=160,192 also failed: q8_0 KV failed allocating 12.06 GiB CUDA buffer; q4_0 then failed compute-buffer allocation (864 MiB) after context setup.
• Practical high-concurrency ceiling on a10g-small for this model/runtime appears near the previous successful npl=160 configuration with -c rounded to 204800 and q8/q4 KV, yielding 554.90 aggregate decode tok/s. Larger max contexts can fail even if the measured N_KV looks only moderately larger.

Failed/abandoned: MTP GGUF speculative CLI smoke

Job: https://huggingface.co/jobs/abidlabs/6a2f55b5234ca64b60123fef

Downloaded unsloth/Qwen3.6-35B-A3B-MTP-GGUF UD-IQ1_M (11 GiB) and attempted llama-cli with --spec-type none then draft-mtp. The first CLI run produced no progress for several minutes after starting, so the job was canceled to avoid wasting GPU time. Need a more controlled MTP-specific smoke test before treating MTP as promising.

Running next

• Test MXFP4_MOE GGUF single-sequence throughput; it is larger (~21.7 GB file) but may offer a quality/throughput tradeoff if it fits: https://huggingface.co/jobs/abidlabs/6a2f5d11871c005b5352ceb8

Negative: npl=192 does not fit even with smaller batch/ubatch

Job: https://huggingface.co/jobs/abidlabs/6a2f5dcc871c005b5352cec8

Tried UD-IQ1_M, q4_0 KV, -c 147456, npl=192, npp=512, ntg=256, sweeping b ∈ {1024,2048,3072} and ub ∈ {256,512}. All failed during context creation with CUDA OOM for compute pp buffers (~844–864 MiB) after allocating model/KV. This reinforces npl=160 as the practical high-concurrency ceiling under current flags on A10G 24GB.

Completed result: exact context for npl=160 recovers F16 KV and improves aggregate throughput

Job: https://huggingface.co/jobs/abidlabs/6a2f5ef6234ca64b601240e1

Previous high-concurrency runs used a larger max context (-c 196608, rounded to n_kv_max=204800), which made F16 KV fail and required q8/q4 KV. Setting context exactly to the required 160 * (512 + 256) = 122880 allowed F16 KV to fit and improved throughput.

Settings: UD-IQ1_M, -ngl 999 -fa 1 --poll 0 -c 122880 -b 4096 -ub 512 -npp 512 -ntg 256 -npl 160.

New best aggregate decode throughput: 563.77 tok/s. Important optimization: for batched serving, keep ctx-size as tight as possible to the served batch (npl * (prompt + generation budget)), otherwise extra KV/rs-cache allocation can force quantized KV or OOM and reduce throughput.

Completed result: MXFP4_MOE single-sequence throughput

Job: https://huggingface.co/jobs/abidlabs/6a2f5d11871c005b5352ceb8

MXFP4_MOE is a large GGUF (~20.21 GiB in llama.cpp / 21 GiB file) and does fit with full GPU offload on A10G 24GB, but it is slower than smaller Unsloth dynamic quants.

Settings: -ngl 999 -fa 1 --poll 0 -p 512 -n 256 -b 1024 -ub 512 -r 3.

Conclusion: MXFP4_MOE may be useful for quality, but not for maximizing tok/s on a10g-small; it is slower than IQ1/IQ2/Q3/Q4_K_S in these decode benchmarks.

Negative/inconclusive: shared-prefix -pps batched-bench

Job: https://huggingface.co/jobs/abidlabs/6a2f621e234ca64b60124126

Tried llama-batched-bench -pps (shared prompt across parallel sequences) with UD-IQ1_M, F16 KV, npl ∈ {64,160,256,384} and tight-ish contexts. Results were not useful:

• npl=64 and npl=160 exited with rc=0 but printed no benchmark rows.
• npl=256 OOMed during rs-cache allocation (~16.08 GiB).
• npl=384 is invalid because n_seq_max must be <= 256.

No throughput improvement recorded from -pps; treat as unsupported/inconclusive in this benchmark path.

Running next

• Test --no-host for single-sequence and npl=160 exact-context batched serving: https://huggingface.co/jobs/abidlabs/6a2f633a871c005b5352cf26

Completed result: --no-host check

Jobs:

• Partial/failed first attempt due bad llama-batched-bench --no-host 0 syntax: https://huggingface.co/jobs/abidlabs/6a2f633a871c005b5352cf26
• Corrected run: https://huggingface.co/jobs/abidlabs/6a2f6472871c005b5352cf38

Single-sequence (UD-IQ1_M, F16 KV, -ngl 999 -fa 1 --poll 0 -b 1024 -ub 512):

• --no-host 1: 122.70 ± 0.49 tok/s decode, 2771.54 ± 15.67 pp tok/s.
• This is only a tiny/noisy improvement over the prior best 122.65 tok/s.

Batched exact-context npl=160:

Conclusion: --no-host is not a meaningful win here; leave it off for batched serving. For single-sequence, it is within noise and can be tried, but the main recommendation remains IQ1_M + F16 KV + full offload + flash attention.

Completed result: CPU threads / priority check

Job: https://huggingface.co/jobs/abidlabs/6a2f6611871c005b5352cf56

Single-sequence UD-IQ1_M, F16 KV, -ngl 999 -fa 1 --poll 0 -b 1024 -ub 512.

Conclusion: CPU thread count 2 vs 4 is noise-level; elevated priority is not permitted in Jobs.

Running next batch 2026-06-15

Launched follow-up attempts to find incremental improvements beyond current bests (single 122.70 tok/s; batched aggregate 563.91 tok/s):

• KV matrix (ctk/ctv mixed F16/BF16/Q8/Q4) for single and npl=160 batched: https://huggingface.co/jobs/abidlabs/6a2fb04b871c005b5352d3c0
• Batched npl=160 batch-size/ubatch sweep: https://huggingface.co/jobs/abidlabs/6a2fb060871c005b5352d3c2
• GPU power/clock lock attempt plus single/batched benchmark: https://huggingface.co/jobs/abidlabs/6a2fb078234ca64b60124903

Completed/partial result: KV cache type matrix

Job: https://huggingface.co/jobs/abidlabs/6a2fb04b871c005b5352d3c0 (canceled after enough signal; mixed-KV batch cases were extremely slow)

Single-sequence UD-IQ1_M, -ngl 999 -fa 1 --poll 0 -b 1024 -ub 512:

Mixed K/V quantization is a major regression for this model/runtime. Symmetric F16/F16 is best; BF16 is close but slightly slower for decode.

Batched npl=160, exact context -c 122880, -b 4096 -ub 512:

• F16/F16: 565.22 aggregate decode tok/s, 2780.32 pp tok/s, 1205.51 total tok/s (new best, though within noise of prior 563.91).
• q8_0/f16: 147.23 aggregate decode tok/s (bad).
• f16/q8_0: 111.68 aggregate decode tok/s (bad).

Conclusion: keep K and V the same type; for both latency and batched throughput, F16/F16 remains the best if it fits.

Completed result: batch / ubatch sweep for npl=160

Job: https://huggingface.co/jobs/abidlabs/6a2fb060871c005b5352d3c2

UD-IQ1_M, F16 KV, exact context -c 122880, -npp 512 -ntg 256 -npl 160.

Conclusion: ubatch=512 is important for prompt throughput; ubatch=128 hurts both pp and decode. batch between 1024 and 4096 has similar decode; b=1024, ub=512 had best total tok/s in this sweep. Previous b=4096,ub=512 remains fine if serving code prefers a larger logical batch.

Completed result: GPU clock/power lock attempt

Job: https://huggingface.co/jobs/abidlabs/6a2fb078234ca64b60124903

nvidia-smi reports A10G already at 300W power limit and max app clocks 1710/6251 MHz. Attempts to set power limit or lock clocks failed with insufficient permissions. Benchmark results were baseline-like: single 122.81 tok/s; batched 565.17 aggregate decode tok/s.

Running next

• Boundary search above npl=160 with exact contexts (npl=168,176,184) and F16/Q4 KV, using smaller logical batch to reduce compute buffer pressure: https://huggingface.co/jobs/abidlabs/6a2fb81a871c005b5352d3f7

Completed result: boundary search above npl=160

Job: https://huggingface.co/jobs/abidlabs/6a2fb81a871c005b5352d3f7

Goal was to see if concurrency above npl=160 could beat the exact-context F16-KV best. Used exact c = npl * (512 + 256), smaller logical batch -b 1024 -ub 512, and tried F16 plus q4_0 KV.

Conclusion: although q4_0 KV allows higher npl, it still underperforms npl=160 with F16 KV (565.22 tok/s). Current best remains npl=160, exact ctx, F16 KV.