Filter by type:
Filter by tags:
qwen3-30b-a1.5b-high-speed
This repo contains the full precision source code, in "safe tensors" format to generate GGUFs, GPTQ, EXL2, AWQ, HQQ and other formats. The source code can also be used directly.
This is a simple "finetune" of the Qwen's "Qwen 30B-A3B" (MOE) model, setting the experts in use from 8 to 4 (out of 128 experts).
This method close to doubles the speed of the model and uses 1.5B (of 30B) parameters instead of 3B (of 30B) parameters. Depending on the application you may want to use the regular model ("30B-A3B"), and use this model for simpler use case(s) although I did not notice any loss of function during routine (but not extensive) testing.
Example generation (Q4KS, CPU) at the bottom of this page using 4 experts / this model.
More complex use cases may benefit from using the normal version.
For reference:
Cpu only operation Q4KS (windows 11) jumps from 12 t/s to 23 t/s.
GPU performance IQ3S jumps from 75 t/s to over 125 t/s. (low to mid level card)
Context size: 32K + 8K for output (40k total)
Repository: localaiLicense: apache-2.0