Model Gallery
Discover and install AI models from our curated collection
Find Your Perfect Model
Filter by Model Type
Browse by Tags
gemma-4-e2b-it-qat-mtp
Repository: localaiLicense: apache-2.0
gemma-4-e2b-it-qat-mtp
Gemma 4 E2B IT QAT (Google DeepMind) paired with its Multi-Token Prediction (MTP) drafter head for speculative decoding on the llama.cpp backend. The Q4_K_XL target carries the full multimodal (text + image) model; the small `mtp-gemma-4-E2B-it` head predicts several tokens ahead which the target verifies in parallel, accelerating generation with no change to output quality. E2B is a MatFormer "effective 2B" elastic variant, well suited to lightweight and on-device deployments. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here. It uses the upstream `gemma4-assistant` architecture registered by llama.cpp PR #23398, so it loads on stock llama.cpp without any patch. License: Apache 2.0 | Authors: Google DeepMind (target/drafter checkpoints), Unsloth (GGUF conversion)
Links
- https://huggingface.co/google/gemma-4-E2B-it
- https://huggingface.co/unsloth/gemma-4-E2B-it-qat-GGUF
- https://github.com/ggml-org/llama.cpp/pull/23398
Tags
gemma-4-e4b-it-qat-mtp
Repository: localaiLicense: apache-2.0
gemma-4-e4b-it-qat-mtp
Gemma 4 E4B IT QAT (Google DeepMind) paired with its Multi-Token Prediction (MTP) drafter head for speculative decoding on the llama.cpp backend. The Q4_K_XL target carries the full multimodal (text + image) model; the small `mtp-gemma-4-E4B-it` head predicts several tokens ahead which the target verifies in parallel, accelerating generation with no change to output quality. E4B is a MatFormer "effective 4B" elastic variant, balancing quality and footprint for on-device and edge deployments. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here. It uses the upstream `gemma4-assistant` architecture registered by llama.cpp PR #23398, so it loads on stock llama.cpp without any patch. License: Apache 2.0 | Authors: Google DeepMind (target/drafter checkpoints), Unsloth (GGUF conversion)
Links
- https://huggingface.co/google/gemma-4-E4B-it
- https://huggingface.co/unsloth/gemma-4-E4B-it-qat-GGUF
- https://github.com/ggml-org/llama.cpp/pull/23398
Tags
gemma-4-12b-it-qat-mtp
Repository: localaiLicense: apache-2.0
gemma-4-12b-it-qat-mtp
Gemma 4 12B IT QAT (Google DeepMind) paired with its Multi-Token Prediction (MTP) drafter head for speculative decoding on the llama.cpp backend. The Q4_K_XL target carries the full multimodal (text + image) model; the small `mtp-gemma-4-12B-it` head predicts several tokens ahead which the target verifies in parallel, accelerating generation with no change to output quality. As a dense model, Gemma 4 12B is among the sizes that benefit most from MTP, with the llama.cpp PR reporting well over 1.4x decode speedup. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here. It uses the upstream `gemma4-assistant` architecture registered by llama.cpp PR #23398, so it loads on stock llama.cpp without any patch. License: Apache 2.0 | Authors: Google DeepMind (target/drafter checkpoints), Unsloth (GGUF conversion)
Links
- https://huggingface.co/google/gemma-4-12B-it
- https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF
- https://github.com/ggml-org/llama.cpp/pull/23398
Tags
gemma-4-31b-it-qat-mtp
Repository: localaiLicense: apache-2.0
gemma-4-31b-it-qat-mtp
Gemma 4 31B IT QAT (Google DeepMind), the largest dense model in the family, paired with its Multi-Token Prediction (MTP) drafter head for speculative decoding on the llama.cpp backend. The Q4_K_XL target carries the full multimodal (text + image) model; the small `mtp-gemma-4-31B-it` head predicts several tokens ahead which the target verifies in parallel, accelerating generation with no change to output quality. Dense models like 31B are the sizes that benefit most from MTP. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here. It uses the upstream `gemma4-assistant` architecture registered by llama.cpp PR #23398, so it loads on stock llama.cpp without any patch. License: Apache 2.0 | Authors: Google DeepMind (target/drafter checkpoints), Unsloth (GGUF conversion)
Links
- https://huggingface.co/google/gemma-4-31B-it
- https://huggingface.co/unsloth/gemma-4-31B-it-qat-GGUF
- https://github.com/ggml-org/llama.cpp/pull/23398
Tags
alamios_mistral-small-3.1-draft-0.5b
Repository: localaiLicense: apache-2.0
alamios_mistral-small-3.1-draft-0.5b
This model is meant to be used as draft model for speculative decoding with mistralai/Mistral-Small-3.1-24B-Instruct-2503 or mistralai/Mistral-Small-24B-Instruct-2501 Data info The data are Mistral's outputs and includes all kind of tasks from various datasets in English, French, German, Spanish, Italian and Portuguese. It has been trained for 2 epochs on 20k unique examples, for a total of 12 million tokens per epoch.
Links
- https://huggingface.co/alamios/Mistral-Small-3.1-DRAFT-0.5B
- https://huggingface.co/bartowski/alamios_Mistral-Small-3.1-DRAFT-0.5B-GGUF
Tags
gemma-4-e2b-it:sglang-mtp
Repository: localaiLicense: gemma
gemma-4-e2b-it:sglang-mtp
Google Gemma 4 E2B-IT served by SGLang with Multi-Token Prediction (MTP) speculative decoding. The companion drafter google/gemma-4-E2B-it-assistant lets the target accept several tokens per step. Flags are a 1:1 transcription of the SGLang cookbook's MTP command (NEXTN algorithm, num_steps=5, num_draft_tokens=6, eagle_topk=1, mem_fraction_static=0.85). The E2B variant has 5B total / 2B effective parameters and targets the smaller end of consumer GPUs.
Links
- https://huggingface.co/google/gemma-4-E2B-it
- https://huggingface.co/google/gemma-4-E2B-it-assistant
- https://docs.sglang.io/cookbook/autoregressive/Google/Gemma4
Tags
gemma-4-e4b-it:sglang-mtp
Repository: localaiLicense: gemma
gemma-4-e4b-it:sglang-mtp
Google Gemma 4 E4B-IT served by SGLang with Multi-Token Prediction (MTP) speculative decoding. The companion drafter google/gemma-4-E4B-it-assistant lets the target accept several tokens per step. Flags are a 1:1 transcription of the SGLang cookbook's MTP command (NEXTN algorithm, num_steps=5, num_draft_tokens=6, eagle_topk=1, mem_fraction_static=0.85). The E4B variant has 8B total / 4B effective parameters — the natural pick for consumer GPUs in the 16–24 GB range.
Links
- https://huggingface.co/google/gemma-4-E4B-it
- https://huggingface.co/google/gemma-4-E4B-it-assistant
- https://docs.sglang.io/cookbook/autoregressive/Google/Gemma4
Tags
mimo-7b-mtp:sglang
Repository: localaiLicense: mit
mimo-7b-mtp:sglang
Xiaomi MiMo-7B-RL served by SGLang with built-in Multi-Token Prediction (MTP) heads (no separate drafter needed) plus online fp8 weight quantization to fit on a 16 GB consumer GPU. ~90% acceptance per the model card. Verified end-to-end at ~88 tok/s on an RTX 5070 Ti (16 GB). Note: mem_fraction_static is dropped to 0.7 (vs sglang's 0.85 default) because the MTP draft worker's vocab embedding is loaded unquantised (~1.2 GiB) and OOMs the static reservation otherwise.
Links
Tags
deepseek-v4-flash-q2-mtp
Repository: localai
deepseek-v4-flash-q2-mtp
DeepSeek V4 Flash (IQ2XXS GGUF, ~81 GB) paired with the optional MTP speculative-decoding weights (~3.5 GB) for a slight speedup. Only loadable via the ds4 backend; requires >=128 GB RAM. MTP helps only with greedy decoding (temperature 0), so the override pins temperature to 0. Metal (Darwin) or CUDA (Linux). See https://github.com/antirez/ds4 for details.
Links
Tags