## Overview
llama.cpp v0.7.0 adds the LiliCorr speculative decoding technique and MTP for GLM5-Next, a new GPU cache for MoE experts kept in host memory (`--moe-cache-mib`, up to ~2.2x decode speedup when experts are offloaded), and a large batch of new models: K2 Horizon (dense and MoVA), Prism Bonsai 2 27B, EmbeddingGemma-2 (text+vision+audio), the LiquidAI d1/d1-omni and vLLM decision 2/d3 decision models, pplx-decider, MiniCPM-V 4.7, Nemotron Super 3.5 VL / Nemotron Bolt and maion-coder. The server gets a refactored chat session API (`common_chat_session`, part-based formatted output), much better slot and prompt-cache restore behavior, decision-model support for the new architectures and a new default port (9931), the web UI gains a full models manager, and ggml is updated to v0.27.0.
### Highlights
- New GPU cache for host-memory MoE experts (`--moe-cache-mib`): experts run on the GPU from an LRU cache, up to 2.2x faster decode [#29887](https://github.com/ggml-org/llama.cpp/pull/29887)
- The expert cache is now shared across multiple GPUs [#30112](https://github.com/ggml-org/llama.cpp/pull/30112)
- LiliCorr speculative decoding: candidate selection with a small draft transformer on top of DFlash [#30506](https://github.com/ggml-org/llama.cpp/pull/30506)
- GLM5-Next: MTP speculative decoding with an optimized sparse attention path [#29928](https://github.com/ggml-org/llama.cpp/pull/29928)
- K2 Horizon support in both dense and MoVA variants: conversion, tokenizer, chat template, `-sm tensor` splitting [#29535](https://github.com/ggml-org/llama.cpp/pull/29535)
- Prism Bonsai 2 27B ternary model support [#29600](https://github.com/ggml-org/llama.cpp/pull/29600)
- EmbeddingGemma-2: text + vision + audio embedding model [#30054](https://github.com/ggml-org/llama.cpp/pull/30054)
- New decision models: LiquidAI d1-3B, d1-omni-600M, vLLM decision 2, vLLM d3 and pplx-decider [#30110](https://github.com/ggml-org/llama.cpp/pull/30110) [#30114](https://github.com/ggml-org/llama.cpp/pull/30114) [#30378](https://github.com/ggml-org/llama.cpp/pull/30378) [#30383](https://github.com/ggml-org/llama.cpp/pull/30383) [#30044](https://github.com/ggml-org/llama.cpp/pull/30044)
- PLaMo-3 tokenizer implements reference pre-segmentation for exact tokenization parity [#30045](https://github.com/ggml-org/llama.cpp/pull/30045)
- RPC server supports `-sm tensor` (tensor-parallel RPC, RPC protocol v8) [#26610](https://github.com/ggml-org/llama.cpp/pull/26610)
- Server now listens on port 9931 by default instead of 8080 [#30159](https://github.com/ggml-org/llama.cpp/pull/30159)
### API changes
- `include/llama.h`: new `moe_cache_size` context param for the host-expert GPU cache [#29887](https://github.com/ggml-org/llama.cpp/pull/29887)
- `include/llama.h`: new `LLAMA_VOCAB_TYPE_PLAMO3` vocab type [#30045](https://github.com/ggml-org/llama.cpp/pull/30045)
- `include/llama.h`: `llama_perf_context_data` reports graph reallocations during inference [#30329](https://github.com/ggml-org/llama.cpp/pull/30329)
- `include/llama.h`: `embeddings` now skips the LM head when a pooling type other than NONE is used [#30445](https://github.com/ggml-org/llama.cpp/pull/30445)
- `include/llama.h`: `llama_batch_ext_set_embd_state` returns false when the context has no state embedding, M-RoPE order documented [#30506](https://github.com/ggml-org/llama.cpp/pull/30506)
- `tools/server`: chat layer refactored around `common_chat_session` with `feed()` / `finish()` [#30210](https://github.com/ggml-org/llama.cpp/pull/30210)
- `tools/server`: formatted chat output is now an array of parts instead of a single string [#30389](https://github.com/ggml-org/llama.cpp/pull/30389)
- `tools/server`: default listening port changed to 9931 (Docker images keep 8080 inside the container) [#30159](https://github.com/ggml-org/llama.cpp/pull/30159)
### New models
- K2 Horizon: dense and MoVA variants with conversion, tokenizer, chat template and `-sm tensor` support [#29535](https://github.com/ggml-org/llama.cpp/pull/29535)
- Prism Bonsai 2 27B (ternary) [#29600](https://github.com/ggml-org/llama.cpp/pull/29600)
- EmbeddingGemma-2: text + vision + audio embeddings [#30054](https://github.com/ggml-org/llama.cpp/pull/30054)
- LiquidAI d1-3B and d1-omni-600M decision models [#30110](https://github.com/ggml-org/llama.cpp/pull/30110) [#30114](https://github.com/ggml-org/llama.cpp/pull/30114)
- vLLM decision 2 and vLLM d3 models [#30378](https://github.com/ggml-org/llama.cpp/pull/30378) [#30383](https://github.com/ggml-org/llama.cpp/pull/30383)
- pplx-decider decision model [#30044](https://github.com/ggml-org/llama.cpp/pull/30044)
- MiniCPM-V 4.7 multimodal model [#29416](https://github.com/ggml-org/llama.cpp/pull/29416)
- Nemotron Super 3.5 VL and Nemotron Bolt 74B A9B [#30520](https://github.com/ggml-org/llama.cpp/pull/30520)
- maion-coder architecture [#29778](https://github.com/ggml-org/llama.cpp/pull/29778)
- Mellum: MoE gating parameters read from GGUF and unified RoPE handling [#30578](https://github.com/ggml-org/llama.cpp/pull/30578)
- LiliCorr draft model conversion and graph [#30506](https://github.com/ggml-org/llama.cpp/pull/30506)
- Rerankers: `classifier_activation` support, exact GELU for ModernBERT encoders [#29692](https://github.com/ggml-org/llama.cpp/pull/29692) [#30108](https://github.com/ggml-org/llama.cpp/pull/30108)
### Core changes
- Speculative decoding: MTP for GLM5-Next, LiliCorr drafts, correct mtmd handling in MTP graphs [#29928](https://github.com/ggml-org/llama.cpp/pull/29928) [#30506](https://github.com/ggml-org/llama.cpp/pull/30506) [#30257](https://github.com/ggml-org/llama.cpp/pull/30257)
- Speculative decoding: shared nextn tensor flags with consolidated row cropping and sched re-reserve on flag change [#30097](https://github.com/ggml-org/llama.cpp/pull/30097) [#30017](https://github.com/ggml-org/llama.cpp/pull/30017) [#30020](https://github.com/ggml-org/llama.cpp/pull/30020)
- Speculative decoding: fixed draft sampling and rejection handling, DFlash output head sharing [#30095](https://github.com/ggml-org/llama.cpp/pull/30095) [#30111](https://github.com/ggml-org/llama.cpp/pull/30111)
- New `--spec-draft-n-ubatch` to size the draft context batch separately and save VRAM [#30459](https://github.com/ggml-org/llama.cpp/pull/30459)
- MoE expert cache: host-memory experts served from a device LRU cache, supported over multiple GPUs [#29887](https://github.com/ggml-org/llama.cpp/pull/29887) [#30112](https://github.com/ggml-org/llama.cpp/pull/30112)
- Embedding models: skip the logits buffer and the LM head when pooling is enabled (up to 10x less VRAM) [#30255](https://github.com/ggml-org/llama.cpp/pull/30255) [#30445](https://github.com/ggml-org/llama.cpp/pull/30445)
- Embedding graphs: reordered `get_rows`, dropped the redundant `t_embd` expand and leftover `ggml_cont` calls [#30160](https://github.com/ggml-org/llama.cpp/pull/30160) [#30461](https://github.com/ggml-org/llama.cpp/pull/30461) [#30198](https://github.com/ggml-org/llama.cpp/pull/30198)
- No-cache attention: padded KV dimension and KQ bias, avoided graph reallocations [#30416](https://github.com/ggml-org/llama.cpp/pull/30416) [#30446](https://github.com/ggml-org/llama.cpp/pull/30446) [#30439](https://github.com/ggml-org/llama.cpp/pull/30439)
- k-pool models: selection built in the causal mask layout, fixed a scatter data race on shared sequences [#30480](https://github.com/ggml-org/llama.cpp/pull/30480) [#29994](https://github.com/ggml-org/llama.cpp/pull/29994)
- GLM5-Next sparse attention: removed the gather path [#30042](https://github.com/ggml-org/llama.cpp/pull/30042)
- KV cache: honor `LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY` so draft checkpoints stop carrying the full KV [#30514](https://github.com/ggml-org/llama.cpp/pull/30514)
- Prefetching: merge ranges before issuing system prefetch calls, never map foreign ranges into a backend buffer [#30532](https://github.com/ggml-org/llama.cpp/pull/30532) [#30405](https://github.com/ggml-org/llama.cpp/pull/30405)
- Sampling: greedy selection for eligible temperature-zero chains, backend sampling graph kept static across ubatches [#29797](https://github.com/ggml-org/llama.cpp/pull/29797) [#30223](https://github.com/ggml-org/llama.cpp/pull/30223)
- Tokenizers: PLaMo-3 pre-segmentation and FiM tokens, opt-in NFC normalization for Unigram, strict UTF-8 validation [#30045](https://github.com/ggml-org/llama.cpp/pull/30045) [#30090](https://github.com/ggml-org/llama.cpp/pull/30090) [#30390](https://github.com/ggml-org/llama.cpp/pull/30390) [#30485](https://github.com/ggml-org/llama.cpp/pull/30485)
- Jinja: fixed loop variable scoping, Python-compatible `tojson` rendering, faster context copy [#30533](https://github.com/ggml-org/llama.cpp/pull/30533) [#30483](https://github.com/ggml-org/llama.cpp/pull/30483) [#30362](https://github.com/ggml-org/llama.cpp/pull/30362) [#30562](https://github.com/ggml-org/llama.cpp/pull/30562)
- Model split: distribute layers evenly when no device reports free memory [#29677](https://github.com/ggml-org/llama.cpp/pull/29677)
### Multi-modality changes
- Added `mtmd_build_inp_attn_mask` for building non-causal multimodal attention masks [#30259](https://github.com/ggml-org/llama.cpp/pull/30259)
- Added cohere2 vision encoder support [#30062](https://github.com/ggml-org/llama.cpp/pull/30062)
- MTP drafts now handle mtmd inputs, Qwen3.8 Flash Next accepts image embeddings [#30257](https://github.com/ggml-org/llama.cpp/pull/30257) [#30397](https://github.com/ggml-org/llama.cpp/pull/30397)
- New multimodal architectures: MiniCPM-V 4.7, Nemotron Super 3.5 VL, EmbeddingGemma-2 (vision + audio) [#29416](https://github.com/ggml-org/llama.cpp/pull/29416) [#30520](https://github.com/ggml-org/llama.cpp/pull/30520) [#30054](https://github.com/ggml-org/llama.cpp/pull/30054)
### Server changes
- `/v1/systemone` extended to the new decision models: d1-3B, d1-omni, vLLM decision 2/d3, pplx-decider [#30110](https://github.com/ggml-org/llama.cpp/pull/30110) [#30114](https://github.com/ggml-org/llama.cpp/pull/30114) [#30378](https://github.com/ggml-org/llama.cpp/pull/30378) [#30383](https://github.com/ggml-org/llama.cpp/pull/30383) [#30044](https://github.com/ggml-org/llama.cpp/pull/30044)
- Chat refactor: session-based parsing and part-based formatted output [#30210](https://github.com/ggml-org/llama.cpp/pull/30210) [#30389](https://github.com/ggml-org/llama.cpp/pull/30389)
- Generated text and tokens are now accumulated as parser input, preparing token-aware parsing [#29876](https://github.com/ggml-org/llama.cpp/pull/29876)
- Prompt cache: only restorable slot states are saved, so rerank and embedding requests no longer fill the cache [#30498](https://github.com/ggml-org/llama.cpp/pull/30498)
- Prompt cache: child tasks prefer empty slots and preserve the state they would overwrite [#30567](https://github.com/ggml-org/llama.cpp/pull/30567)
- A pinned busy slot is left untouched until the request can run on it [#30295](https://github.com/ggml-org/llama.cpp/pull/30295)
- Slot save/restore now keeps context checkpoints and the draft model state [#26004](https://github.com/ggml-org/llama.cpp/pull/26004) [#30574](https://github.com/ggml-org/llama.cpp/pull/30574)
- Fixed an off-by-two in the SWA reuse threshold that discarded restored states [#30580](https://github.com/ggml-org/llama.cpp/pull/30580)
- Fixed a stack-overflow crash when escaping special tokens with `std::regex` [#30243](https://github.com/ggml-org/llama.cpp/pull/30243)
- `LLAMA_ARG_SLOT_SAVE_PATH` environment variable for `--slot-save-path` [#30272](https://github.com/ggml-org/llama.cpp/pull/30272)
- New per-component server tests; vendored cpp-httplib updated to 0.61.0 [#30312](https://github.com/ggml-org/llama.cpp/pull/30312) [#30494](https://github.com/ggml-org/llama.cpp/pull/30494) [#30134](https://github.com/ggml-org/llama.cpp/pull/30134) [#30081](https://github.com/ggml-org/llama.cpp/pull/30081)
### UI changes
- New Models Manager: one table of all served models with expandable quants, search, filters, sorting and favorites [#29583](https://github.com/ggml-org/llama.cpp/pull/29583)
- Row actions: load/unload, delete from disk, hide/unhide, resumable downloads with progress [#29583](https://github.com/ggml-org/llama.cpp/pull/29583)
- Model configuration side drawer with model info, chat template and "start a new chat" / "use in this chat" actions [#29583](https://github.com/ggml-org/llama.cpp/pull/29583)
- Models manager follow-ups: GGUF training context, favorite UX, filter and sort fixes [#30228](https://github.com/ggml-org/llama.cpp/pull/30228)
- Fixed `ui_settings` not being applied on the first visit in router mode [#29668](https://github.com/ggml-org/llama.cpp/pull/29668)
- Fixed text attachment preview overflow for large pasted text [#29898](https://github.com/ggml-org/llama.cpp/pull/29898)
- Fixed the default agentic turns limit falling back to 10 instead of 100 [#30582](https://github.com/ggml-org/llama.cpp/pull/30582)
### ggml changes
- ggml updated to [v0.27.0](https://github.com/ggml-org/ggml/releases/tag/v0.27.0): the backend scheduler gains user callbacks for split weight copying and compute-buffer reallocation, and the RPC protocol is bumped to v8 for tensor-parallel RPC servers. Backends gained chunked gated-delta-net kernels on CUDA/HIP, KleidiAI SME2 k-quant matmuls and AVX2 Q2_0 dot products on CPU, L2-aware coopmat2 scheduling and bf16 activations on Vulkan, reordered-layout MoE kernels (MXFP4, IQ3, Q5_K, grouped XMX GEMM) on SYCL, dp4a MoE GEMMs on OpenCL, tensor-API and wider-head-dim flash attention on Metal, and a DMA/HVX copy overhaul with pooling and 3D im2col ops on Hexagon. Correctness work focused on non-contiguous and strided views, grid and workgroup size limits, out-of-bounds accesses and well-defined `mul_mat_id` behavior for duplicate expert ids; symbol visibility now defaults to hidden
## Assets
**Nightly build:** [b11669](https://github.com/ggml-org/llama.cpp/releases/tag/b11669)
## More info
- [Releases and versioning of `ggml-org` projects](https://github.com/ggml-org/ggml/discussions/1579)
## Changelog since [v0.6.0](https://github.com/ggml-org/llama.cpp/releases/tag/v0.6.0)
620d2225c llama.cpp : bump version to 0.7.0 (#30603)
cd0a5b6a1 models : add Nemotron Super 3.5 VL and Nemotron Bolt 74B A9B support (#30520)
a11b6530b scripts : add ci-status.sh to list active CI runs (#30586)
61ad80bf9 server: fix the off-by-two in the SWA reuse threshold (#30580)
a6b91d8c5 model : Mellum - read MoE gating params from GGUF, unify RoPE (#30578)
ea736c1d5 server: save the draft state in the slot file (#30574)
b0d239164 server: prefer empty slots and keep states overwritten by child tasks (#30567)
c0cb53ac4 sync : ggml
4ed1c8507 ggml : bump version to 0.27.0
711780ffc ci : bump ty to 0.0.88 (#30587)
fd55cb98f fix(ui): take the agentic turns default from DEFAULT_AGENTIC_CONFIG (#30582)
3e4dfa8f3 models : add maion-coder (#29778)
08698c81e jinja : improve context copy constructor performance (#30562)
82e1e2fd8 jinja : write nan and inf in tojson as json.dumps does (#30483)
47f5a64ae vulkan : fix concurrent first device init (#29766)
e36df6047 sycl : replace oneAPI package by DLE+oneDNN package (#30131)
9166ceaf1 opencl: fix OOB tail access for adreno noshuffle gemv kernels (#30421)
2130ca847 jinja : fix loop body variable scoping (#30533)
23f83e056 ggml-cpu : optimize pad_f32 (#30526)
2ce20d22f unicode : reject overlong UTF-8 and out-of-range codepoints (#30485)
68ac76572 llama : split llama_prefetch() into merge and prefetch steps (#30532)
77c1f3992 opencl: apply flash attn softcap before mask (#30546)
733f57a99 vulkan : copy strided f16 KV once on AMD so reads spread across memory channels (#27703)
5a638aedb cpu: add F16 SWIGLU_OAI reference implementation and validation (#29200)
6e66cb5d4 metal : add BF16 variant of the tensor API flash attention kernel (#30491)
e467e5b5e kv-cache: honor PARTIAL_ONLY in the plain KV cache (#30514)
1ddc26255 graph: keep the graph non empty without embeddings (#30522)
2639a3b4e server: save only restorable slot states to the prompt cache (#30498)
843e7dcc1 spec : add lilicorr support (#30506)
3e390af18 vulkan: Use L2-friendly workgroup scheduling for cm2 mulmat (#30215)
a0e6dc738 opencl: support CLAMP on non-contiguous views (#30186)
23deec025 devops: incorrect COPY instruction (#30529)
683bbc0dc gguf-py : detect nfc normalization (#30540)
529ac893e sycl: IQ3 code reorder (#29107)
685f70894 ggml-cpu : keep the FP16->FP32 lookup table on x86 with F16C (#30342) (#30343)
c114bd4c5 docs: update IBM zDNN backend docs (#30527)
5cdadfc32 ggml-cpu : size set_rows conversion buffer for all threads (#30454)
62df1815b cuda, hip: GDN chunked kernel (#29353)
37d6d653e skill: Update `code-review` skill for UI development (#30509)
6868a52cb ggml: fix conv_1d result layout for batched input (#28738)
3fb47677e common : add --spec-draft-n-ubatch (#30459)
318a471a6 docs : clarify installation recommendation for Intel oneAPI Toolkit (#30407)
8c2134a87 ggml : add note about mul_mat_id + disable mul_mat_id_dup tests (#30501)
df18bee11 model: split layers evenly when no device reports free memory (#29677)
3eb101360 vendor : update cpp-httplib to 0.61.0 (#30494)
af0579c78 ggml : improve RPC tensor transfer logs (#30481)
62bacfa01 cuda : use SIMT implicit GEMM conv2d on AMD without WMMA (#30325)
77390baa1 jinja : write floats in tojson as Python does (#30362)
c5a997624 opencl: use buf+offset instead of subbuffer for adreno moe expert sort (#30418)
21a19cd3b opencl: fix q8_0 dp4a mul_mat_id and mark PREC_F32 mul_mat_id unsupported for adreno (#30365)
9cf967a04 models: drop the t_embd expand of the embedding graphs (#30461)
4dd6dd4b9 HIP: use 64-row MMQ tiles for some quant types on RDNA3.5 (#28195)
14310190c vulkan: fix compile error (#30495)
e7268bf6a llama : build the k-pool selection in the causal mask layout (#30480)
cdce056a3 vulkan: Add workaround for device free memory > total memory on Intel Windows (#29835)
d524bb660 vulkan: Define the shader count for Intel Arc Pro B65 + B70 (#30431)
7ec420bc0 vulkan: support mulmat bf16 activations (#29069)
10da8c463 unicode : add opt-in NFC normalization for Unigram tokenizers (#30390)
eca5458ce convert : export YaRN truncation config (#30288)
091b6a7fc Enable OpenSSL in FP32/FP16 SYCL builds to restore HF HTTPS downloads (#30330)
fcac19079 webgpu: (Fix CI) fix mul_mat_id with duplicate expert ids (#30451)
04f7226a2 server : escape the special tokens without std::regex (#30243)
ff026f8c0 chat: replace output string with parts (#30389)
1b70350c9 ggml : simplify backend search path error handling (#28367)
d71e324a4 docs : update zDNN supported ops (#30472)
45451c366 kleidiai: add SME2 Q4_K and Q6_K matmul support (#29219)
77c4540ce vulkan: Optimize pool_2d when the pooling window is large (#30119)
ecdd29e8b meta: recreate all subgraphs when the arena is reallocated (#30453)
ec0bfac17 hexagon: add configurable HTP DCVS/HMX power mode (#30423)
ace20f679 graph: pad the KQ bias of the no-cache attention (#30446)
b9f3599fb llama : do not map foreign ranges into a backend buffer (#30405)
f441d2d9d model: load YaRN beta metadata for laguna models (#30331)
cb55ffe6e graph: skip the LM head for pooled embeddings (#30445)
a17e94d22 tests : consolidate context state tests into test-llama-context (#30023)
b476d9a76 llama : fix log message (#30452)
e0bcaccbe vulkan: allow multiple devices to access gpu-visible host memory (#29741)
0a731cced ggml-cpu : add AVX2 dot product for Q2_0 (#30447)
f851e244d cuda : add GGML_CUDA_GRAPH_DEBUG and read env vars once (#30327)
bfe65e1df cuda : fix compiler warnings in mmq-load-tiles.cuh (#30443)
411d54851 vulkan : split matmul dispatch when it would exceed maxComputeWorkGroupCount (#29533)
296538ade spec : remove tokens assert from dsv4 graph_mtp (#30444)
7f51953d7 model : allow Qwen3.8 Flash Next to accept image embds for MTP (#30397)
882d67166 cuda : fix warnings from dead breaks (#30441)
30f7df1a8 cuda : warnings cleanup (#30420)
9d7d50498 convert : fix LoRA conversion with --outtype auto (#30433)
ed019cd8b llama : skip logits buffer allocation when graph produces no logits (#30255)
988015788 graph : avoid reallocations in the "no cache" path (#30439)
af50f2f24 ggml-zdnn : introduce Q8_0 mat mul (#29836)
071b553fb models : remove redundant ggml_cont calls now supported by all unary kernels (#30198)
4413a1924 hexagon: extend scalar DMA broadcasting support for binary ops (#30245)
8f574b0ad ci : build OpenVINO releases with GGML_NATIVE=OFF and CPU backend variants (#30324)
a49f58dcc ci: fix OpenVINO download links (#30401)
cbe28bb3c model: fix vllm d3 multi seq case (#30408)
358a420ec graph: pad the KV dimension of the no-cache attention to 256 (#30416)
48e985756 metal: fix metal tensor kernel_mul_mm error for large tensor (#28748)
8bb3cf0d2 hexagon: optimize CPY/CONCAT — round 3, short-rows edition (#30341)
0f701c882 vulkan: pack FMA for f16 accumulation, no dot product op (#29877)
efe3626b1 model: support vllm d3 models (#30383)
097cf33d9 model: support vllm decision 2 model (#30378)
ccdbe2905 tests : link each test against only the libraries it uses (#30299)
073b74034 metal : fix mul_mat_id with duplicate ids (#30373)
77757a346 examples : fix draft sampling and rejection handling (#30095)
972b5fcc1 llama : report compute graph reallocations during inference (#30329)
2f160ba59 ui: fix text attachment preview overflow (#29898)
f9a95aeb2 sycl: guard DMMV reads at row tails (#29338)
08ffba21e vulkan: use ternary expressions to avoid switch lowering for CM2 quant block size (#30164)
a05e58c98 ggml: Change default visibility of symbols to hidden (#30014)
b2f036c0f vulkan: fix buffer logging on 32-bit platforms after source split (#30326)
e937b17a2 hexagon: optimize q2_k bpw and activation (#30035)
b13a93a3d hexagon: add IM2COL_3D (#30305)
f2cc7282c opencl: improve fa, allow dk512 for gemma-4, improve dk64 (#30266)
c83f3058b opencl: fix image limit for q4_k dense bin kernels, q4_0 bcast and rms_norm (#30310)
5c8854fd8 rpc : turn GGML_RPC_DEBUG into a verbosity level and add logs (#29544)
a9d27ac69 model : support for Prism Bonsai 2 27B (#29600)
1cf776776 cpu: fix AVX2 CPUs not using F16C when available (#30270)
966baae76 vulkan: handle mul_mat_id duplicates in the prepass rather than looping (#29998)
62a6f74ba server: add component tests (#30312)
23b0202a1 server: leave a busy slot untouched when a request pins it (#30295)
69f201a20 model : support MiniCPM-V 4.7 (#29416)
abee0c847 cmake(s390x): disable z17 target for unsupported compilers (#30297)
aa94f2086 args: add LLAMA_ARG_SLOT_SAVE_PATH env for --slot-save-path (#30272)
781dbc5ac spec: properly handle mtmd input for mtp (#30257)
0fd868cbc mtmd: add build_inp_attn_mask (#30259)
1623d8ce4 cuda: always use MMVQ for MUL_MAT_ID on sm_60 (#27828)
1bb2b9fcb CUDA/HIP: fix race in flash_attn_ext_f16_process_tile when nbatch_combine != DKQ/2 (#30103)
2bbca8f20 ggml-cpu: vectorize fp32 to fp16 conversion (#30157)
404f557b5 vulkan: use 4 rows for NVIDIA MUL_MAT_ID MMVQ (#29274)
b797c82c7 ggml: fix s390x all cpu build (#30140)
f2918cabb opencl: add bin kernels kernel_gemm_moe_q4_k_q8_1_dp4a_bin, kernel_gemm_moe_q6_k_q8_1_dp4a_bin (#30187)
1e6f04a75 sycl : accelerate MXFP4 MoE with arithmetic decoding and weight reordering (#29809)
10a60cf30 vendor: apply deep nested json patch from upstream (#30253)
79e2e74eb CUDA: fix round issue, under MSVC the CPU and GPU agree (#30229)
8e2d31e0e graph : reorder get_rows for embeddings (#30160)
baef3ed9a ui: Models Manager Follow-up Improvements (#30228)
f39148a95 llama-bench: respect -fitc if bigger than required benchmark size (#28331)
50e3e3e48 CUDA: Remove redundant CUDA copies after SSM_SCAN (#29807)
64df9183f opencl: fix kernel compilation for a6x GPUs (#30176)
6184e92c5 model : use exact GELU for ModernBERT encoders (#30108)
8b5436102 chat : refactor API (#30210)
a518119d3 llama: keep the backend sampling graph static across ubatches (#30223)
8ae386707 ggml: fix OOB write in ggml_acc with negative offset (#30135)
e60eff95f meta : handle host views (#30217)
ba6439a6b webgpu: use 2D workgroup dispatch for all the ops which use 1D dispatch (e.g., rms_norm) (#30219)
5e4878e97 vulkan: fix rms_norm workgroup count overflow (#30145)
609290be6 convert : support compressed-tensor mixed-precision NVFP4 checkpoint (#28636)
86a283532 musa: drop mp_21 from the default architectures (#30203)
e94acad85 ci: fix Models Backend webgpu by supporting GGML_OP_DUP (#30216)
5e1d74043 ci : set run-name for publish release (#30214)
b42b7e6d3 server: use port 9931 by default (#30159)
b013e56a7 ui : add the models manager (#29583)
2c564f43d ci: fix HIP quality check by ignoring the 320/256 FA kernel spill (#30212)
1f8fa5231 metal: add the 128/96 flash attention kernels, accept every compiled pair (#30209)
8a1a9b512 CUDA: pass src1 precision to host MMQ config helpers (#30168)
d4d82d67f hexagon: fix IM2COL patch-embed DMA ring overflow (#30189)
3d65c90d0 sycl : Q5_K reorder-layout MMVQ and fused GLU (#29375)
de7fa0a3c Musa FWHT fix (#30167)
71ad0590f CUDA: improve top-k algorithm selection (#28713)
a11f57ba9 model : fix DFlash output head sharing (#30111)
fc9ce6b9d CUDA: fix MMQ out-of-bounds reads (#29953)
c35b66744 CUDA : looped PAD kernel for more than 65535 rows or slices (#30147)
1167d3f42 CUDA: fix CCCL version guard breaking on major version rollover (#29453)
4f92965a7 ui: apply ui_settings on first visit in router mode (#29668)
c811cb8f0 llama: support MoE cache over multiple GPUs (#30112)
033df86b6 server : preserve context checkpoints across slot save/restore (#26004)
ff5888f99 vulkan : fix TOP_K for +inf/NaN inputs and k = 1 on negative values (#30107)
dac308739 vulkan: extend sparse FA support to coopmat2 (#30003)
03aa006ac vendor : update cpp-httplib to 0.60.1 (#30134)
08246a28f cuda : support arbitrary striding for unary ops on f16, f32, and bf16 (#29781)
46baf1f1f sycl: FWHT optimizations (#29605)
097f5b533 hex-mmadd: do not assume aligned read/write when bias-add is fused (#30133)
d88801604 cuda : Use byte strides for roll to allow non-contiguous ROLL operations (#29547)
fda186661 CUDA: fix norm family kernels when ne[2]/ne[3] exceed grid dim limits (#28175)
ff30363a0 sycl: fuse the delta-net alpha gate (add + unary + mul) (#29687)
000bee54a sycl: stage bulk uploads (model loading) through a pinned ring buffer (#29608)
37ac63456 model : support classifier_activation for rerankers (#29692)
24e41838e ggml-cuda: assign four GDN state columns per warp (#30087)
847f447c3 sycl: add grouped MoE XMX GEMM (#29245)
75118a3a5 convert : support Qwen3.5 embedding models (#27920)
9b4ed0ca5 sycl: remove duplicate block-size defines from op headers (#29507)
9c2e0e491 hexagon: enable alloc_buffer_n (#30126)
06cad0b9e hexagon: Q6_K weight dequant speedup (#30121)
a657f7e98 model : add LiquidAI/d1-omni-600M decision model (#30114)
aa5e0092f hexagon: support tiled Q4_K and Q6_K GET_ROWS (#30115)
70815103c hexagon: improved GELU accuracy (#30104)
bd4eeaa04 chat: fix jinja parser for TranslateGemma (#30096)
5de733437 chat : name tool and argument parser rules by index (#30088)
88dcc460d model : add LiquidAI/d1-3B decision model (#30110)
b86d2f075 cuda: FWHT kernels for block widths above 512 (#29100)
50a6c5cf7 mtmd: add cohere2 vision support (#30062)
d6cf9acb2 llama : add a GPU cache for MoE experts kept in host memory (#29887)
42c787e8c cuda: update uncoalesced memory reads in pool2d (#29425)
18b5f8b18 server : accumulate generated text and tokens as parse input (#29876)
448147d42 llama: share the nextn tensor flags between models (#30097)
988190680 metal : few-row MMA mat-mul for the remaining src0 types (#30065)
7e8324f5f metal : fix MUL_MAT+ADD fusion when the residual is itself a MUL_MAT (#30100)
b9acf138a feat: add GLM5Next MTP, optimize (#29928)
48499d2e1 qwen3tts : guard speaker_encoder_config patch for CustomVoice variant (#29179)
d0b490f25 sampling : use greedy selection for eligible temperature-zero chains (#29797)
42b021b4d vocab : add plamo fim tokens (#30090)
7481354a1 convert : fix token configuration for PLaMo-3 (#29843)
ad2156533 musa: use the tile lightning indexer kernel (#30080)
b7dafa01e vendor : update cpp-httplib to 0.60.0 (#30081)
26908739b imatrix : include clocale for std::setlocale (#30079)
36a73916e ggml-webgpu: fix flash_attn supports_op check for overlapping KV (#28205)
fa3c2fab3 tests : retain the anchor when testing recurrent rollback (#29923)
005a1e127 [SYCL] fix the issue in mixed different model GPUs in FA (#29071)
d2a79e604 sycl: accelerate GLM MLA prefill with MKL flash attention (#29171)
5e5b628eb sycl : fattn_kv_buffers cleanup (#27689)
4d756bc72 vulkan: fix amd iGPU slow checkpoint read (#30049)
78651c410 sycl: add IQ3_S multi-column MMVQ (#29500)
f498f864f ggml-webgpu: no dawn native features on wasi (#27069)
c479922ac hexagon: CPY/CONCAT/CONT/DUP overhaul to use DMA/HVX for all cases (#30067)
5ad1c5da0 cuda : add BF16 support for XIELU (#29955)
51ce9c11a ggml-cuda: use per-thread stream for buffer-init padding memset (#28782)
abeada335 vocab : implement PLaMo-3 tokenizer pre-segmentation (#30045)
462524043 model : add K2 Horizon dense and MoVA support (#29535)
310991409 llama: remove the gather path of the glm5-next sparse attention (#30042)
4fbc76dec model: support embeddinggemma2 (text+vision+audio) (#30054)
2207c8e57 opencl: fix OOB read in adreno xmem GEMM (#30041)
a46709b68 RPC: add `-sm tensor` (#26610)
65840ed53 ggml: fix CLAMP on non-contiguous views (CPU, CUDA) (#29517)
ab09ea4c1 cuda: BF16/FP16 conversion to f32 chunking (#29442)
da263e727 models: support pplx-decider (#30044)
a043d38a6 metal : fix excess threadgroup memory in quantized flash attention (#29340)
58cb9138e vulkan : check for null vkEnumerateInstanceVersion (#29872)
4f5406761 HIP: use -O0 for host code in debug builds (#29795)
f0c41e016 models : consolidate nextn row cropping into shared helpers (#30017)
d7a695ef6 scripts : limit apiabi checks to libllama and libmtmd (#30038)
6c73b3e12 convert : add text_config as fallback [transformers 5.18] (#30040)
1a3011cc0 llama : re-reserve the sched when the nextn extraction flags change (#30020)
6753a033f ggml: refactor selective expert copying to user code (#29943)
cbb7d52ec test-llama-archs : initialize backends before generating models (#30034)
63bef2728 vendor : update LibreSSL to 4.3.3 [no ci] (#30019)
b9a5a00b8 ggml-openvino: fix CI tests; fix GPU regressions. (#30037)
43fe9c642 llama: fix k-pool scatter data race on shared sequences (#29994)
5e03bdd87 hexagon: ssm-conv updates (#29971)
50569eb87 hexagon: add pool op support (#29995)
7049ff0cb ci : add 1accel label [no ci] (#30016)
c25030496 ci : skip container re-tagging when Require Docker is disabled (#30008)
8345f3339 hexagon: matmul and flash-atten scalability updates (#29974)
▸ 主项目介绍 · ggerganov/llama.cpp
▸ README 原文（同步于 2026-07-19 13:24）
# llama.cpp
[](https://opensource.org/licenses/MIT)
[](https://github.com/ggml-org/llama.cpp/releases)
[](https://github.com/ggml-org/llama.cpp/actions/workflows/server.yml)
[](https://github.com/ggml-org/llama.cpp/actions/workflows/docker.yml)
[](https://github.com/ggml-org/llama.cpp/actions/workflows/winget.yml)
[Manifesto](https://github.com/ggml-org/llama.cpp/discussions/205) / [ggml](https://github.com/ggml-org/ggml) / [ops](https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md) / [maintainer PRs](https://github.com/ggml-org/llama.cpp/issues?q=is%3Apr%20is%3Aopen%20draft%3AFalse%20(author%3Argerganov%20OR%20author%3AKitaitiMakoto%20OR%20author%3Adanbev%20OR%20author%3Aaldehir%20OR%20author%3Amax-krasnyansky%20OR%20author%3ACISC%20OR%20author%3Aggerganov%20OR%20author%3Aam17an%20OR%20author%3Abartowski1182%20OR%20author%3Ahipudding%20OR%20author%3AServeurpersoCom%20OR%20author%3Apwilkin%20OR%20author%3Areeselevine%20OR%20author%3Angxson%20OR%20author%3Ajeffbolznv%20OR%20author%3A0cc4m%20OR%20author%3Aangt%20OR%20author%3AIMbackK%20OR%20author%3Aarthw%20OR%20author%3AJohannesGaessler%20OR%20author%3AORippler%20OR%20author%3Aruixiang63%20OR%20author%3Axctan%20OR%20author%3Aallozaur%20OR%20author%3Ayomaytk%20OR%20author%3Aaendk%20OR%20author%3Agaugarg-nv%20OR%20author%3Ataronaeo%20OR%20author%3Aforforever73%20OR%20author%3Alhez%20OR%20author%3Anetrunnereve%20OR%20author%3Afairydreaming)%20sort%3Aupdated-desc)
LLM inference in C/C++
## Recent API changes
- [Changelog for `libllama` API](https://github.com/ggml-org/llama.cpp/issues/9289)
- [Changelog for `llama-server` REST API](https://github.com/ggml-org/llama.cpp/issues/9291)
## Hot topics
- **Hugging Face cache migration: models downloaded with `-hf` are now stored in the standard Hugging Face cache directory, enabling sharing with other HF tools.**
- **[guide : using the new WebUI of llama.cpp](https://github.com/ggml-org/llama.cpp/discussions/16938)**
- [guide : running gpt-oss with llama.cpp](https://github.com/ggml-org/llama.cpp/discussions/15396)
- [[FEEDBACK] Better packaging for llama.cpp to support downstream consumers 🤗](https://github.com/ggml-org/llama.cpp/discussions/15313)
- Support for the `gpt-oss` model with native MXFP4 format has been added | [PR](https://github.com/ggml-org/llama.cpp/pull/15091) | [Collaboration with NVIDIA](https://blogs.nvidia.com/blog/rtx-ai-garage-openai-oss) | [Comment](https://github.com/ggml-org/llama.cpp/discussions/15095)
- Multimodal support arrived in `llama-server`: [#12898](https://github.com/ggml-org/llama.cpp/pull/12898) | [documentation](./docs/multimodal.md)
- VS Code extension for FIM completions: https://github.com/ggml-org/llama.vscode
- Vim/Neovim plugin for FIM completions: https://github.com/ggml-org/llama.vim
- Hugging Face Inference Endpoints now support GGUF out of the box! https://github.com/ggml-org/llama.cpp/discussions/9669
- Hugging Face GGUF editor: [discussion](https://github.com/ggml-org/llama.cpp/discussions/9268) | [tool](https://huggingface.co/spaces/CISCai/gguf-editor)
- WebGPU support is now available in the browser, see a blog/demo introducing it [here](https://reeselevine.github.io/llamas-on-the-web/).
----
## Quick start
Getting started with llama.cpp is straightforward. Here are several ways to install it on your machine:
- Install `llama.cpp` using [brew, nix, winget, or conda-forge](docs/install.md)
- Run with Docker - see our [Docker documentation](docs/docker.md)
- Download pre-built binaries from the [releases page](https://github.com/ggml-org/llama.cpp/releases)
- Build from source by cloning this repository - check out [our build guide](docs/build.md)
Once installed, you'll need a model to work with. Head to the [Obtaining and quantizing models](#obtaining-and-quantizing-models) section to learn more.
Example command:
```sh
# Use a local model file
llama-cli -m my_model.gguf
# Or download and run a model directly from Hugging Face
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF
# Launch OpenAI-compatible API server
llama-server -hf ggml-org/gemma-3-1b-it-GGUF
```
## Description
The main goal of `llama.cpp` is to enable LLM inference with minimal setup and state-of-the-art performance on a wide
range of hardware - locally and in the cloud.
- Plain C/C++ implementation without any dependencies
- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
- AVX, AVX2, AVX512 and AMX support for x86 architectures
- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
- Vulkan and SYCL backend support
- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
The `llama.cpp` project is the main playground for developing new features for the [ggml](https://github.com/ggml-org/ggml) library.
<details>
<summary>Models</summary>
Typically finetunes of the base models below are supported as well.
Instructions for adding support for new models: [HOWTO-add-model.md](docs/development/HOWTO-add-model.md)
#### Text-only
- [X] LLaMA 🦙
- [x] LLaMA 2 🦙🦙
- [x] LLaMA 3 🦙🦙🦙
- [X] [Mistral 7B](https://huggingface.co/mistralai/Mistral-7B-v0.1)
- [x] [Mixtral MoE](https://huggingface.co/models?search=mistral-ai/Mixtral)
- [x] [DBRX](https://huggingface.co/databricks/dbrx-instruct)
- [x] [Jamba](https://huggingface.co/ai21labs)
- [X] [Falcon](https://huggingface.co/models?search=tiiuae/falcon)
- [X] [Chinese LLaMA / Alpaca](https://github.com/ymcui/Chinese-LLaMA-Alpaca) and [Chinese LLaMA-2 / Alpaca-2](https://github.com/ymcui/Chinese-LLaMA-Alpaca-2)
- [X] [Vigogne (French)](https://github.com/bofenghuang/vigogne)
- [X] [BERT](https://github.com/ggml-org/llama.cpp/pull/5423)
- [X] [Koala](https://bair.berkeley.edu/blog/2023/04/03/koala/)
- [X] [Baichuan 1 & 2](https://huggingface.co/models?search=baichuan-inc/Baichuan) + [derivations](https://huggingface.co/hiyouga/baichuan-7b-sft)
- [X] [Aquila 1 & 2](https://huggingface.co/models?search=BAAI/Aquila)
- [X] [Starcoder models](https://github.com/ggml-org/llama.cpp/pull/3187)
- [X] [Refact](https://huggingface.co/smallcloudai/Refact-1_6B-fim)
- [X] [MPT](https://github.com/ggml-org/llama.cpp/pull/3417)
- [X] [Bloom](https://github.com/ggml-org/llama.cpp/pull/3553)
- [x] [Yi models](https://huggingface.co/models?search=01-ai/Yi)
- [X] [StableLM models](https://huggingface.co/stabilityai)
- [x] [Deepseek models](https://huggingface.co/models?search=deepseek-ai/deepseek)
- [x] [Qwen models](https://huggingface.co/models?search=Qwen/Qwen)
- [x] [PLaMo-13B](https://github.com/ggml-org/llama.cpp/pull/3557)
- [x] [Phi models](https://huggingface.co/models?search=microsoft/phi)
- [x] [PhiMoE](https://github.com/ggml-org/llama.cpp/pull/11003)
- [x] [GPT-2](https://huggingface.co/gpt2)
- [x] [Orion 14B](https://github.com/ggml-org/llama.cpp/pull/5118)
- [x] [InternLM2](https://huggingface.co/models?search=internlm2)
- [x] [CodeShell](https://github.com/WisdomShell/codeshell)
- [x] [Gemma](https://ai.google.dev/gemma)
- [x] [Mamba](https://github.com/state-spaces/mamba)
- [x] [Grok-1](https://huggingface.co/keyfan/grok-1-hf)
- [x] [Xverse](https://huggingface.co/models?search=xverse)
- [x] [Command-R models](https://huggingface.co/models?search=CohereForAI/c4ai-command-r)
- [x] [SEA-LION](https://huggingface.co/models?search=sea-lion)
- [x] [GritLM-7B](https://huggingface.co/GritLM/GritLM-7B) + [GritLM-8x7B](https://huggingface.co/GritLM/GritLM-8x7B)
- [x] [OLMo](https://allenai.org/olmo)
- [x] [OLMo 2](https://allenai.org/olmo)
- [x] [OLMoE](https://huggingface.co/allenai/OLMoE-1B-7B-0924)
- [x] [Granite models](https://huggingface.co/collections/ibm-granite/granite-code-models-6624c5cec322e4c148c8b330)
- [x] [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) + [Pythia](https://github.com/EleutherAI/pythia)
- [x] [Snowflake-Arctic MoE](https://huggingface.co/collections/Snowflake/arctic-66290090abe542894a5ac520)
- [x] [Smaug](https://huggingface.co/models?search=Smaug)
- [x] [Poro 34B](https://huggingface.co/LumiOpen/Poro-34B)
- [x] [Bitnet b1.58 models](https://huggingface.co/1bitLLM)
- [x] [Flan T5](https://huggingface.co/models?search=flan-t5)
- [x] [Open Elm models](https://huggingface.co/collections/apple/openelm-instruct-models-6619ad295d7ae9f868b759ca)
- [x] [ChatGLM3-6b](https://huggingface.co/THUDM/chatglm3-6b) + [ChatGLM4-9b](https://huggingface.co/THUDM/glm-4-9b) + [GLMEdge-1.5b](https://huggingface.co/THUDM/glm-edge-1.5b-chat) + [GLMEdge-4b](https://huggingface.co/THUDM/glm-edge-4b-chat)
- [x] [GLM-4-0414](https://huggingface.co/collections/THUDM/glm-4-0414-67f3cbcb34dd9d252707cb2e)
- [x] [SmolLM](https://huggingface.co/collections/HuggingFaceTB/smollm-6695016cad7167254ce15966)
- [x] [EXAONE-3.0-7.8B-Instruct](https://huggingface.co/LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct)
- [x] [FalconMamba Models](https://huggingface.co/collections/tiiuae/falconmamba-7b-66b9a580324dd1598b0f6d4a)
- [x] [Jais](https://huggingface.co/inceptionai/jais-13b-chat)
- [x] [Bielik-11B-v2.3](https://huggingface.co/collections/speakleash/bielik-11b-v23-66ee813238d9b526a072408a)
- [x] [RWKV-7](https://huggingface.co/collections/shoumenchougou/rwkv7-gxx-gguf)
- [x] [RWKV-6](https://github.com/BlinkDL/RWKV-LM)
- [x] [QRWKV-6](https://huggingface.co/recursal/QRWKV6-32B-Instruct-Preview-v0.1)
- [x] [GigaChat-20B-A3B](https://huggingface.co/ai-sage/GigaChat-20B-A3B-instruct)
- [X] [Trillion-7B-preview](https://huggingface.co/trillionlabs/Trillion-7B-preview)
- [x] [Ling models](https://huggingface.co/collections/inclusionAI/ling-67c51c85b34a7ea0aba94c32)
- [x] [Liquid LFM2 models](https://huggingface.co/collections/LiquidAI/lfm2)
- [x] [Liquid LFM2.5 models](https://huggingface.co/collections/LiquidAI/lfm25)
- [x] [Liquid Nanos](https://huggingface.co/collections/LiquidAI/liquid-nanos)
- [x] [Hunyuan models](https://huggingface.co/collections/tencent/hunyuan-dense-model-6890632cda26b19119c9c5e7)
- [x] [BailingMoeV2 (Ring/Ling 2.0) models](https://huggingface.co/collections/inclusionAI/ling-v2-68bf1dd2fc34c306c1fa6f86)
- [x] [Mellum models](https://huggingface.co/JetBrains/models?search=mellum)
#### Multimodal
- [x] [LLaVA 1.5 models](https://huggingface.co/collections/liuhaotian/llava-15-653aac15d994e992e2677a7e), [LLaVA 1.6 models](https://huggingface.co/collections/liuhaotian/llava-16-65b9e40155f60fd046a5ccf2)
- [x] [BakLLaVA](https://huggingface.co/models?search=SkunkworksAI/Bakllava)
- [x] [Obsidian](https://huggingface.co/NousResearch/Obsidian-3B-V0.5)
- [x] [ShareGPT4V](https://huggingface.co/models?search=Lin-Chen/ShareGPT4V)
- [x] [MobileVLM 1.7B/3B models](https://huggingface.co/models?search=mobileVLM)
- [x] [Yi-VL](https://huggingface.co/models?search=Yi-VL)
- [x] [Mini CPM](https://huggingface.co/models?search=MiniCPM)
- [x] [Moondream](https://huggingface.co/vikhyatk/moondream2)
- [x] [Bunny](https://github.com/BAAI-DCAI/Bunny)
- [x] [GLM-EDGE](https://huggingface.co/models?search=glm-edge)
- [x] [Qwen2-VL](https://huggingface.co/collections/Qwen/qwen2-vl-66cee7455501d7126940800d)
- [x] [LFM2-VL](https://huggingface.co/collections/LiquidAI/lfm2-vl-68963bbc84a610f7638d5ffa)
</details>
<details>
<summary>Bindings</summary>
- Python: [ddh0/easy-llama](https://github.com/ddh0/easy-llama)
- Python: [abetlen/llama-cpp-python](https://github.com/abetlen/llama-cpp-python)
- Go: [go-skynet/go-llama.cpp](https://github.com/go-skynet/go-llama.cpp)
- Node.js: [withcatai/node-llama-cpp](https://github.com/withcatai/node-llama-cpp)
- JS/TS (llama.cpp server client): [lgrammel/modelfusion](https://modelfusion.dev/integration/model-provider/llamacpp)
- JS/TS (Programmable Prompt Engine CLI): [offline-ai/cli](https://github.com/offline-ai/cli)
- JavaScript/Wasm (works in browser): [tangledgroup/llama-cpp-wasm](https://github.com/tangledgroup/llama-cpp-wasm)
- Typescript/Wasm (nicer API, available on npm): [ngxson/wllama](https://github.com/ngxson/wllama)
- Ruby: [yoshoku/llama_cpp.rb](https://github.com/yoshoku/llama_cpp.rb)
- Ruby: [docusealco/rllama](https://github.com/docusealco/rllama)
- Rust (more features): [edgenai/llama_cpp-rs](https://github.com/edgenai/llama_cpp-rs)
- Rust (nicer API): [mdrokz/rust-llama.cpp](https://github.com/mdrokz/rust-llama.cpp)
- Rust (more direct bindings): [utilityai/llama-cpp-rs](https://github.com/utilityai/llama-cpp-rs)
- Rust (automated build from crates.io): [ShelbyJenkins/llm_client](https://github.com/ShelbyJenkins/llm_client)
- C#/.NET: [SciSharp/LLamaSharp](https://github.com/SciSharp/LLamaSharp)
- C#/VB.NET (more features - community license): [LM-Kit.NET](https://docs.lm-kit.com/lm-kit-net/index.html)
- Scala 3: [donderom/llm4s](https://github.com/donderom/llm4s)
- Clojure: [phronmophobic/llama.clj](https://github.com/phronmophobic/llama.clj)
- React Native: [mybigday/llama.rn](https://github.com/mybigday/llama.rn)
- Java: [kherud/java-llama.cpp](https://github.com/kherud/java-llama.cpp)
- Java: [QuasarByte/llama-cpp-jna](https://github.com/QuasarByte/llama-cpp-jna)
- Zig: [deins/llama.cpp.zig](https://github.com/Deins/llama.cpp.zig)
- Flutter/Dart: [netdur/llama_cpp_dart](https://github.com/netdur/llama_cpp_dart)
- Flutter: [xuegao-tzx/Fllama](https://github.com/xuegao-tzx/Fllama)
- PHP (API bindings and features built on top of llama.cpp): [distantmagic/resonance](https://github.com/distantmagic/resonance) [(more info)](https://github.com/ggml-org/llama.cpp/pull/6326)
- Guile Scheme: [guile_llama_cpp](https://savannah.nongnu.org/projects/guile-llama-cpp)
- Swift [srgtuszy/llama-cpp-swift](https://github.com/srgtuszy/llama-cpp-swift)
- Swift [ShenghaiWang/SwiftLlama](https://github.com/ShenghaiWang/SwiftLlama)
- Delphi [Embarcadero/llama-cpp-delphi](https://github.com/Embarcadero/llama-cpp-delphi)
- Go (no CGo needed): [hybridgroup/yzma](https://github.com/hybridgroup/yzma)
- Android: [llama.android](/examples/llama.android)
</details>
<details>
<summary>UIs</summary>
*(to have a project listed here, it should clearly state that it depends on `llama.cpp`)*
- [AI Sublime Text plugin](https://github.com/yaroslavyaroslav/OpenAI-sublime-text) (MIT)
- [BonzAI App](https://apps.apple.com/us/app/bonzai-your-local-ai-agent/id6752847988) (proprietary)
- [cztomsik/ava](https://github.com/cztomsik/ava) (MIT)
- [Dot](https://github.com/alexpinel/Dot) (GPL)
- [eva](https://github.com/ylsdamxssjxxdd/eva) (MIT)
- [iohub/collama](https://github.com/iohub/coLLaMA) (Apache-2.0)
- [janhq/jan](https://github.com/janhq/jan) (AGPL)
- [johnbean393/Sidekick](https://github.com/johnbean393/Sidekick) (MIT)
- [KanTV](https://github.com/zhouwg/kantv?tab=readme-ov-file) (Apache-2.0)
- [KodiBot](https://github.com/firatkiral/kodibot) (GPL)
- [llama.vim](https://github.com/ggml-org/llama.vim) (MIT)
- [LARS](https://github.com/abgulati/LARS) (AGPL)
- [Llama Assistant](https://github.com/vietanhdev/llama-assistant) (GPL)
- [LlamaLib](https://github.com/undreamai/LlamaLib) (Apache-2.0)
- [LLMFarm](https://github.com/guinmoon/LLMFarm?tab=readme-ov-file) (MIT)
- [LLMUnity](https://github.com/undreamai/LLMUnity) (MIT)
- [LMStudio](https://lmstudio.ai/) (proprietary)
- [LocalAI](https://github.com/mudler/LocalAI) (MIT)
- [LostRuins/koboldcpp](https://github.com/LostRuins/koboldcpp) (AGPL)
- [MindMac](https://mindmac.app) (proprietary)
- [MindWorkAI/AI-Studio](https://github.com/MindWorkAI/AI-Studio) (FSL-1.1-MIT)
- [Mobile-Artificial-Intelligence/maid](https://github.com/Mobile-Artificial-Intelligence/maid) (MIT)
- [Mozilla-Ocho/llamafile](https://github.com/Mozilla-Ocho/llamafile) (Apache-2.0)
- [nat/openplayground](https://github.com/nat/openplayground) (MIT)
- [nomic-ai/gpt4all](https://github.com/nomic-ai/gpt4all) (MIT)
- [ollama/ollama](https://github.com/ollama/ollama) (MIT)
- [oobabooga/text-generation-webui](https://github.com/oobabooga/text-generation-webui) (AGPL)
- [PocketPal AI](https://github.com/a-ghorbani/pocketpal-ai) (MIT)
- [psugihara/FreeChat](https://github.com/psugihara/FreeChat) (MIT)
- [ptsochantaris/emeltal](https://github.com/ptsochantaris/emeltal) (MIT)
- [pythops/tenere](https://github.com/pythops/tenere) (AGPL)
- [ramalama](https://github.com/containers/ramalama) (MIT)
- [semperai/amica](https://github.com/semperai/amica) (MIT)
- [withcatai/catai](https://github.com/withcatai/catai) (MIT)
- [Autopen](https://github.com/blackhole89/autopen) (GPL)
</details>
<details>
<summary>Tools</summary>
- [akx/ggify](https://github.com/akx/ggify) – download PyTorch models from Hugging Face Hub and convert them to GGML
- [akx/ollama-dl](https://github.com/akx/ollama-dl) – download models from the Ollama library to be used directly with llama.cpp
- [crashr/gppm](https://github.com/crashr/gppm) – launch llama.cpp instances utilizing NVIDIA Tesla P40 or P100 GPUs with reduced idle power consumption
- [gpustack/gguf-parser](https://github.com/gpustack/gguf-parser-go/tree/main/cmd/gguf-parser) - review/check the GGUF file and estimate the memory usage
- [Styled Lines](https://marketplace.unity.com/packages/tools/generative-ai/styled-lines-llama-cpp-model-292902) (proprietary licensed, async wrapper of inference part for game development in Unity3d with pre-built Mobile and Web platform wrappers and a model example)
- [unslothai/unsloth](https://github.com/unslothai/unsloth) – 🦥 exports/saves fine-tuned and trained models to GGUF (Apache-2.0)
</details>
<details>
<summary>Infrastructure</summary>
- [Paddler](https://github.com/intentee/paddler) - Open-source LLMOps platform for hosting and scaling AI in your own infrastructure
- [GPUStack](https://github.com/gpustack/gpustack) - Manage GPU clusters for running LLMs
- [llama_cpp_canister](https://github.com/onicai/llama_cpp_canister) - llama.cpp as a smart contract on the Internet Computer, using WebAssembly
- [llama-swap](https://github.com/mostlygeek/llama-swap) - transparent proxy that adds automatic model switching with llama-server
- [Kalavai](https://github.com/kalavai-net/kalavai-client) - Crowdsource end to end LLM deployment at any scale
- [llmaz](https://github.com/InftyAI/llmaz) - ☸️ Easy, advanced inference platform for large language models on Kubernetes.
- [LLMKube](https://github.com/defilantech/llmkube) - Kubernetes operator for llama.cpp with multi-GPU and Apple Silicon Metal
support"
</details>
<details>
<summary>Games</summary>
- [Lucy's Labyrinth](https://github.com/MorganRO8/Lucys_Labyrinth) - A simple maze game where agents controlled by an AI model will try to trick you.
</details>
## Supported backends
| Backend | Target devices |
| --- | --- |
| [Metal](docs/build.md#metal-build) | Apple Silicon |
| [BLAS](docs/build.md#blas-build) | All |
| [BLIS](docs/backend/BLIS.md) | All |
| [SYCL](docs/backend/SYCL.md) | Intel GPU |
| [OpenVINO [In Progress]](docs/backend/OPENVINO.md) | Intel CPUs, GPUs, and NPUs |
| [MUSA](docs/build.md#musa) | Moore Threads GPU |
| [CUDA](docs/build.md#cuda) | Nvidia GPU |
| [HIP](docs/build.md#hip) | AMD GPU |
| [ZenDNN](docs/build.md#zendnn) | AMD CPU |
| [Vulkan](docs/build.md#vulkan) | GPU |
| [CANN](docs/build.md#cann) | Ascend NPU |
| [OpenCL](docs/backend/OPENCL.md) | Adreno GPU |
| [IBM zDNN](docs/backend/zDNN.md) | IBM Z & LinuxONE |
| [WebGPU](docs/build.md#webgpu) | All |
| [RPC](https://github.com/ggml-org/llama.cpp/tree/master/tools/rpc) | All |
| [Hexagon [In Progress]](docs/backend/snapdragon/README.md) | Snapdragon |
| [VirtGPU](docs/backend/VirtGPU.md) | VirtGPU APIR |
## Obtaining and quantizing models
The [Hugging Face](https://huggingface.co) platform hosts a [number of LLMs](https://huggingface.co/models?library=gguf&sort=trending) compatible with `llama.cpp`:
- [Trending](https://huggingface.co/models?library=gguf&sort=trending)
- [LLaMA](https://huggingface.co/models?sort=trending&search=llama+gguf)
You can either manually download the GGUF file or directly use any `llama.cpp`-compatible models from [Hugging Face](https://huggingface.co/) or other model hosting sites, by using this CLI argument: `-hf <user>/<model>[:quant]`. For example:
```sh
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF
```
By default, the CLI would download from Hugging Face, you can switch to other options with the environment variable `MODEL_ENDPOINT`. The `MODEL_ENDPOINT` must point to a Hugging Face compatible API endpoint.
After downloading a model, use the CLI tools to run it locally - see below.
`llama.cpp` requires the model to be stored in the [GGUF](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) file format. Models in other data formats can be converted to GGUF using the `convert_*.py` Python scripts in this repo.
The Hugging Face platform provides a variety of online tools for converting, quantizing and hosting models with `llama.cpp`:
- Use the [GGUF-my-repo space](https://huggingface.co/spaces/ggml-org/gguf-my-repo) to convert to GGUF format and quantize model weights to smaller sizes
- Use the [GGUF-my-LoRA space](https://huggingface.co/spaces/ggml-org/gguf-my-lora) to convert LoRA adapters to GGUF format (more info: https://github.com/ggml-org/llama.cpp/discussions/10123)
- Use the [GGUF-editor space](https://huggingface.co/spaces/CISCai/gguf-editor) to edit GGUF meta data in the browser (more info: https://github.com/ggml-org/llama.cpp/discussions/9268)
- Use the [Inference Endpoints](https://ui.endpoints.huggingface.co/) to directly host `llama.cpp` in the cloud (more info: https://github.com/ggml-org/llama.cpp/discussions/9669)
To learn more about model quantization, [read this documentation](tools/quantize/README.md)
## [`llama-cli`](tools/cli)
#### A CLI tool for accessing and experimenting with most of `llama.cpp`'s functionality.
- <details open>
<summary>Run in conversation mode</summary>
Models with a built-in chat template will automatically activate conversation mode. If this doesn't occur, you can manually enable it by adding `-cnv` and specifying a suitable chat template with `--chat-template NAME`
```bash
llama-cli -m model.gguf
# > hi, who are you?
# Hi there! I'm your helpful assistant! I'm an AI-powered chatbot designed to assist and provide information to users like you. I'm here to help answer your questions, provide guidance, and offer support on a wide range of topics. I'm a friendly and knowledgeable AI, and I'm always happy to help with anything you need. What's on your mind, and how can I assist you today?
#
# > what is 1+1?
# Easy peasy! The answer to 1+1 is... 2!
```
</details>
- <details>
<summary>Run in conversation mode with custom chat template</summary>
```bash
# use the "chatml" template (use -h to see the list of supported templates)
llama-cli -m model.gguf -cnv --chat-template chatml
# use a custom template
llama-cli -m model.gguf -cnv --in-prefix 'User: ' --reverse-prompt 'User:'
```
</details>
- <details>
<summary>Constrain the output with a custom grammar</summary>
```bash
llama-cli -m model.gguf -n 256 --grammar-file grammars/json.gbnf -p 'Request: schedule a call at 8pm; Command:'
# {"appointmentTime": "8pm", "appointmentDetails": "schedule a a call"}
```
The [grammars/](grammars/) folder contains a handful of sample grammars. To write your own, check out the [GBNF Guide](grammars/README.md).
For authoring more complex JSON grammars, check out https://grammar.intrinsiclabs.ai/
</details>
## [`llama-server`](tools/server)
#### A lightweight, [OpenAI API](https://github.com/openai/openai-openapi) compatible, HTTP server for serving LLMs.
- <details open>
<summary>Start a local HTTP server with default configuration on port 8080</summary>
```bash
llama-server -m model.gguf --port 8080
# Basic web UI can be accessed via browser: http://localhost:8080
# Chat completion endpoint: http://localhost:8080/v1/chat/completions
```
</details>
- <details>
<summary>Support multiple-users and parallel decoding</summary>
```bash
# up to 4 concurrent requests, each with 4096 max context
llama-server -m model.gguf -c 16384 -np 4
```
</details>
- <details>
<summary>Enable speculative decoding</summary>
```bash
# the draft.gguf model should be a small variant of the target model.gguf
llama-server -m model.gguf -md draft.gguf
```
</details>
- <details>
<summary>Serve an embedding model</summary>
```bash
# use the /embedding endpoint
llama-server -m model.gguf --embedding --pooling cls -ub 8192
```
</details>
- <details>
<summary>Serve a reranking model</summary>
```bash
# use the /reranking endpoint
llama-server -m model.gguf --reranking
```
</details>
- <details>
<summary>Constrain all outputs with a grammar</summary>
```bash
# custom grammar
llama-server -m model.gguf --grammar-file grammar.gbnf
# JSON
llama-server -m model.gguf --grammar-file grammars/json.gbnf
```
</details>
## [`llama-perplexity`](tools/perplexity)
#### A tool for measuring the [perplexity](tools/perplexity/README.md) [^1] (and other quality metrics) of a model over a given text.
- <details open>
<summary>Measure the perplexity over a text file</summary>
```bash
llama-perplexity -m model.gguf -f file.txt
# [1]15.2701,[2]5.4007,[3]5.3073,[4]6.2965,[5]5.8940,[6]5.6096,[7]5.7942,[8]4.9297, ...
# Final estimate: PPL = 5.4007 +/- 0.67339
```
</details>
- <details>
<summary>Measure KL divergence</summary>
```bash
# TODO
```
</details>
[^1]: [https://huggingface.co/docs/transformers/perplexity](https://huggingface.co/docs/transformers/perplexity)
## [`llama-bench`](tools/llama-bench)
#### Benchmark the performance of the inference for various parameters.
- <details open>
<summary>Run default benchmark</summary>
```bash
llama-bench -m model.gguf
# Output:
# | model | size | params | backend | threads | test | t/s |
# | ------------------- | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |
# | qwen2 1.5B Q4_0 | 885.97 MiB | 1.54 B | Metal,BLAS | 16 | pp512 | 5765.41 ± 20.55 |
# | qwen2 1.5B Q4_0 | 885.97 MiB | 1.54 B | Metal,BLAS | 16 | tg128 | 197.71 ± 0.81 |
#
# build: 3e0ba0e60 (4229)
```
</details>
## [`llama-simple`](examples/simple)
#### A minimal example for implementing apps with `llama.cpp`. Useful for developers.
- <details>
<summary>Basic text completion</summary>
```bash
llama-simple -m model.gguf
# Hello my name is Kaitlyn and I am a 16 year old girl. I am a junior in high school and I am currently taking a class called "The Art of
```
</details>
## Contributing
- Contributors can open PRs
- Collaborators will be invited based on contributions
- Maintainers can push to branches in the `llama.cpp` repo and merge PRs into the `master` branch
- Any help with managing issues, PRs and projects is very appreciated!
- See [good first issues](https://github.com/ggml-org/llama.cpp/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) for tasks suitable for first contributions
- Read the [CONTRIBUTING.md](CONTRIBUTING.md) for more information
- Make sure to read this: [Inference at the edge](https://github.com/ggml-org/llama.cpp/discussions/205)
- A bit of backstory for those who are interested: [Changelog podcast](https://changelog.com/podcast/532)
## Other documentation
- [cli](tools/cli/README.md)
- [completion](tools/completion/README.md)
- [server](tools/server/README.md)
- [GBNF grammars](grammars/README.md)
#### Development documentation
- [How to build](docs/build.md)
- [Running on Docker](docs/docker.md)
- [Build on Android](docs/android.md)
- [Multi-GPU usage](docs/multi-gpu.md)
- [Performance troubleshooting](docs/development/token_generation_performance_tips.md)
- [GGML tips & tricks](https://github.com/ggml-org/llama.cpp/wiki/GGML-Tips-&-Tricks)
#### Seminal papers and background on the models
If your issue is with model generation quality, then please at least scan the following links and papers to understand the limitations of LLaMA models. This is especially important when choosing an appropriate model size and appreciating both the significant and subtle differences between LLaMA models and ChatGPT:
- LLaMA:
- [Introducing LLaMA: A foundational, 65-billion-parameter large language model](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/)
- [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971)
- GPT-3
- [Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)
- GPT-3.5 / InstructGPT / ChatGPT:
- [Aligning language models to follow instructions](https://openai.com/research/instruction-following)
- [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155)
## XCFramework
The XCFramework is a precompiled version of the library for iOS, visionOS, tvOS,
and macOS. It can be used in Swift projects without the need to compile the
library from source. For example:
```swift
// swift-tools-version: 5.10
// The swift-tools-version declares the minimum version of Swift required to build this package.
import PackageDescription
let package = Package(
name: "MyLlamaPackage",
targets: [
.executableTarget(
name: "MyLlamaPackage",
dependencies: [
"LlamaFramework"
]),
.binaryTarget(
name: "LlamaFramework",
url: "https://github.com/ggml-org/llama.cpp/releases/download/b5046/llama-b5046-xcframework.zip",
checksum: "c19be78b5f00d8d29a25da41042cb7afa094cbf6280a225abe614b03b20029ab"
)
]
)
```
The above example is using an intermediate build `b5046` of the library. This can be modified
to use a different version by changing the URL and checksum.
## Completions
Command-line completion is available for some environments.
#### Bash Completion
```bash
$ build/bin/llama-cli --completion-bash > ~/.llama-completion.bash
$ source ~/.llama-completion.bash
```
Optionally this can be added to your `.bashrc` or `.bash_profile` to load it
automatically. For example:
```console
$ echo "source ~/.llama-completion.bash" >> ~/.bashrc
```
## Dependencies
- [yhirose/cpp-httplib](https://github.com/yhirose/cpp-httplib) - Single-header HTTP server, used by `llama-server` - MIT license
- [stb-image](https://github.com/nothings/stb) - Single-header image format decoder, used by multimodal subsystem - Public domain
- [nlohmann/json](https://github.com/nlohmann/json) - Single-header JSON library, used by various tools/examples - MIT License
- [miniaudio.h](https://github.com/mackron/miniaudio) - Single-header audio format decoder, used by multimodal subsystem - Public domain
- [subprocess.h](https://github.com/sheredom/subprocess.h) - Single-header process launching solution for C and C++ - Public domain