跳到主要内容

llama.cpp

llama

License: MIT Release Server

Roadmap / Project status / Manifesto / ggml

Inference of Meta's LLaMA model (and others) in pure C/C++

Recent API changes​

Hot topics​

Quick start​

Getting started with llama.cpp is straightforward. Here are several ways to install it on your machine:

Once installed, you'll need a model to work with. Head to the Obtaining and quantizing models section to learn more.

Example command:

# Use a local model file
llama-cli -m my_model.gguf

# Or download and run a model directly from Hugging Face
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

# Launch OpenAI-compatible API server
llama-server -hf ggml-org/gemma-3-1b-it-GGUF

Description​

The main goal of llama.cpp is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

  • Plain C/C++ implementation without any dependencies
  • Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
  • AVX, AVX2, AVX512 and AMX support for x86 architectures
  • 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
  • Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
  • Vulkan and SYCL backend support
  • CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is the main playground for developing new features for the ggml library.

Models

Typically finetunes of the base models below are supported as well.

Instructions for adding support for new models: HOWTO-add-model.md

Text-only​

Multimodal​

Bindings
UIs

(to have a project listed here, it should clearly state that it depends on llama.cpp)

Tools
  • akx/ggify – download PyTorch models from HuggingFace Hub and convert them to GGML
  • akx/ollama-dl – download models from the Ollama library to be used directly with llama.cpp
  • crashr/gppm – launch llama.cpp instances utilizing NVIDIA Tesla P40 or P100 GPUs with reduced idle power consumption
  • gpustack/gguf-parser - review/check the GGUF file and estimate the memory usage
  • Styled Lines (proprietary licensed, async wrapper of inference part for game development in Unity3d with pre-built Mobile and Web platform wrappers and a model example)
Infrastructure
  • Paddler - Stateful load balancer custom-tailored for llama.cpp
  • GPUStack - Manage GPU clusters for running LLMs
  • llama_cpp_canister - llama.cpp as a smart contract on the Internet Computer, using WebAssembly
  • llama-swap - transparent proxy that adds automatic model switching with llama-server
  • Kalavai - Crowdsource end to end LLM deployment at any scale
  • llmaz - ☸️ Easy, advanced inference platform for large language models on Kubernetes.
Games
  • Lucy's Labyrinth - A simple maze game where agents controlled by an AI model will try to trick you.

Supported backends​

BackendTarget devices
MetalApple Silicon
BLASAll
BLISAll
SYCLIntel and Nvidia GPU
MUSAMoore Threads GPU
CUDANvidia GPU
HIPAMD GPU
VulkanGPU
CANNAscend NPU
OpenCLAdreno GPU
RPCAll

Obtaining and quantizing models​

The Hugging Face platform hosts a number of LLMs compatible with llama.cpp:

You can either manually download the GGUF file or directly use any llama.cpp-compatible models from Hugging Face or other model hosting sites, such as ModelScope, by using this CLI argument: -hf <user>/<model>[:quant]. For example:

llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

By default, the CLI would download from Hugging Face, you can switch to other options with the environment variable MODEL_ENDPOINT. For example, you may opt to downloading model checkpoints from ModelScope or other model sharing communities by setting the environment variable, e.g. MODEL_ENDPOINT=https://www.modelscope.cn/.

After downloading a model, use the CLI tools to run it locally - see below.

llama.cpp requires the model to be stored in the GGUF file format. Models in other data formats can be converted to GGUF using the convert_*.py Python scripts in this repo.

The Hugging Face platform provides a variety of online tools for converting, quantizing and hosting models with llama.cpp:

To learn more about model quantization, read this documentation

llama-cli​

A CLI tool for accessing and experimenting with most of llama.cpp's functionality.​

  • Run in conversation mode

    Models with a built-in chat template will automatically activate conversation mode. If this doesn't occur, you can manually enable it by adding -cnv and specifying a suitable chat template with --chat-template NAME

    llama-cli -m model.gguf

    # > hi, who are you?
    # Hi there! I'm your helpful assistant! I'm an AI-powered chatbot designed to assist and provide information to users like you. I'm here to help answer your questions, provide guidance, and offer support on a wide range of topics. I'm a friendly and knowledgeable AI, and I'm always happy to help with anything you need. What's on your mind, and how can I assist you today?
    #
    # > what is 1+1?
    # Easy peasy! The answer to 1+1 is... 2!
  • Run in conversation mode with custom chat template
    # use the "chatml" template (use -h to see the list of supported templates)
    llama-cli -m model.gguf -cnv --chat-template chatml

    # use a custom template
    llama-cli -m model.gguf -cnv --in-prefix 'User: ' --reverse-prompt 'User:'
  • Run simple text completion

    To disable conversation mode explicitly, use -no-cnv

    llama-cli -m model.gguf -p "I believe the meaning of life is" -n 128 -no-cnv

    # I believe the meaning of life is to find your own truth and to live in accordance with it. For me, this means being true to myself and following my passions, even if they don't align with societal expectations. I think that's what I love about yoga – it's not just a physical practice, but a spiritual one too. It's about connecting with yourself, listening to your inner voice, and honoring your own unique journey.
  • Constrain the output with a custom grammar
    llama-cli -m model.gguf -n 256 --grammar-file grammars/json.gbnf -p 'Request: schedule a call at 8pm; Command:'

    # {"appointmentTime": "8pm", "appointmentDetails": "schedule a a call"}

    The grammars/ folder contains a handful of sample grammars. To write your own, check out the GBNF Guide.

    For authoring more complex JSON grammars, check out https://grammar.intrinsiclabs.ai/

llama-server​

A lightweight, OpenAI API compatible, HTTP server for serving LLMs.​

  • Start a local HTTP server with default configuration on port 8080
    llama-server -m model.gguf --port 8080

    # Basic web UI can be accessed via browser: http://localhost:8080
    # Chat completion endpoint: http://localhost:8080/v1/chat/completions
  • Support multiple-users and parallel decoding
    # up to 4 concurrent requests, each with 4096 max context
    llama-server -m model.gguf -c 16384 -np 4
  • Enable speculative decoding
    # the draft.gguf model should be a small variant of the target model.gguf
    llama-server -m model.gguf -md draft.gguf
  • Serve an embedding model
    # use the /embedding endpoint
    llama-server -m model.gguf --embedding --pooling cls -ub 8192
  • Serve a reranking model
    # use the /reranking endpoint
    llama-server -m model.gguf --reranking
  • Constrain all outputs with a grammar
    # custom grammar
    llama-server -m model.gguf --grammar-file grammar.gbnf

    # JSON
    llama-server -m model.gguf --grammar-file grammars/json.gbnf

llama-perplexity​

A tool for measuring the perplexity 12 (and other quality metrics) of a model over a given text.​

  • Measure the perplexity over a text file
    llama-perplexity -m model.gguf -f file.txt

    # [1]15.2701,[2]5.4007,[3]5.3073,[4]6.2965,[5]5.8940,[6]5.6096,[7]5.7942,[8]4.9297, ...
    # Final estimate: PPL = 5.4007 +/- 0.67339
  • Measure KL divergence
    # TODO

llama-bench​

Benchmark the performance of the inference for various parameters.​

  • Run default benchmark
    llama-bench -m model.gguf

    # Output:
    # | model               |       size |     params | backend    | threads |          test |                  t/s |
    # | ------------------- | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |
    # | qwen2 1.5B Q4_0     | 885.97 MiB |     1.54 B | Metal,BLAS |      16 |         pp512 |      5765.41 ± 20.55 |
    # | qwen2 1.5B Q4_0     | 885.97 MiB |     1.54 B | Metal,BLAS |      16 |         tg128 |        197.71 ± 0.81 |
    #
    # build: 3e0ba0e60 (4229)

llama-run​

A comprehensive example for running llama.cpp models. Useful for inferencing. Used with RamaLama 3.​

  • Run a model with a specific prompt (by default it's pulled from Ollama registry)
    llama-run granite-code

llama-simple​

A minimal example for implementing apps with llama.cpp. Useful for developers.​

  • Basic text completion
    llama-simple -m model.gguf

    # Hello my name is Kaitlyn and I am a 16 year old girl. I am a junior in high school and I am currently taking a class called "The Art of

Contributing​

  • 贡献者可以开启 PR（Pull Request，拉取请求）
  • 协作者可以推送到 llama.cpp 仓库的分支并将 PR 合并到 master 主分支
  • 我们会根据贡献情况邀请协作者
  • 非常感谢任何在管理问题、PR 和项目方面的帮助！
  • 查看适合首次贡献的问题，这些任务适合初次贡献
  • 阅读 CONTRIBUTING.md 获取更多信息
  • 务必阅读：边缘推理
  • 对于感兴趣的朋友，这里有一些背景故事：Changelog 播客

其他文档​

开发文档​

相关论文和模型背景​

如果您遇到的问题与模型生成质量有关，请至少浏览以下链接和论文，以了解 LLaMA 模型的局限性。这在选择合适的模型大小以及理解 LLaMA 模型与 ChatGPT 之间的显著差异和细微差别时尤为重要：

XCFramework​

XCFramework 是针对 iOS、visionOS、tvOS 和 macOS 的预编译库版本。它可以在 Swift 项目中使用，无需从源代码编译库。例如：

// swift-tools-version: 5.10
// swift-tools-version 声明构建此包所需的最低 Swift 版本。

import PackageDescription

let package = Package(
    name: "MyLlamaPackage",
    targets: [
        .executableTarget(
            name: "MyLlamaPackage",
            dependencies: [
                "LlamaFramework"
            ]),
        .binaryTarget(
            name: "LlamaFramework",
            url: "https://github.com/ggml-org/llama.cpp/releases/download/b5046/llama-b5046-xcframework.zip",
            checksum: "c19be78b5f00d8d29a25da41042cb7afa094cbf6280a225abe614b03b20029ab"
        )
    ]
)
上述示例使用的是库的中间构建版本 `b5046`。您可以通过更改 URL 和校验和来修改为使用不同的版本。

## 命令行补全

某些环境支持命令行补全功能。

#### Bash 补全

```bash
$ build/bin/llama-cli --completion-bash > ~/.llama-completion.bash
$ source ~/.llama-completion.bash

您可以选择将此命令添加到 .bashrc 或 .bash_profile 中以便自动加载。例如：

$ echo "source ~/.llama-completion.bash" >> ~/.bashrc

依赖项​

  • yhirose/cpp-httplib - 单头文件 HTTP 服务器，由 llama-server 使用 - MIT 许可证
  • stb-image - 单头文件图像格式解码器，由多模态子系统使用 - 公共域
  • nlohmann/json - 单头文件 JSON 库，由各种工具/示例使用 - MIT 许可证
  • minja - C++ 中的最小 Jinja 解析器，由各种工具/示例使用 - MIT 许可证
  • linenoise.cpp - 提供类似 readline 的行编辑功能的 C++ 库，由 llama-run 使用 - BSD 2-Clause 许可证
  • curl - 客户端 URL 传输库，由各种工具/示例使用 - CURL 许可证
  • miniaudio.h - 单头文件音频格式解码器，由多模态子系统使用 - 公共域

Footnotes​

  1. tools/perplexity/README.md ↩

  2. https://huggingface.co/docs/transformers/perplexity ↩

  3. RamaLama ↩