Skip to content


All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.






  • fix(security): Remote Code Execution by Server-Side Template Injection in Model Metadata by @retr0reg in b454f40
  • fix(security): Update remaining jinja chat templates to use immutable sandbox by @CISC in #1441






  • fix: Ensure image renders before text in chat formats regardless of message content order by @abetlen in 3489ef0
  • fix(ci): Fix bug in use of upload-artifact failing to merge multiple artifacts into a single release by @abetlen in d03f15b



















  • fix: module 'llama_cpp.llama_cpp' has no attribute 'c_uint8' in Llama.save_state by @abetlen in db776a8
  • feat: Auto detect Mixtral's slightly different format by @lukestanley in #1214


  • feat: Update llama.cpp to ggerganov/llama.cpp@15499eb
  • feat: Add Google's Gemma formatting via chat_format="gemma" by @alvarobartt in #1210
  • feat: support minItems/maxItems in JSON grammar converter by @nopperl in 3921e10
  • fix: Update from_pretrained defaults to match hf_hub_download and pull to local cache folder by @abetlen in e6d6260
  • fix: Raise exceptions when llama model or context fails to load by @abetlen in dd22010
  • docs: Update to fix pip install llama cpp server by @audip in #1187
























  • feat(server): Multi model support by @D4ve-R in #931
  • feat(server): Support none defaulting to infinity for completions by @swg in #111
  • feat(server): Implement openai api compatible authentication by @docmeth2 in #1010
  • fix: text_offset of multi-token characters by @twaka in #1037
  • fix: ctypes bindings for kv override by @phiharri in #1011
  • fix: ctypes definitions of llama_kv_cache_view_update and llama_kv_cache_view_free. by @e-c-d in #1028





  • Update llama.cpp to ggerganov/llama.cpp@64e64aa
  • Make building llava optional by setting CMAKE_ARGS="-DLLAVA_BUILD=OFF" and using LLAVA_CPP_LIB to specify alternative path to shared library by @abetlen in e3941d9






  • Update llama.cpp to ggerganov/llama.cp@a75fa57
  • Add set_seed to Llama class by @abetlen in fd41ed3
  • Fix server doc arguments by @kjunggithub in #892
  • Fix response_format handler in llava chat handler by @abetlen in b62c449
  • Fix default max_tokens, chat completion is now unlimited (to context length) and completion is 16 tokens to match OpenAI defaults by @abetlen in e7962d2
  • Fix json_schema_to_gbnf helper so that it takes a json schema string as input instead by @abetlen in faeae18
  • Add support for $ref and $def in json_schema_to_gbnf to handle more complex function schemas by @abetlen in 770df34
  • Update functionary chat handler for new OpenAI api by abetlen in 1b376c6
  • Fix add default stop sequence to chatml chat format by @abetlen in b84d76a
  • Fix sampling bug when logits_all=False by @abetlen in 6f0b0b1






  • Fix bug in llama_model_params object has no attribute logits_all by @abetlen in d696251


  • Fix bug 'llama_model_params' object has no attribute 'embedding' by @abetlen in 42bb721


  • Fix critical bug in pip installation of v0.2.8 due to .git directory in ac853e0






  • Add NUMA support. NOTE low level api users must call llama_backend_init at the start of their programs by abetlen in f4090a0
  • Fix tensor_split server cli argument by @abetlen in c4c440b
  • Made all Llama init parameters into keyword-only parameters by @abetlen in c8f9b8a
  • Added server params for low_vram, main_gpu, lora_base, and lora_path by @abetlen in 2920c4b
  • Removed server params for rms_norm_eps and n_gqa by @abetlen in 2920c4b
  • Fix boolean cli options by @abetlen in c999325 and 0449d29
  • Silence Pydantic Settings warnings about model_alias setting by @earonesty in #705



  • Fix bug in pip install of v0.2.1 due to scikit-build-core removing all .metal files in the source distribution (see #701)


  • Fix bug in pip install of v0.2.0 due to .git folder being included in the source distribution (see #701)


  • Migrated to scikit-build-core build system by @abetlen in #499
  • Use numpy views for LogitsProcessor and StoppingCriteria instead of python lists by @abetlen in #499
  • Drop support for end-of-life Python3.7 by @abetlen in #499
  • Convert low level llama.cpp constants to use basic python types instead of ctypes types by @abetlen in #499



  • Update llama.cpp


  • Update llama.cpp


  • Update llama.cpp


  • Update llama.cpp


  • Update llama.cpp


  • GGUF Support (breaking change requiring new model format)


  • Grammar based sampling via LlamaGrammar which can be passed to completions
  • Make n_gpu_layers == -1 offload all layers


  • (llama.cpp) Update llama.cpp add support for LLaMa 2 70B
  • (server) Add temporary n_gqa and rms_norm_eps parameters required for LLaMa 2 70B


  • (llama.cpp) Update llama.cpp add support for LLaMa 2 70B


  • Update llama.cpp


  • (server) OpenAI style error responses


  • (server) Add rope parameters to server settings


  • (llama.cpp) Update llama.cpp added custom_rope for extended context lengths


  • (llama.cpp) Update llama.cpp

  • (server) Fix several pydantic v2 migration bugs


  • (Llama.create_completion) Revert change so that max_tokens is not truncated to context_size in create_completion
  • (server) Fixed changed settings field names from pydantic v2 migration


  • (server) Streaming requests can are now interrupted pre-maturely when a concurrent request is made. Can be controlled with the interrupt_requests setting.
  • (server) Moved to fastapi v0.100.0 and pydantic v2
  • (docker) Added a new "simple" image that builds llama.cpp from source when started.
  • (server) performance improvements by avoiding unnecessary memory allocations during sampling


  • (llama.cpp) Update llama.cpp


  • Fix performance bug in Llama model by pre-allocating memory tokens and logits.
  • Fix bug in Llama model where the model was not free'd after use.


  • (llama.cpp) New model API

  • Performance issue during eval caused by looped np.concatenate call

  • State pickling issue when saving cache to disk


  • (llama.cpp) Fix struct misalignment bug


  • (llama.cpp) Update llama.cpp
  • Fix docs for seed. Set -1 for random.


  • (llama.cpp) Add full gpu utilisation in CUDA
  • (llama.cpp) Add get_vocab
  • (llama.cpp) Add low_vram parameter
  • (server) Add logit_bias parameter


  • Metal support working
  • Cache re-enabled


  • Fix broken pip installation


NOTE: This release was deleted due to a bug with the packaging system that caused pip installations to fail.

  • Truncate max_tokens in create_completion so requested tokens doesn't exceed context size.
  • Temporarily disable cache for completion requests


  • (llama.cpp) k-quants support
  • (server) mirostat sampling parameters to server
  • Support both .so and .dylib for libllama on MacOS


  • (llama.cpp) Metal Silicon support


  • (llama.cpp) OpenLlama 3B support


  • (misc) Added first version of the changelog
  • (server) Use async routes
  • (python-api) Use numpy for internal buffers to reduce memory usage and improve performance.
  • (python-api) Performance bug in stop sequence check slowing down streaming.