llama.cpp
3 articles in the last 30 days.
-
180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
The author compares how llama.cpp, vLLM and Transformers place and move MoE experts when a model doesn't fit in memory. Streaming experts from SSD via Strata gets within 6% of Strata's token generation speed.
-
The No AI Slop Manifesto
llama.cpp and ggml reached 1.0 and removed 35 deprecated functions, and SGLang also announced 1.0. The No AI Slop manifesto from AIE CODE reached the Hacker News front page.
-
llama.cpp v1.0.0: A stronger foundation for the local AI ecosystem
llama.cpp and ggml reach v1.0.0 and will signal API and ABI breaks with major-version bumps. Hardware vendors are adding self-hosted CI runners to support the growing set of downstream users.