Strata
1 article in the last 30 days.
-
180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
The author compares how llama.cpp, vLLM and Transformers place and move MoE experts when a model doesn't fit in memory. Streaming experts from SSD via Strata gets within 6% of Strata's token generation speed.