Transformers
2 articles in the last 30 days.
-
180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
The author compares how llama.cpp, vLLM and Transformers place and move MoE experts when a model doesn't fit in memory. Streaming experts from SSD via Strata gets within 6% of Strata's token generation speed.
-
Transformers Model Integration Contracts
A July 2026 change let UMT5's decoder attend to future tokens during training, and the bug stayed in main for almost two months. The post proposes transformers-mic, which pairs shrunk fixture models with recorded baselines to catch such bugs on PRs.