Local AI Digest: Multi-User Scheduling, Small Model Surprises, and Long-Context Unlocks

News

Microsoft's Phi-3 mini hits Hugging Face with optimized GGUF quantized versions

Phi-3 mini in 3.8B punches near GPT-3.5 on reasoning benchmarks and runs comfortably on consumer GPUs and even phones. Microsoft is making the strongest case yet for small models at the edge.

LocalAI adds experimental tools API matching OpenAI function-calling spec

LocalAI adding OpenAI-compatible tools API means you can drop in local models without rewriting your agent code. This lowers the switching cost from hosted APIs to local inference dramatically.

Tools

vLLM 0.5.0 ships with experimental long-context RoPE scaling for open models

vLLM's RoPE scaling trick means you can feed long documents into locally served models without retraining. For anyone building local RAG pipelines, this is a practical unlock.

Apple MLX examples expand with Stable Diffusion FP16 on Apple Silicon

Apple's latest MLX additions let you run competitive image generation entirely on M-series chips with near-zero latency. The gap between needing a cloud GPU and your Macbook for quality image gen is closing fast.

Analysis

Mozilla's llamafile closes performance gap against Ollama on laptop benchmarks

The benchmarks show a llamafile running Gemma 2 9B outperforms Ollama on the same hardware by 15 percent on long-context tasks. If you care about single-binary portability and raw throughput, llamafile deserves a second look.