Ollama as the practical baseline
Ollama was the easiest way to get useful local Gemma 4 runs on the Mac Mini. It worked as a clean baseline for smoke testing and comparison, especially for smaller-model and fallback use.
Local models and benchmarking
What I learned running local models on Atlas
I used an Apple Silicon Mac Mini as a practical local-model lab: comparing Ollama, raw MLX research, and oMLX serving, then building a repeatable coding benchmark to measure speed, reliability, and the real memory ceilings that shape what local AI can actually do.
Why this work mattered
Local AI stops being abstract when the machine pushes back
I wanted to understand local models as working infrastructure, not as a vague privacy talking point. That meant testing what actually ran on Atlas, how cleanly it integrated with Hermes, and where the limits showed up in speed, memory pressure, and repeatability.
Researching MLX, then moving to oMLX
Raw MLX was promising but awkward for the specific Gemma 4 checkpoint shape I tested. oMLX mattered because it exposed an OpenAI-compatible local API, which made the local serving path much more usable through Hermes.
Atlas baseline
The machine and setup I actually tested
The public value of this page is not “local AI is magic.” It is showing what was really tested, on what hardware, and why the serving path changed over time.
Apple Silicon Mac Mini
Atlas is an Apple M4 Mac Mini with 24GB unified memory and a 256GB SSD. That hardware was enough for meaningful local experiments, but still small enough that memory ceilings and serving overhead mattered.
Ollama → MLX research → oMLX
Ollama handled the first useful runs. Raw MLX stayed in the research lane. oMLX became the first 12B serving path that felt clean enough to use through Hermes as real workflow infrastructure.
| Path | What I verified | What I learned |
|---|---|---|
| Ollama | gemma4:e4b and gemma4:12b-mlx both ran successfully |
Great baseline and fallback, but the 12B Ollama path was substantially slower on Atlas. |
| Raw MLX / MLX-LM | Installed and tested directly against a Gemma 4 checkpoint | Promising for Apple Silicon research, but not clean enough yet for the daily-serving lane I needed. |
| oMLX | Served a 12B Gemma 4 model through an OpenAI-compatible local API and worked through Hermes | The best practical integration shape for Atlas local serving so far. |
Benchmark results
A repeatable coding benchmark exposed the real tradeoffs
I built a local benchmark around a realistic Python scripting task and graded the generated code with a deterministic four-test pytest suite. That made it possible to compare local models on something more meaningful than a single subjective prompt.
| Model | Best observed result | Best observed time | What the result really means |
|---|---|---|---|
Qwen3.6-35B-A3B-RotorQuant-MLX-3bit |
3/4 tests | 62.03s | Fastest strong run I observed, but another run the same day failed completely with invalid Python. |
gemma-4-12B-it-8bit |
3/4 tests | 239.09s | Much slower, but more conventional and easier to reason about; failed CSV complex-value encoding. |
Ornith-1.0-9B-4bit |
2/4 tests | 147.88s | Faster than Gemma but materially weaker on parsing correctness and CSV behavior. |
Qwen3.6-27B-MLX-4bit |
0/4 tests | n/a | Could not be benchmarked fairly under the current Atlas Metal memory ceiling. |
The fastest result was not automatically the best result
Qwen35 produced the fastest strong run, but it was not stable enough yet to call the benchmark solved. That was one of the most useful lessons: local model evaluation needs repeated runs, not cherry-picked wins.
Correctness needed to be testable
The task required parsing, CLI behavior, serialization, and error handling. That made failures legible: not just “this answer felt weak,” but exactly where the generated code broke down.
Operational lessons
The bottleneck was often systems behavior, not the model itself
The most transferable lesson from Atlas was that local AI work is constrained systems engineering. The machine's memory behavior, serving process, and benchmark discipline mattered as much as the model name.
What this taught me about practical AI work
Running local models on Atlas made the tradeoffs concrete. Privacy and cost are real reasons to care about local serving, but the more interesting lesson was architectural: the useful questions are about serving shape, repeatability, memory ceilings, routing, and when a local model is actually good enough for a specific job.
That is the angle I find professionally useful. Local AI is not just a novelty demo. It is a systems problem with measurable tradeoffs, and solving it well looks a lot like other kinds of software and infrastructure work.