Ollama as the practical baseline
Ollama was the easiest way to get useful local Gemma 4 runs on the Mac Mini. It worked as a clean baseline for smoke testing and comparison, especially for smaller-model and fallback use.
Local models and benchmarking
What I learned running local models on a Mac Mini
I used an Apple Silicon Mac Mini as a practical local-model lab: comparing Ollama, raw MLX research, and oMLX serving, then building a repeatable coding benchmark to measure speed, reliability, and the real memory ceilings that shape what local AI can actually do.
Why this work mattered
Local AI stops being abstract when the machine pushes back
I wanted to understand local models as working infrastructure, not as a vague privacy talking point. That meant testing what actually ran on a Mac Mini, how cleanly it integrated with Hermes, and where the limits showed up in speed, memory pressure, and repeatability.
Researching MLX, then moving to oMLX
Raw MLX was promising but awkward for the specific Gemma 4 checkpoint shape I tested. oMLX mattered because it exposed an OpenAI-compatible local API, which made the local serving path much more usable through Hermes.
Mac Mini baseline
The machine and setup I actually tested
The public value of this page is not “local AI is magic.” It is showing what was really tested, on what hardware, and why the serving path changed over time.
Apple Silicon Mac Mini
The test machine is an Apple M4 Mac Mini with 24GB unified memory and a 256GB SSD. That hardware was enough for meaningful local experiments, but still small enough that memory ceilings and serving overhead mattered.
Ollama → MLX research → oMLX
Ollama handled the first useful runs. Raw MLX stayed in the research lane. oMLX became the first 12B serving path that felt clean enough to use through Hermes as real workflow infrastructure.
| Path | What I verified | What I learned |
|---|---|---|
| Ollama | gemma4:e4b and gemma4:12b-mlx both ran successfully |
Great baseline and fallback, but the 12B Ollama path was substantially slower on the Mac Mini. |
| Raw MLX / MLX-LM | Installed and tested directly against a Gemma 4 checkpoint | Promising for Apple Silicon research, but not clean enough yet for the daily-serving lane I needed. |
| oMLX | Served a 12B Gemma 4 model through an OpenAI-compatible local API and worked through Hermes | The best practical integration shape for this Mac Mini local serving setup so far. |
Benchmark results
A repeatable coding benchmark exposed the real tradeoffs
I built a local benchmark around a realistic Python scripting task and graded the generated code with a deterministic four-test pytest suite. That made it possible to compare local models on something more meaningful than a single subjective prompt.
| Model | Best observed result | Best observed time | What the result really means |
|---|---|---|---|
Qwen3.6-35B-A3B-RotorQuant-MLX-3bit |
4/4 tests | 72.33s | First perfect sample on the canonical benchmark, though earlier runs still showed meaningful variance. |
gemma-4-12B-it-8bit |
3/4 tests | 239.09s | Much slower, but more conventional and easier to reason about; failed CSV complex-value encoding. |
Qwen3.5-9B-Fable-5-v1-oQ8 |
2/4 tests | 202.04s | Ran cleanly under oMLX, but a model-specific precise-coding profile made the result worse: 1/4 tests in 220.39s. |
Ornith-1.0-9B-4bit |
2/4 tests | 147.88s | Faster than Gemma but materially weaker on parsing correctness and CSV behavior. |
Ternary-Bonsai-27B-mlx-2bit |
1/4 tests | 167.67s | Compact at 8.92GB and operational in oMLX, but weaker than Qwen3.6 35B on the canonical coding task. |
Qwen3.6-27B-MLX-4bit |
No completed run | n/a | Could not complete under the earlier default Mac Mini Metal memory ceiling; it has not been rerun under the raised limit. |
The highest score still needed context
Qwen3.6 35B produced the first 4/4 sample, but earlier runs ranged from 3/4 to invalid Python. The result established the highest observed ceiling, not proof of perfect run-to-run reliability.
Correctness needed to be testable
The task required parsing, CLI behavior, serialization, and error handling. That made failures legible: not just “this answer felt weak,” but exactly where the generated code broke down.
Task-specific model fit
Bonsai was stronger for chat, but not for coding
I tested Ternary Bonsai 27B and a 12B Gemma 4 variant on the same ten conversational tasks, covering explanation, rewriting, troubleshooting, safety questions, factual caution, creative language, and basic reasoning. The comparison reinforced why coding and conversation need separate evaluations.
| Model and configuration | Completed result | Wall time | Model footprint |
|---|---|---|---|
Ternary-Bonsai-27B-mlx-2bit, thinking disabled |
Completed all 10 tasks; both math problems correct | 88.26s | 8.92GB |
gemma-4-12b-coder-fable5-composer2.5-4bit |
Completed all 10 tasks; failed the chickens-and-rabbits problem | 99.34s | 7.03GB |
Bonsai became the preferred local chat model
With thinking disabled, Bonsai completed the full prompt 11.2% faster than Gemma, answered more concisely, and correctly solved both math tasks. It used more memory, but its 8.92GB footprint still fit comfortably.
The better chat result came with caveats
Bonsai's default thinking mode exposed a long scratchpad and exhausted a 2,200-token limit before the final answer. It also passed only 1/4 coding tests, so I kept Qwen3.6 35B as the local coding leader rather than treating one model as the best choice for every task.
Operational lessons
The bottleneck was often systems behavior, not the model itself
The most transferable lesson from this Mac Mini setup was that local AI work is constrained systems engineering. The machine's memory behavior, serving process, and benchmark discipline mattered as much as the model name.
What this taught me about practical AI work
Running local models on a Mac Mini made the tradeoffs concrete. Privacy and cost are real reasons to care about local serving, but the more interesting lesson was architectural: the useful questions are about serving shape, repeatability, memory ceilings, routing, and when a local model is actually good enough for a specific job. Coding correctness, conversational usefulness, and reasoning reliability turned out to be separate qualities—not one general model score.
That is the angle I find professionally useful. Local AI is not just a novelty demo. It is a systems problem with measurable tradeoffs, and solving it well looks a lot like other kinds of software and infrastructure work.