I Don't File Anything. So I Built an Agent to Do It, and It Runs a 35B Model on My Mac.

Years ago I made a decision: I don't work for my computer, it works for me. I have one folder. Everything I own lives inside it. I do not sort documents into neat trees of subfolders, because that is work, and search already does that job better than I ever will. My email is the same. One inbox, no folders, and I search when I need something.

The only time I touch any of it is on long flights, when I'm bored and have nothing better to do. I delete duplicate files, and I sort the handful of files I actually use on a regular basis. That is the whole system. It is maintenance, not organization.

At some point I decided that even that small amount of work was something I should not be doing by hand. So I built an agent to do it for me. This is the story of what happened to that agent, and how a single paper took it from a small model that barely kept up to a 35-billion-parameter model running on my own machine.

The First Version: Ollama and a 7B Model

The agent itself is simple. It walks my folder, looks at each file, and decides where it belongs and whether it is a duplicate. The judgment call, "what is this file and what should happen to it," is the part that needs a language model.

I started with Ollama serving Qwen 2.5 7B on my Mac, a MacBook Pro (M2 Pro, 32GB). Ollama ran on localhost:11434, and the agent called it through the OpenAI-compatible API. It worked.

But the model was the bottleneck. Seven billion parameters is enough for easy cases and not much more. It reasoned poorly about ambiguous files, it was weak on my Hebrew filenames, and when it wasn't sure it dumped things into "Other," which is exactly the pile I was trying to get rid of. The agent was only ever going to be as smart as the model behind it, and the model was not smart enough.

The Paper That Changed the Math

Then a link landed in my WhatsApp: arxiv.org/abs/2609.18063↗, "The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction."

The claim sounded impossible. Run a 35-billion-parameter model on consumer hardware using only about 3GB of active memory, at more than 20 tokens per second. I run local models all the time and that number did not fit anything I knew.

How Edge0 Works

Large models choke on a normal machine because of three things at once, not one: the model's architecture, the storage hierarchy, and the hardware. A 35B Mixture-of-Experts model activates only about 3B parameters per token, so the compute is genuinely cheap. The expensive part is memory. You still have to keep all 35B parameters somewhere you can reach fast, and at 4-bit that is roughly 19.5GB, almost the entire memory of a 24GB machine, with nothing left for the actual work. Edge0's move is to stop treating RAM as the only fast tier and stream the rest from the SSD in time, on hardware built to move it without copying.

It gets there with three ideas working together.

SSD expert streaming. The expert weights live on the SSD, not in RAM. They are memory-mapped and read on demand, and only the active experts, about 3B out of 35B, are loaded at any moment. Your memory footprint is bounded by what is active, not by the size of the model.

A prerouter. This is the clever part. A small prediction head guesses which experts the next token will need, one step ahead, so the system can pull those weights off the SSD while the GPU is still busy with the current token. And it does not just inform the loading, it replaces the router entirely. What it predicts is what gets used. Prediction becomes routing, so nothing is fetched and wasted.

A recovery LoRA. A small adapter that restores the quality lost to 4-bit quantization and the router replacement. It has to be served unmerged, because folding it back into the quantized weights and re-quantizing destroys almost all of its effect.

The Migration: Twenty Minutes, No Code Changes

Moving my agent from Ollama and Qwen 7B to MLX and Edge0 35B took about twenty minutes, and I did not touch the agent's logic at all.

# Install MLX LM
python3 -m pip install mlx-lm --break-system-packages

# Download the model (~19.5GB, one time)
hf download Edge0/Edge0-35B-A3B-preview --local-dir ~/Edge0-35b

# Start an OpenAI-compatible server
mlx_lm.server --model ~/Edge0-35b --port 8080

The only change to the agent was pointing it at http://localhost:8080/v1 instead of http://localhost:11434/v1. Same OpenAI-compatible protocol, different engine underneath. That is the quiet advantage of this whole ecosystem: Ollama, MLX, vLLM, they all speak the same API, so swapping the brain of your agent is a one-line change.

There was one gotcha. Edge0 is a thinking model. It writes its chain of reasoning into a reasoning field before it produces the real answer in content. My default token limit was too low, so it ran out of room mid-thought and never produced output. Raising max_tokens to 4096 fixed it. If you point an agent at a reasoning model, give it headroom to finish thinking and still answer.

The Numbers

Qwen 2.5 7B on Ollama: 7B parameters, fully loaded in RAM, around 30 to 40 tokens per second, about 5GB of memory, and weak reasoning on multilingual filenames.

Edge0 35B on MLX: 35B parameters streamed from the SSD, 63.7 tokens per second measured, 19.6GB peak memory but only about 3GB active at any moment, and classification that is in a different league because the model actually understands context.

Sit with that for a second. A model five times larger, running close to twice as fast, using comparable active memory. It is not magic, it is architecture. MoE sparsity means only 3B parameters fire per token, and SSD streaming means the other 32B sit on disk instead of crowding my RAM.

On my machine the peak was higher than the paper's 3GB, closer to 19.6GB, because with 32GB to spare MLX keeps more of the experts resident in memory instead of streaming every one from disk. That is why I get well above the paper's 20 tokens per second. Apple Silicon's unified memory, where the CPU and GPU share one pool with no copying between them, is what makes this practical.

Why This Matters

The received wisdom falls apart once you stop staring at a single number. It was never only about how much RAM you have. It is about how three things line up: the model's sparsity, the bandwidth of your SSD, and hardware designed to feed the GPU without copying.

A modern Mac's NVMe SSD reads at 5 to 7 GB per second. If you can predict what you will need one token ahead, you can stream weights off that disk fast enough to keep the GPU busy, and the disk stops being a place you load from once and becomes part of the live memory path.

This is not a demo. It is an open framework with published checkpoints, and it is running on my MacBook Pro right now, classifying thousands of files with a 35B model.

Which brings me back to where I started. The whole point was to stop doing the boring maintenance myself. Now the agent does it, and it does it with a model I was told could not run on a machine like mine. I still keep one folder. I just have a much smarter assistant keeping it tidy.

Takeaways

MoE plus SSD streaming makes large models viable on consumer hardware today, not someday. The prerouter is the enabling trick: predict routing one token ahead and use the prediction as the routing itself. MLX on Apple Silicon is the runtime that makes it practical, because unified memory removes the copy overhead between CPU and GPU. The OpenAI-compatible API means switching engines is a one-line change. And if you build an agent on a thinking model, give it enough max_tokens to finish reasoning and still answer.

The paper: arxiv.org/abs/2609.18063↗ The model: huggingface.co/Edge0/Edge0-35B-A3B-preview↗