Split Your AI Stack: Offline Models for the Work You Cannot Paste Into a Chatbot

It is 10 pm in an office. A consultant wants to summarize today's client meeting before going home, and her cursor is parked in the ChatGPT input box, not moving. The text she is about to paste has client names, contract amounts, and personal details in it.

That pause is the subject of a quietly useful article by Mina, a Japanese data analyst who publishes as planetdive. She kept hearing the same story from lawyers, tax accountants, counselors, and HR people: the moment a task touches someone else's confidential information, cloud AI stops being usable, and a draft that should take five minutes takes thirty, done by hand.

Her answer was not a better cloud model. It was a split stack: cloud AI for everything public, and a small local model for the confidential slice only.

The pause is not about model quality

The instinct worth examining is the one that says "AI is getting smarter, so this problem will solve itself." It will not, because the two concerns sit on different axes. Model capability decides how good the summary is. Whether you are allowed to send a client's contract terms to an external server is a governance and ethics constraint that no amount of capability removes. As corporate governance tightens, the "safe to use" test gets stricter, not looser.

This is why the frozen cursor at 10 pm will exist in three years exactly as it exists today. What changes is which tools make the split easy.

The mistake almost everyone makes first

Mina's first attempt is documented with unusual honesty. She downloaded a 70B-parameter model on the assumption that bigger means smarter. The file was over 40GB. The download took 42 minutes on the office line. When she ran it, the process died instantly, because her laptop has 16GB of RAM and the model needed more. She fell back to a 7B model, discovered its Japanese was clumsy, swapped models three times total, and burned close to an hour on selection alone.

The cause was not Ollama being hard to use. It was choosing on a single axis, parameter count, while ignoring the two constraints that actually bind: the memory your machine has, and whether the model is good at your specific task. Anyone who has picked an image model purely by leaderboard position will recognize the shape of that hour.

Pick by memory first, task second

The practical selection order for a local summarization or drafting assistant:

1. Memory before merit. A model that does not fit your RAM does not run, and a model that barely fits will be painfully slow. File size is only the floor: runtime adds the context window and KV-cache, backend overhead, and whatever else the machine is running, so a model can pass the file-size check and still swap hard. On a 16GB machine, 7B-class models are the sane default; 13B at a stretch with quantization and a safety margin. Check the quantized file size, then benchmark one real document at a representative context length before you commit.
2. Task fit over benchmark rank. For summarizing documents in a non-English language, raw parameter count is a poor predictor. Test three or four small models on one real, already-public document of yours and compare outputs directly. Fifteen minutes of that beats an hour of spec-sheet swapping.
3. Speed is a feature. A local 7B model summarizes slower than a frontier cloud model. If you process hundreds of documents a month, local is the wrong tool. For the confidential ten documents a week, throughput does not matter.

What belongs local, and what does not

The article's most transferable idea is the routing rule, not the software. Cloud AI keeps what you are actually allowed to send: research, brainstorming, code without secrets, drafting in general. "Public" alone is not authorization — routing client material cloud-side assumes you have checked the provider's data-handling terms, retention, training on inputs, jurisdiction, and have the consent or organizational policy to send it. Local AI gets only the slice where the constraint is real: contracts with counterparty names, counseling notes, evaluation memos with real employee names, anything an NDA or professional ethics rule covers.

Two limits travel with the technique. Local small models are genuinely worse at nuanced summarization and complex instructions than frontier cloud models; you trade polish for containment. And if you cannot install software on a work-issued machine, the whole approach is closed to you before it starts.

There is a third limit Mina does not state but implies: segmentation only works if the boundary is honest. The moment you get lazy and paste a confidential clause into the cloud tab anyway, because it is just one clause, the split stops protecting anything.

Why this matters for image work, too

The same routing logic applies to visual generation, and it is where the parallel earns its keep. Reference photos of identifiable people, unreleased product renders under NDA, internal brand assets before launch. These are the image-side equivalents of the client record, and pasting them into a cloud generator carries the same category of exposure. For identifiable people the consent gate applies too: an identifiable face deserves the same provider-terms and consent check before it reaches a hosted model that a client record needs before it reaches a cloud chat. The reflex to check "where does this asset go" before "which model is best" is the transferable skill.

It also frames local vs cloud as an economic decision rather than a moral one. As we saw when local models stopped being a compromise, the practical question stopped being "can my machine run this" and became "what exactly am I giving up." For confidential work, the answer is often "polish," and that is a price many professionals will happily pay.

If your visual work has a confidential slice, client portraits, unreleased designs, the same split applies: keep the public ideation in a hosted tool with models worth the metadata, and keep the sensitive references on your side of the wall until they stop being sensitive. Phosphene sits on the hosted side of that line, which is exactly why the boundary should be drawn before you generate, not after.