How I Built a Private Local AI Workspace

I classify the task before I choose a model

I write three labels before I start. The first describes the data: private, review before sharing, or public. The second describes the task: retrieval, drafting, coding, classification, or open research. The third describes the allowed output channels. A private note search and a public web research task do not belong inside the same default environment.

Then I define success with a few representative prompts. The model needs to retrieve the right note, follow the project format, use the required tool, and finish within a useful time. This test set stops me from choosing a model because it looked good in an unrelated benchmark.

I also decide what the agent should never see. Credentials stay outside prompts and memory. A project agent receives its project directory, not the whole archive. A research agent receives public context and a network connection, not private notes.

I choose the smallest model that completes the loop

I test the exact model file and quantization on the machine that will run it. I record task quality, memory use, time to first token, prompt processing, generation speed, context slowdown, heat, and power. Then I run the full Hermes task with tools. Tokens per second says little if the model chooses the wrong file, loops on a command, or loses the instruction after a large tool result.

This exposed a common mistake in my early setup. A larger model produced a better single answer but slowed each tool handoff enough to make the agent frustrating. A smaller quantized model completed the bounded coding and note tasks faster because the whole loop stayed responsive.

I serve the chosen model through llama.cpp and connect Hermes to the local endpoint. I pin the model, context, sampling settings, prompt template, and server version as one configuration. A model upgrade becomes a tested change, not an invisible replacement.

“The difference is that this is only needed for the code that enforces the isolation, not for all of the code.”

Joanna Rutkowska, Qubes OS founder↗

I place each workflow in the right compartment

Qubes OS lets me separate local inference and sensitive files from network research and disposable tasks. I do not treat the compartments as labels. I decide which files, clipboard actions, network routes, and tool processes each workflow needs.

A private analysis task runs with no network and a narrow file mount. Public research runs in a networked compartment with no access to the private workspace. When I need a public source inside the private task, I review and transfer the file. I do not give the sensitive agent open browsing for convenience.

Tool permissions follow the same structure. Read access does not imply write access. Terminal access does not imply every command. Discord stays outside private work. This makes the agent useful inside a clear boundary and keeps failures easier to contain. My accountable agent workflow uses the same ownership rule.

I make memory useful and intentionally limited

Hermes becomes more useful when it remembers stable project facts, decisions, and working preferences. It also becomes harder to reason about when memory stores every conversation. I define the kinds of facts that deserve persistence and keep task scratch work out.

Each memory needs a source, scope, and reason to stay. Project memory stays with that project. General preferences stay separate. Credentials, raw client material, temporary paths, and unverified conclusions never enter long term memory. I review and remove stale entries instead of letting them quietly steer later work.

I audit the less obvious memory too. Shell history, downloaded files, model server logs, crash reports, caches, temporary exports, and chat bridges can outlive a session. A private workflow is only as private as its most persistent copy.

I test the boundary by trying to break it

I give the agent prompts that tempt it to cross the boundary. I ask it to inspect a neighboring directory, browse when the network should be absent, send a private result through Discord, retrieve a deleted memory, and expose secrets through an error. The right outcome is a clear refusal or a blocked tool call.

I also test ordinary failures. I restart the model server during a task, fill the context with tool output, change the mounted folder, update llama.cpp, and reconnect Hermes. I check whether the agent recovers without widening access or silently losing the task contract.

Local models still carry an operating cost. I track model files and licenses, pin known configurations, test upgrades, back up the memory I intend to keep, and delete the rest. Maintenance is part of the privacy promise because an abandoned stack collects outdated software and forgotten data.

• Classify the data, task, and allowed output channel.
• Benchmark the full tool loop on your own hardware.
• Give each compartment only the files and network it needs.
• Keep long term memory small, sourced, and reviewable.
• Test whether the agent can cross the boundary.

I keep a deliberate path to cloud models

Some tasks need more capability or context than my local setup delivers efficiently. I do not force them through a local model to protect the idea of a private stack. I first check whether the data policy allows an external service. Then I remove private details or create a separate public question I can review.

The cloud path uses its own workspace and tools. A sanitized prompt never gains access to the private source just because it came from the same project. If the task cannot be separated safely, it stays local or remains undone.

This hybrid method gives me a useful choice. I keep sensitive retrieval, notes, code, and bounded agent work within the local boundary. I use stronger hosted capability only when the content and outcome justify it. My coding agent method gives both paths the same plan, tests, handoffs, and human validation.