The Hidden Taxes of Local AI
Part one was about what AI agents on my laptop can reach by default, and about putting a dev container around each project so that the things I don’t want touched simply aren’t there. This part is the invoice.
Short recap for anyone arriving here first. I’m not an engineer. I’ve spent the last eighteen months building real things with coding agents, on my own machine, for my own projects. I’ve watched a lot of YouTube and read a lot of guides on this, and almost all of them stop at the moment the setup starts working. The months afterwards - which parts break, what breaks them, what you quietly gave up - is the part I’ve never seen written down, so that’s what this is.
What the container gives me that nothing else does
Start with the upside, because it’s the reason I put up with everything below, and because it’s not the upside people advertise.
The container is how I get a detailed conversation history tied to a specific project. Every session inside a project gets exported and archived under that project’s name, so months later there is a body of material that is about this codebase - decisions, dead ends, the reasoning behind things I’d otherwise have to reconstruct.
The desktop apps don’t give me that. They give me my history: one long undifferentiated stream of everything I’ve ever discussed, in the order I discussed it. For a single project that’s fine. For work spread across five, the difference between “my history” and “this project’s history” is the entire game, and it’s the thing I’d least want to give back.
The invoice
The configuration breaks, periodically, and always at the wrong moment. Usually after an editor extension updates: a rebuild fails, a CLI vanishes from the path, a mount doesn’t come up, something that worked on Friday doesn’t on Monday. None of it is dramatic. All of it arrives while you were about to do something else.
Which is why I have an entire separate project whose only job is this setup. It holds the container templates, the setup script, the decisions and why each one was made, the operational notes, and a log of every failure and its fix. When something breaks I open that project, and the agent working there already knows the full history of my configuration rather than guessing from a stack trace. That sounds like a lot of ceremony for “a few config files” and it isn’t: the setup is a system, it drifts, and before I wrote it down I kept re-solving the same breakages from first principles, badly, at ten at night.
This is the memory idea from part three, applied to the thing that makes everything else run. It’s also, I think, the single most transferable piece of my setup for someone who isn’t an engineer: the environment is a project, so give it a project’s memory.
Rebuilds are rare, which is good and also a trap. My machine is almost always on, so containers keep running for weeks at a time. I rebuild only when something is genuinely broken, or when memory is exhausted. So configuration drift accumulates quietly in between, and the rebuild is the moment you discover all of it at once - usually the moment you least wanted a research project.
And the history you keep isn’t the history you can read. Mining preserves conversations verbatim, but into a database. It’s searchable; it isn’t readable. I can’t scroll it. In the desktop app I can open a conversation from last Tuesday and page through it like a document, skim the shape of it, spot the thing I half-remember by recognising it rather than by describing it. In my setup I have a store that will answer a question about last Tuesday if I phrase the question well.
Those are genuinely different capabilities and I picked the searchable one. Most days that’s the right trade. Some days I want to reread rather than retrieve, and I can’t, and I feel it.
I’d still choose the container. But “safe, and here’s what it costs you” is a more honest pitch than the one I keep being given.
The thing I found last week that changed my day
Both of these tools now do remote control of a running session, including sessions inside a container. Claude’s works for me: I can pick up a live session from my phone, answer a permission prompt, keep something moving while I’m away from the desk. After months of being physically tethered to the machine where the container runs, this is the largest quality-of-life improvement I’ve had in a while, and I found it more or less by accident.
Two caveats, since I’d rather nobody rebuild their week around this. It’s remote control, not a session that outlives you - the session still lives and dies with the container, so this doesn’t turn a laptop into a server. And the other agent has an equivalent that I haven’t got working: it keeps dropping. My best guess, from my own notes, is that its remote daemon expects the standalone installation and refuses the editor-extension binary my container actually uses, which would explain the behaviour exactly. Unconfirmed, and on my list.
The bridge that made the rest survivable
Containers can’t reach the laptop’s notification centre, so for a while I sat watching a terminal to catch permission prompts, which rather defeats the point of delegating anything. Now a small host-side bridge turns container events into native notifications: the agent asks for permission, my laptop pings, I answer, it continues.
It’s a small thing and it’s the reason the rest of this setup survived contact with a normal working week. Most of what makes a safe setup fail isn’t a flaw in the design, it’s ten seconds of friction repeated forty times a day until you quietly stop.
The rule I learned by installing it twice: exactly one hook layer per product. Both of these tools merge hook definitions from the user level and the project level, so if you configure both, every event fires twice. It’s harmless and maddening, and it took me longer than I’d like to admit to work out why my laptop was double-pinging.
The server move I parked on purpose
The plan was to move everything - containers, memory store, compute - onto a small server reachable from anywhere over a private mesh network. Mobile access is a hard requirement for me, which ruled out a few otherwise-good options. On paper it wins: more compute than a laptop, nothing exposed publicly, and the laptop stops running hot.
Two things made me park it.
My sizing comparison was wrong. I’d been comparing a candidate server against my whole laptop, until I noticed the container runtime on it was capped at 6 GB of RAM anyway. Against that real baseline - not the laptop’s specifications, the actual ceiling my containers ever see - a roughly $20/month box already exceeds what I use. Useful to establish before committing to something four times the price.
“Everything’s in git, so why isolate?” I made this argument to myself and it’s a category error. Git makes bad edits revertible. It does nothing about a key leaked from another project’s environment file, a mutation to a live database that isn’t version-controlled, or an outbound request that has already happened. Those are exactly the failures containment is for. And putting every project on one shared machine - with every credential, the database, and the deploy keys reachable from any single session - makes the blast radius larger than it is today, not smaller. If I move, the per-project containers move with me.
There’s also a piece I haven’t verified. Part of why I run so many containers isn’t security at all: my memory tooling infers the project name from the working directory, so one container per project became a convenient way to keep things separated. That’s a tooling quirk, not a requirement, and I don’t yet know whether the separation survives without the container boundary. Until I’ve tested it, I’m not making decisions on top of it.
What I’d want someone to take from this part
The container is not free. It costs a recurring maintenance tax, an occasional evening, and one capability I actually miss. It buys structural safety and per-project history, and for me that trade is clearly worth it - but I’d rather someone chose it knowing both columns.
And the maintenance tax is the part you can design away. Not by making the setup simpler, but by giving it a memory: write down what you configured and why, keep the log of what broke and what fixed it, and the next breakage costs twenty minutes instead of an evening.
Which is the subject of part three, where the same idea gets applied to the projects themselves - and where I got it wrong twice before getting it right.
← A non-engineer’s coding setup, part one — safety · A non-engineer’s coding setup, part three — memory →