Stop Explaining Yourself Every Day
Part one was about what AI agents on my laptop can reach and the container that stops them reaching further. Part two was about what that container costs me. This part is about the concern that actually costs money and hours, and the one I’d fix first if I were starting over: I explain the same project every single day.
Short recap for anyone arriving here first. I’m not an engineer. I’ve spent the last eighteen months building real things with coding agents, and the technical setup around them is the part nobody hands you - every product documents its own settings, and nobody documents what it’s like to live with the combination for months. This is a home setup, for personal projects, on one laptop.
Alongside these three parts there are two working notes, published unfinished because both are live experiments: the skill library I run and my attempt to build an assistant that holds every project at once. Neither has concluded, and I’d rather show the state of them than wait.
The problem, stated properly
On a large project, the context is the work. Which decisions were already made and why. What we tried in March that didn’t work. Which of the four things called “pipeline” I mean this time. Which approach I abandoned, and whether I abandoned it for a good reason.
Retyping that into every new session is expensive in tokens and considerably more expensive in my own time. And there’s a failure mode worse than the cost: I’d re-explain it slightly differently each time. So the agent would act - confidently, always confidently - on a version of my project that had never actually existed. Not hallucination. Just my own inconsistent summaries, faithfully executed.
I got the fix wrong twice before getting it right, and both wrong turns are more instructive than the answer.
Attempt one: archive everything
The obvious fix for re-explaining yourself is to record everything. So I did: every conversation exported and indexed into a local vector store (MemPalace, which runs entirely on the machine - no API key, nothing leaving the laptop), with the agent searching it at the start of each session.
It doesn’t work, and it fails in a way that’s easy to misread as a tuning problem. Every session started by burning tokens on retrieval before any actual work began. Worse, what came back was chatter - half-formed ideas, an approach I’d abandoned two weeks later, my own confused first pass at a problem I’d since solved. The agent couldn’t tell which was current, and neither could the retrieval, because relevance and currency are different things and a vector search only knows about the first.
I had replaced re-explaining with re-reading. That’s worse. When I explain something myself, at least I know it’s true today.
The lesson, stated as plainly as I can: a conversation archive is evidence, not working memory. It’s genuinely valuable - for “when did we decide this?”, for reconstructing something I’ve lost, for forensics. It is not a substitute for knowing where things stand.
Attempt two: a wiki the agent maintains
What actually works is a small, deliberately compiled set of pages that the agent reads at the start of every session: current state, architecture, decisions with their rationale, workflows, operations, backlog, lessons learned. Plus an index page listing what exists, so it can read selectively rather than swallowing everything.
The pattern has a name - Andrej Karpathy published an “LLM Wiki” spec in April this year describing more or less exactly this: raw sources stay immutable, a wiki layer gets written and maintained by the model on top of them, and a schema file defines the conventions. His framing is that knowledge should be compiled over time, like code, rather than retrieved on demand. I’d been fumbling toward this and getting the details wrong; reading his version is what made me stop treating retrieval as the goal.
What I added on top, for what it’s worth: written workflow contracts, and a log. The log turned out to matter more than expected - it’s the difference between “the wiki says X” and “the wiki says X, changed on this date, because of this”.
The distinction that made everything click:
| The vector store | The wiki | |
|---|---|---|
| Where it comes from | Automatically, from files and transcripts | Written deliberately, by me or the agent |
| Question it answers | ”Where was this mentioned?" | "How do things stand right now?” |
| How it enters context | Never whole - only search results | Whole, every session |
Once those are separate, the token problem dissolves. The thing that goes into context is small and curated. The thing that’s huge is searched, rarely, on purpose.
One more piece: I keep formal process diagrams, but agents don’t execute from them. They execute from a written contract - steps, gates, branches, who decides what, where to stop and ask. Models lose things in diagrams: the approval gate disappears, two branches merge into one, the distinction between “the orchestrator decides” and “the sub-task does” evaporates. The diagram is for me. The contract is for the agent.
There’s a version of this that isn’t about code at all, and it’s the one I’d point a non-engineer to first. The same structure works for the setup itself - I keep a project whose only content is how my environment is configured and why, and when something breaks, the agent working in it already knows the history instead of guessing. Any domain where you’d otherwise re-explain yourself qualifies.
Two debugging stories
The search that only worked in English. Retrieval quality was uneven in a way I couldn’t pin down, so I ran a controlled test: same index, same project, same question, asked once in Russian and once in English.
| Project | Query language | Similarity | Result |
|---|---|---|---|
| One | Russian | 0.58 | Missed |
| One | English | 0.65 | Exact answer |
| Two | English | 0.885 | Exact answer |
| Two | Russian | 0.76 | Missed |
The decisive detail: in the second project, the Russian query failed to find a note written in Russian that the English query found immediately. That’s not a data problem. The index had been built with the default embedding model, which is trained on English only - its own documentation puts cross-lingual similarity around 0.35, against roughly 0.88 for a multilingual model. Most of my material is in Russian. I’d built a Russian-language knowledge base on an English-only index and spent weeks blaming my notes.
The conclusion I’d already drawn, wrongly. Before that test, I’d confidently written down that project files index well and conversations index badly, and I’d started planning around it. It was wrong. I’d tested one project in English and the other in Russian, so “files versus conversations” was confounded with query language from the start. The primary variable was the language of the question; the mining mode was noise.
I’m including this because it’s the part I’d normally quietly delete. If your retrieval is bad, check your embedding model’s training languages before you redesign anything. And if you drew a conclusion from two tests that differed in more than one way, you didn’t draw a conclusion.
Three rules that generalise
Always search with a project filter. Without one, five unrelated projects answer at once and the top result is confidently from the wrong universe.
Measure the corpus before you index it. I nearly indexed a large cloud-storage folder on the basis of its size in gigabytes. Images, video and archives aren’t indexed at all, so the real text volume was a fraction of that. Count files by extension first, then decide.
Originals never move. Indexing reads a file in place, extracts text in memory, stores a chunk plus a vector plus the path. There’s no reason to copy everything into a central folder, and I’d designed one before checking. That “lake” would have been pure maintenance cost.
The thing memory still hasn’t solved
Everything above works inside one project. What I don’t have is an assistant that holds all of them at once - what’s a priority, what’s blocked, what I said I’d do three weeks ago about a project I haven’t opened since. The coding agents are excellent inside a repository and have no idea the other four exist.
I’ve spent several weeks testing an orchestrator built for exactly this, attracted by its layered memory model, and containerised the same way as everything else. The interim result is honest and unfinished: good at narrow one-off research, not what I hoped for, and its delegation half is deliberately switched off until I’ve designed the approval gates around it.
That experiment has its own working note - trying to build a life admin - published as notes rather than conclusions, because it hasn’t concluded. The question it leaves open is the one I’d most like an answer to: what do you use as a cross-project assistant when you don’t want to be tied to a single vendor’s subscription?
What I’d tell myself at the start
Re-explaining myself: mostly solved, and it took two wrong attempts. The wiki gets read every session and it’s usually right. It still drifts when I’m moving fast and skip writing things down - which is a discipline problem, not an architecture one.
The rest, compressed:
- An archive is evidence, not memory.
- The thing that enters context should be small and curated. The thing that’s huge should be searched, rarely, on purpose.
- Check your embedding model’s training languages before you blame your data.
- If two tests differed in more than one variable, you don’t have a result.
- Always search with a project filter.
- Measure the corpus before you index it.
- Write the log, not just the state. “Changed on this date, because of this” is worth more than the fact itself.
- Being specific about what you haven’t solved is part of the system, not an admission.
And the one I’d most want to go back and tell myself: build the small wiki first, and add search later, when you know what you’re actually looking for. I built the expensive thing first, and the expensive thing was the wrong shape.
Sources, and who’s worth reading
- Andrej Karpathy, LLM Wiki - the pattern behind everything in this part: raw sources stay immutable, the wiki is compiled on top of them, and knowledge accumulates instead of being re-retrieved. If you read one thing from this list, read this.
- MemPalace - the local vector store and conversation archive I use as the cold layer. Runs on the machine, no API key required.