Giving Claude a Second Brain

Every conversation I have with Claude starts the same way: with Claude knowing nothing about me, nothing about my homelab, nothing about any of the projects I'm working on. I fix this the obvious way, by pasting context, pointing at files, or re-explaining decisions I've already explained fifty times.

That works, and it costs more than it looks. Everything I paste lands in the context window, and the context window is finite. On a standard plan it is finite enough to feel, which changes the problem entirely: the tokens spent explaining the background are the same tokens the actual work needs. Front-load enough history to make a session useful and a real portion of the budget is gone before anything has been done. Run long enough and the conversation gets compacted, the summary quietly loses the detail that was pasted in, and it gets pasted again. I have had sessions where the first ten minutes were me re-explaining my own infrastructure, and the first thing lost to compaction was that same explanation.

The deeper problem is that this is the wrong shape. Handing over everything that might be relevant, up front, on the off-chance, is the opposite of how retrieval should work. What I want is for the model to pull the two or three things that actually bear on the conversation at hand, at the moment it needs them, from something that already knows them. Then the cost scales with relevance rather than with the size of my history. The model is capable; the context it gets handed is threadbare, and it is threadbare because the only mechanism for filling it is expensive at both ends.

I have a lot of working contexts, each with its own repo, its own vocabulary, and in several cases its own rules about what can be discussed where.

This post is about a system I built to do exactly that, and have now been running for a few months. I'm calling it claude-brain. Some of what follows is the design as I planned it; some of it is what the design became once it met real use, which was not the same thing. No plan survives contact with the enemy, and on a personal project the enemy is you, a few weeks later, using the thing you built.

What separation has to mean

When I sit down to work on the novel, I want Claude to remember what I decided about a character three chapters ago. When I open the homelab, I want it to know which services I run and which decisions I've already taken about my network. When I'm writing a blog post, I want it to remember the standing methodology from the last three posts on the same topic so I'm not reinventing my own voice every time.

What I emphatically do not want is for any of those contexts to bleed into another one. The novel's plotting should not leak into my technical writing. The homelab's secrets should not turn up in a book draft. Different projects have different rules about what belongs in them, and the right place to enforce those rules is the system itself, not my own vigilance at three in the morning.

A discipline-based solution ("just remember not to mix them") fails on a long enough timeline. Humans forget, models hallucinate, and a system that requires constant attention to stay safe will eventually fail. A system that makes the correct direction the easy direction, and the wrong direction outright impossible, will not.

Prior art: Karpathy's LLM Wiki Pattern

The starting point was Andrej Karpathy's short gist describing a pattern he calls the LLM Wiki. The sketch is three layers:

1. Raw sources, immutable. Meeting notes, logs, papers, transcripts, chat history.
2. An LLM-maintained wiki of interlinked markdown pages, compiled from those sources by the model itself.
3. A schema document that describes the structural conventions the wiki is supposed to follow.

And three operations: Ingest (convert raw sources into wiki updates), Query (answer questions from the wiki, not from the raw sources), and Lint (periodically clean up contradictions, duplicates, and stale links).

The key move is treating the wiki as a distillation of the raw material, not a replacement for it. The raw stuff never gets edited, so there's always a ground truth. The wiki is the index the model actually reads, and the model is the one maintaining it.

This is right, and I've adopted it almost wholesale. But I've made one significant change, and it is easier to see side by side:

That right-hand column is the design I started from, not the system I ended up with. Two of those rows did not survive contact with using it.

Where claude-brain diverges: the knowledge layer isn't for humans

Karpathy's wiki is a folder full of markdown, readable by humans as a side benefit. My knowledge layer is a Postgres database, and it is explicitly not shaped for human reading. It is shaped for Claude.

The reason is that I already have an enormous amount of human-readable content. Blog posts. Book drafts. Homelab documentation. Novel chapters. These are the raw sources. Producing another human-readable layer on top of them would just create a second thing to maintain. What I actually need is a distilled, structured index that Claude can query cheaply and reason over quickly. That's not a wiki; that's a database.

There's a second reason, and in daily use it has turned out to matter more than the first. A folder of markdown lives on a machine. Whichever machine that is becomes the place the knowledge is, and every other device is a second-class citizen: you sync it, or you clone it, or you go without. A database on the network belongs to no device in particular. It sits on the homelab and anything that can reach it gets the same answers.

That distinction shows up constantly. Claude Desktop on the Mac and Claude Code in a terminal are talking to the same store, not to two copies that have quietly diverged. If I work from a different machine, there is nothing to synchronise first. A card written in a terminal session at lunchtime is there in a desktop conversation that evening, because there was only ever one copy of it.

The caveat is that this is not free or automatic. Every app has to be configured to point at the MCP endpoint, and each has to be pointed at the right subject. What the database buys me is that once a device is configured, it is genuinely equivalent to every other device rather than approximately equivalent. There's no sync to have failed and no local copy to be stale. The knowledge is somewhere, singular, and everything else just reads from it.

So the claude-brain schema has two core entity types:

• Topic cards, which describe what something is. A methodology, a tool, a person, a standing decision, a character, a location. Small, atomic, updatable.
• Episodes, which describe what happened. A work session, an incident, a deployment, a decision being taken, a scene being written.

Everything Claude knows gets modelled as either a thing or an event. Tags, links, and source pointers live in join tables so the graph can be traversed.

Retrieval started as Postgres full-text search (tsvector) and nothing else. That was a deliberate choice to avoid the complexity of a vector store on day one, but I kept the MCP tool surface agnostic about how search actually worked, so it could change underneath without anything above it noticing. It since has: search is now hybrid, full-text alongside pgvector embeddings generated on write by a small embedding model running on its own box. Every card in the brain is embedded. Nothing above the search function changed when that landed, which is the whole argument for putting the seam there in the first place.

The rules engine

A database isn't enough. What makes this work is the rules engine sitting on top of it, and that engine has three jobs: stop the wrong things being written, put the right things in the right place, and keep the brain clean over time.

Subjects and Postgres roles

The top-level concept is a subject. A subject is a namespace with its own access rules, and each body of work I keep separate gets its own subject. Throughout this post I'll use a shared personal subject plus one each for the homelab, the novel and the book. The labels are illustrative; the structure holds whatever you put in it. Subjects are not enforced in application code. They're enforced at the database layer as Postgres schemas, with one Postgres role per subject. A role has full access to its own schema and no access at all to any other subject's schema.

The MCP server opens its connection as the role matching the subject the session was launched with. A write aimed at the wrong schema fails at the wire, before any of my code gets an opinion about it.

The whole isolation model is about a dozen lines of SQL, and it is worth reading them rather than taking my word for it:

-- A project role owns its own schema outright.
GRANT USAGE ON SCHEMA homelab TO role_homelab;
GRANT SELECT, INSERT, UPDATE, DELETE
    ON ALL TABLES IN SCHEMA homelab TO role_homelab;
 
-- ...and reaches the shared schema, deliberately, including writes.
-- Note the absence of DELETE: an upcast can add and amend, never remove.
GRANT USAGE ON SCHEMA personal TO role_homelab;
GRANT SELECT, INSERT, UPDATE
    ON ALL TABLES IN SCHEMA personal TO role_homelab;
 
-- No GRANT to novel, book, or any other subject's schema.
-- The isolation is the statement that isn't there.

Sequence grants and default privileges are left out for brevity, so this is not a working bootstrap script: paste it as-is and your first insert fails on a sequence permission. Both omissions fail closed rather than open, which is the right direction for a mistake to run in.

That last comment is the entire security model. There is no rule engine deciding what a session may touch, no policy file, no middleware. A role can reach what it has been granted and nothing else, and the grants for the other subjects were simply never written.

The asymmetry is the point, and it is worth being precise about which direction is sealed. The personal role has grants on the personal schema and nothing else. It cannot read any project schema, at all, ever. That is the property I actually care about: nothing from the novel can surface in a homelab session, and no amount of clever prompting will get it there, because the data is not visible to the connection.

The other direction is deliberately open. A project role can read the personal schema, so reusable methodology flows down into whatever I'm working on automatically. It can also write to it. The first version of this design, the one in my notes, said the boundary was sealed in both directions. It was not, and I would have told you it was. It's called an upcast: something learnt inside one project that turns out to be generic, a Postgres gotcha found while building the brain itself, or a backup pattern worked out for the homelab, promoted into the shared space so the next project inherits it. That has to be possible, or the shared brain never learns anything from the work that actually happens.

What stops an upcast becoming a leak is not the role model, because the grant is there. It's two things layered on top. A deny-list scans every write into the personal schema for tokens that tie it to one specific project, and for PII-shaped patterns, and refuses on a hit; it is deliberately noisy, because a false positive just means the card stays in the project schema instead, which is the safe outcome. And an outbox table records every write with an actor, an operation, and a reason, so an upcast without a justification is a flag rather than a silent event.

So the honest version is this: the direction that must never happen is structurally impossible, and the direction that should be rare is possible, guarded, and audited. I don't have to remember to stay inside the lines going one way. Going the other way, I have to say why.

The kind vocabulary

Every topic and every episode has a kind. The list of valid kinds lives in a single YAML file, loaded by the server at startup and consulted by every write. Unknown kinds are rejected outright with an error listing the valid ones.

The rule I set for myself was that every kind is either a thing or an event, never both. The same distinction runs through the technical work and the fiction, which is what convinced me the split was real rather than convenient:

Read the pairs across and the discipline shows up. decision is a topic (the standing decision itself). decision-made is an episode (the moment the decision was taken). chapter-summary is a topic (what the chapter is about). chapter-write is an episode (the session where it was drafted). A bug is always an episode, because bugs happen at a moment in time; a recurring class of bug is a design pattern, which is a topic.

Having a single authoritative file forces discipline. Claude cannot quietly invent a new kind on the fly; if a new category is needed, I edit the YAML. That rigidity is deliberate. Without it, a year from now I'd have seventeen near-synonyms for "thing I once wrote down" and no way to query reliably.

Search before add

The most important rule in the curation loop is this: before creating any new topic, Claude must search the brain for existing matches. If a confident match comes back, the existing topic is updated rather than duplicated. If loosely related topics come back, a new topic is created with mentions links drawn to each of them. Only if the search returns nothing meaningful is a fresh island topic created.

This is the single most effective quality control I've added. Without it, every session would re-create the same methodology card from scratch, with slight variations, and within a week the brain would be a lumpy sediment of near-duplicates. With it, topics accumulate information over time rather than piling up sideways.

Capture at session end, and why I turned it off

The original design had this fully automated: a SessionEnd hook fired at the close of every Claude Code session, ran a headless subtask that read the transcript, and wrote topics and episodes without my involvement. It printed a one-line receipt so I could see what it had done.

I built it, ran it from April until the start of July, and then retired it. It's worth saying why, because the reasoning generalises.

The problem with a headless job that writes to a database is that it can fail quietly. Mine did. A routing bug meant a run of sessions distilled into the wrong place, and because nobody was watching a process designed not to need watching, it ran that way for most of the time it was live. The isolation guards held throughout, which is the one genuinely reassuring part of the story. The cost was history: I recovered the most recent month by re-running the distillation by hand against the transcripts still on disk, and lost the two months before that outright, because Claude Code prunes transcripts after about thirty days.

So capture is now deliberate. A /session-close command sweeps the session, proposes a set of subject-tagged cards against an explicit rubric for what is worth keeping, and writes nothing until I confirm. What I see at the end of a session looks like this:

Proposed for homelab:
  + episode  deploy          "Rebuilt brain stack on pgvector image"
  + topic    environment     "embeddings host, nomic-embed-text, 768-dim"
 
Proposed for personal (upcast):
  + topic    design-pattern  "Structural enforcement beats procedural"
      reason: generic, no homelab specifics
  ~ topic    tool            "psycopg3"  (append: search_path scoping)
      reason: applies to any psycopg3 work
 
  4 proposed, 2 upcasts. Write these? [y/n/edit]

The ~ is an update to a card that already exists, which is the search-before-add rule doing its job rather than creating a fifth near-duplicate of the same thought. It is more friction and I think the friction is correct: every write is now something I have seen, with a subject I have agreed, and there is no unattended model run that can drift without anyone noticing.

The trade is real and I'll state it plainly. If I don't run /session-close, nothing is captured. I chose deliberate over silent, having been burnt by silent.

The lint loop

A brain_lint pass walks a subject looking for things curation got wrong: near-duplicate topics, contradictory statements, orphans, dead links, stale source pointers, unexplained upcasts, and anything else out of place. Some fixes get applied directly; higher-risk ones are proposed for review. This is Karpathy's lint operation applied to a structured store rather than a markdown tree.

It runs on demand rather than on a schedule. I keep meaning to put it on a timer and keep not doing it, partly because the same argument that retired the capture hook applies here too: an unattended job that rewrites my knowledge base is exactly the kind of thing I'd want to be watching.

How it's actually deployed

The rules engine is the interesting part, but none of it matters if the thing falls over or gets rebuilt inconsistently, so the deployment story matters too. Claude-brain runs on my homelab, and like everything else in the homelab, it lives entirely in code. Laid out, it is unremarkable, which is the point:

Each row is worth a sentence or two on why it is that way.

Postgres and the MCP servers run as a Docker Compose stack on an apps LXC on Proxmox. One container for Postgres, and then one brain-mcp container per subject, each written in Python (FastMCP + psycopg3). That last detail is not incidental: each container is handed exactly one subject and exactly one role's credentials, so the subject boundary is a process boundary as well as a database one. A container physically does not hold the credentials to reach another subject's schema. The compose file, Dockerfile, migrations, and server code all live in a single repo alongside the schema design and this post's raw notes.

The rest is the homelab's standard machinery. An Ansible role creates the stack, clones the repo, renders .env from the same Vault everything else uses, and brings compose up, so nothing on the LXC is configured by hand and a fresh host rebuilds from a playbook run, schema migrations included. Caddy fronts it, terminating TLS against the internal CA and enforcing the bearer token on every request. brain-mcp.sbd.org.uk resolves from dnsmasq on the router, declared in the same network configuration repo as every other internal name, with no public DNS record anywhere. Going external later means a Terraform record and a tunnel route, in the same pattern as the handful of services that already are, rather than a new deployment approach.

The point of labouring this is that the rules engine only holds if the deployment does. A careful schema design and a locked-down role model are worth nothing if the next deploy overwrites the Postgres container with a fresh volume and loses the lot. The same config-as-code discipline I apply to endpoint management isn't a nice-to-have here; it's load-bearing.

What it feels like to use

The change is small in any individual session, but compounding. When I ask Claude in one project's context "have we seen this pattern before", it answers from the personal brain, because the session has read access to the shared space. When I ask the same question in a different project, I get that project's knowledge plus the shared space, and nothing from anywhere else. When I close out a chapter of the novel and the episode gets recorded, the next time I want to ask "what was I doing last time in this storyline", there's an answer. None of that context had to be handed over by me at the start of the session. It was already there.

What I notice most is how much less I'm pasting. Claude knows things. Not everything, not yet, but enough that the baseline starting state of a conversation is no longer zero. That changes the economics of a session in the way I hoped it would: the window fills with the work rather than with the preamble, and a few retrieved cards cost a fraction of what dumping the background used to. The sessions that used to run out of room now mostly don't.

What I learnt building it

Before the lessons, the scoreboard, and it is worth separating the things that changed cheaply from the things that changed expensively:

The top two cost me an afternoon. The next two cost me months, and one of them cost me data I cannot get back. That gap is the whole lesson, and three smaller ones follow from it.

First: structural enforcement beats procedural enforcement every single time. I initially considered doing subject isolation in the MCP server's application code. Moving the boundary down to Postgres roles made the direction that matters impossible rather than merely discouraged. A rule a database enforces is a rule; a rule a program enforces is a suggestion. That is not a claim about Postgres. It is the same argument as enforcing access in the directory rather than in the application, and it is worth asking of any control you have built: is the wrong action impossible, or merely discouraged? The corollary is that you should be honest with yourself about which rules you have actually enforced and which ones you have only written down. My first draft of this design said the boundary was sealed in both directions. It was not, and a review caught it before any code existed, which is the cheapest possible moment to find out.

Second: a rigid vocabulary is a feature, not a limitation. The temptation with LLMs is to let them be flexible because they can. Resisting that temptation, pinning the vocabulary to a YAML file that the model cannot bypass, is what stops the brain becoming mush over time.

Third: I had this one backwards. I originally concluded that automation only works if it's cheap and visible, and that the automatic curator plus a one-line receipt had struck the right balance. What actually happened is that the automatic half failed quietly and the receipt was not enough to catch it, because a receipt only tells you what a process thinks it did. The lesson I'd write now is narrower: automate the retrieval, because being wrong there costs you a bad answer you can see. Be far more careful about automating the writes, because being wrong there costs you a corrupted knowledge base you cannot see.

What's next

The brain is still internal only, reachable from my Mac over the homelab network. Exposing it externally via Cloudflare Tunnel, with proper MCP remote-server authentication so Claude on mobile and the web app can reach it, remains the plan and remains undone. I want to be able to ask "what did we decide about X" from my phone, on the road, away from the laptop, and get the same answer I would at my desk. It keeps losing to work that was more urgent, which is the honest reason most roadmap items slip.

The other outstanding piece is a human-readable view over the same Postgres store, not to replace the brain but to let me browse what has accumulated and see the health of it: duplicate clusters, orphans, subjects that have gone quiet. That one is designed and not built.

The code isn't public, and I'm undecided about whether it should be. The interesting part is the shape rather than my particular implementation of it, and the shape is entirely described above.

The brain started empty. It grew by use, which is exactly what I hoped for and slower than I expected. My phrase for it during the design was "training it", and that still captures how it feels: each session leaves a slightly smarter version behind. The next conversation inherits that. And the one after that. What I'd add, a few months in, is that the compounding is real and it is not free. The system needs tending, and the rules need to survive contact with use. The parts I was proudest of designing are not the parts that turned out to matter. Claude no longer starts every session from zero, which was what I wanted. Getting there just involved being wrong about more of the details than I expected.