Personal Assistant AI: From Second Brain to Executive Assistant OS

A half-sentence over a messaging app

I fire off seven words on the way out of a meeting: "felt drained after the standup, API redesign dragging." I do not tag it, file it, or open an app. By the time I sit down for my weekly review on Friday, the system has already decided that line was a diary entry, extracted the mood signal, linked it to the project it names, noticed that this is the third low-energy Monday in a row, and put that pattern in front of me unasked.

That is the whole premise, taken from David Allen: your brain is for having ideas, not storing them. The methodologies that act on that premise - Building a Second Brain, Getting Things Done, Zettelkasten - all work, and all demand that you keep doing the remembering. I wanted the discipline to run itself. Not a chatbot you visit; a cognitive extension that is always on, capturing what you tell it, organising it into durable memory, noticing the patterns you would miss, and surfacing the right thing at the right moment.

So the question I set for Personal Assistant AI was not "can a model sound useful?" It was harsher: would I trust it with my own context long enough for it to earn a place in daily use? It now runs 24/7 in the cloud on a single small server, not a cluster, and is backed by a test suite with coverage and mutation gates on every push. It is an ongoing independent project, not employer work, not a public SaaS product.

Talking to your assistant is not one task

It is a dozen tasks wearing the same costume. Logging a feeling is nothing like researching a decision, which is nothing like breaking a project into milestones, which is nothing like deciding whether a notification deserves to interrupt you. Make one prompt and one model handle all of them and you get a system that is fluent and mediocre at everything at once.

So the work is split across eight specialist agents, each with its own model, prompt, and output contract, covering capture, memory, research, review, planning, communications, tools, and notification. On top of them sits a small panel of specialist roles that can teach rather than file - challenging, clarifying, and connecting an idea instead of just storing it - for the times the system should help me think something through. More agents are not automatically better. A specialist earns its place only when its responsibility, input, output contract, and failure mode can be made explicit. Otherwise it is an unaccountable committee that sounds busy and ships nothing.

The harder design choice is the half that uses no model at all. The notification specialist is pure logic: it decides whether an alert fires now, gets batched into a digest, or is suppressed, and it never calls an LLM to do it. The point of a multi-agent system is not to maximise model usage. It is to spend reasoning where it changes the outcome and keep everything else fast, inspectable, and cheap.

The router would rather admit it is unsure

Every message hits a router whose only job is to send it to the right specialist, and it works in two stages. The first stage is deterministic: the unambiguous messages get classified without spending a model call at all, because there is no reason to pay a model to decide something the shape of the message already tells you. The second stage, for the genuinely ambiguous remainder, hands off to a model that returns the specialist to route to, a confidence score, and its reasoning.

Then there is the conversational fallback. When confidence is low and the message is not really a capture - a meta-question like "what can you do?", a correction, an aside - the system does not quietly shove it into the diary. It routes to a path that says it is unsure and lists what it can actually do. I would rather the system tell me it does not understand than pretend it does. A misfiled diary entry is a lie the assistant tells itself, and a second brain that lies to itself is worthless.

Memory is the product

Not chat history. Memory. The difference is the difference between a transcript archive and something that knows what matters to you. It is modelled on how human recall actually works rather than on what a database makes easy, and that single choice shapes everything downstream of it.

It is a layered memory system that escalates from keyword to semantic to relationship-graph retrieval, climbing only as far up that ladder as the question needs. A plain lookup is fast and cheap and surprisingly good on its own. When that is not enough, keyword and meaning are blended so a memory that is both lexically and semantically relevant rises above one that only matched a word. When the question is really about a person or a thread, retrieval walks the relationship graph, so a question about someone can surface the projects they touch and the people connected through them.

Two modifiers turn that from search into recall. The memory fades what is stale and strengthens what you return to: older entries gently lose prominence without ever being erased, and the things I revisit become easier to surface. Retrieval strengthens memory; the loop is small, and it is what makes the assistant feel like it genuinely knows me rather than indexing me.

The half that works while I sleep

A second brain that only thinks when you are looking at it is not a second brain. The proactive half runs on its own schedule. A morning briefing assembles the day. An overnight pass consolidates what I captured, clustering the day's entries, writing summary notes, and growing the knowledge graph, so the system literally sleeps on what I told it and wakes up having organised it. And in the evening, instead of another notification, it can place a reflective phone call and talk me through the day's review. A phone call I answer is a very different thing from a form I will never fill in.

The hardest part of proactive software is not sending alerts; it is earning the right to send them. So the notification layer is a fatigue manager, not a megaphone: it batches low-priority items into digests, suppresses the categories I keep dismissing, and honours do-not-disturb and focus windows. It is allowed to interrupt me only when interrupting is the right call.

And when I want to think rather than file, a five-role Socratic engine takes over: a tutor, a devil's advocate, a clarifier, a connector that ties the question back to my own notes, and an invisible assessor that decides which voice speaks next and watches the ratio of my words to the system's, so it never slides into lecturing me. It is the difference between a tool that stores my thinking and one that sharpens it.

Spend intelligence only where it changes the outcome

Models are matched to task difficulty. Classification and triage stay cheap; the bulk of everyday work uses a capable everyday model; deep reasoning is reserved for research and review, where a better answer is worth the cost. This is the same instinct behind the no-model notification logic: knowing when not to call the model is as much of the engineering as knowing when to.

The reliability stance that holds the structured outputs together is to fail loud. When the model errors or returns malformed output, the system surfaces it rather than silently degrading: no quiet downgrade to a weaker answer, no retry-and-hope that buries the problem. A quiet degradation would erode the trust the whole system is built to earn, so the failure is made visible instead of hidden.

Quality is engineered, not hoped for

When software holds your life's context, silent degradation is unacceptable. A one-line prompt change can improve one specialist and quietly damage another, and you would never know. So a self-checking quality loop scores real interactions continuously, watching for exactly that kind of invisible regression.

Something has to watch the watcher. That quality loop is itself periodically recalibrated against my own judgment, and when its judgement drifts too far from mine it locks changes until I bring the two back into agreement. You cannot tune against a judge you no longer trust, so the system refuses to let me.

What a technical reader can assess

This is a private project in daily personal use, so what I can show publicly is the engineering, not market metrics. Every figure traces back to a private codebase, and the public diagrams stay conceptual by design.

What is on the table to assess is the engineering: a two-stage router with a deliberate "I am not sure" path, a layered memory engine that escalates from keyword to semantic to relationship-graph retrieval with a recall-strengthening feedback loop, models matched to task difficulty, a fail-loud reliability stance, and a self-calibrating quality loop that locks changes when its own judgement drifts. Those are design choices that stand or fall on their own terms, and they are built the way a system meant to hold real context has to be built.

The refusal to fail quietly

What holds this system together is not the agent count. It is the refusal to let anything fail in silence — the router that admits doubt instead of guessing, the judge that locks edits when its own judgement drifts from mine, the model layer that errors loudly rather than degrading in the dark. A second brain only works if you can trust that it will tell you when it is unsure, because the alternative is a confidently mis-filed memory you never catch. Everything else — the durable recall, the overnight consolidation, the proactive nudges — is downstream of that one guarantee.

What stays private

The wall is deliberate. What I do not publish: the exact prompts and routing rules, the data model and retrieval weighting, the consolidation logic, the evaluator rubrics and datasets, the background-job schedules, the deployment and model-provider choices, real dashboard data, and any personal context that ran through the system. Public diagrams stay conceptual - capture, coordination, memory, proactive support, quality monitoring, operations - and stop short of the sequence detail a competitor would need to copy the product.

The portfolio signal is the engineering: a private, account-scoped multi-agent assistant with durable memory, proactive operation, and a self-calibrating quality system, in daily use. What stays protected is the exact behaviour a competitor would need to copy the product.