Why I built this
I keep hitting the same wall on every multi-agent project. The agents forget everything between runs. Every session starts from zero.
The fix everyone uses is to bolt on a vector database for similarity search, add a graph database for relationships, then drop in SQLite for the structured stuff. Three backends. They don't agree on what matters, what's expired, or what happens when two agents update the same fact at the same time. I got tired of building this for the third time.
So I spent the last few weeks building what I think agent memory should look like. CogDB. v0.4.0 just shipped. The Phase 2 benchmark on tri-memory retrieval came back at 90.7 / 100. This post is what I learned getting there.
What's broken right now
Mem0 wraps a vector database with extraction logic on top. Zep adds a temporal graph layer. Letta gives agents a memory tool to manage their own context. MemPalace got 27K stars in a weekend by putting a spatial metaphor on top of ChromaDB. Each one is good at something specific.
But none of them combine all three memory types from cognitive science. Episodic — what happened. Semantic — what is known. Procedural — how to do things. The third one is the gap I kept noticing. I went looking and couldn't find a single production system that captures and replays agent workflows as first-class memory.
The other thing nobody does well: token-aware retrieval. Every memory system I tried dumps a wall of context into the LLM and hopes it finds what matters. You're paying per token. An agent doesn't need 4,000 tokens of stuff. It needs the right 500.
The three stores in one engine
I built three stores behind a single interface, sharing one query planner.
Episodic memory sits in a vector store with importance scores, agent IDs, and timestamps. Similarity search with rank blending — HNSW scores get combined with importance scores at query time so recent and important memories surface together.
Semantic memory is a knowledge graph where every fact has a validity window. When a newer fact contradicts an older one, the older one gets superseded automatically. So when your agent learns the API just shipped v2.4, the old (api_service, version, v2.3) triple expires on its own. Confidence scores resolve ties when two agents make contradicting claims.
Procedural memory was the hardest piece to design. When an agent successfully completes a multi-step task — fixing a CORS error, debugging a flaky test, deploying to production — the sequence gets captured as a template. Success rate tracks across executions. Next time a similar context shows up, the template surfaces.
These three aren't stitched together at the application layer like Mem0 or MemPalace do it. They share the same storage engine, the same consistency model, the same MCP interface.
Token-aware retrieval
This took the longest to get right.
Instead of returning whatever ranks highest in the vector search, CogDB fills your token budget in tiers. L0 is identity (around 50 tokens). L1 is critical facts from the knowledge graph (around 200). L2 is task-relevant memories from episodic and procedural stores (around 250). L3 is deep search across everything else.
You set the total. The retriever fills from the top with the highest-importance memories that fit. When the budget runs out, lower tiers get truncated cleanly. The agent gets what it needs and not a token more.
Latency on Phase 2 is sub-10ms for most queries. That's running on the Rust storage engine with PyO3 bindings, not the Python proof-of-concept anymore.
Multi-agent memory scopes
When agents collaborate they need different access levels. CogDB has four formal scopes. Private — single agent, isolated. Team — defined group, read-write with conflict resolution. Organization — all agents read, permissioned write. Session — ephemeral, auto-deleted when the conversation ends.
Contradiction detection runs at write time. Concurrent writes get resolved by confidence comparison. There's a UCSD paper from March (Multi-Agent Memory from a Computer Architecture Perspective) that frames this as a cache coherence problem. I borrowed the consistency model from there.
What I learned
A surprising amount of database design transfers directly. Learned indexes from Kraska et al. (2018) — replacing B-Trees with neural networks — apply to vector retrieval in agent contexts. The CoALA framework from Princeton gave me the cognitive memory taxonomy for free. Cache coherence handled the consistency problem.
The hard part was procedural memory. Almost no academic literature on capturing agent workflows as reusable templates. I ended up modeling it as success-rate-weighted retrieval over tokenized procedure descriptions, with applicable contexts as extra matching signal. Works well enough. I think there's a better formulation hiding somewhere — possibly using the JEPA direction LeCun keeps talking about, where procedures predict their own outcomes before being recommended. That's Phase 4 territory.
The thesis connection is direct. Resource-efficient multi-agent systems means making each agent more capable per inference call. A budget of 500 tokens with the right memories beats 4,000 with random ones. Every reduction in context size cuts cost and latency together. CogDB started as a side project and became infrastructure my thesis actually needs.
What I'm still chewing on: how should agents evolve their own memory schemas? Right now CogDB has typed metadata schemas but agents can't introduce new memory types or relationships through interaction. Memory that grows with the agent, instead of memory the agent has to fit into. That's where I want to go next.
Papers & Resources
- Sumers, Yao et al. (2024) — Cognitive Architectures for Language Agents — arxiv.org/abs/2309.02427
- LeCun (2022) — A Path Towards Autonomous Machine Intelligence — openreview.net/forum?id=BZ5a1r-kVsf
- Kraska et al. (2018) — The Case for Learned Index Structures — arxiv.org/abs/1712.01208
- Multi-Agent Memory from a Computer Architecture Perspective (UCSD, 2026) — arxiv.org/html/2603.10062v1
- CogDB repository — github.com/MuLIAICHI/cogdb
I'm Mustapha Liaichi, an AI engineer exploring the frontier of LLM systems and autonomous agents. I'm preparing a PhD on resource-efficient multi-agent LLM systems at Mohammed V University. These notes document my research journey. Reach me at mustaphaliaichi@gmail.com