Agent Memory System

A hierarchical, Markdown-based memory system for AI agents with semantic search capabilities.

Overview

The memory system allows agents to maintain persistent knowledge across conversations. Memories are stored as Markdown files in a virtual filesystem backed by PostgreSQL, enabling:

• Hierarchical organization via paths (e.g., /memories/project/abc123/index.md)
• Multi-level scoping (tenant, project, user)
• Recursive includes for composing complex memory documents
• Semantic search via vector embeddings (pgvector)
• Automatic memory creation from conversation history

Code Pointers

• Models: backend/apps/ai/models/memory_document.py, backend/apps/ai/models/memory_chunk.py, backend/apps/ai/models/memory_event.py
• Store/compiler/search: backend/apps/ai/memory/store.py, backend/apps/ai/memory/context.py, backend/apps/ai/memory/search.py
• Indexing: backend/apps/ai/memory/indexing.py

Architecture

flowchart TB
    subgraph Frontend
        UI[Chat UI]
    end

    subgraph "Chat Handler"
        CH[handle_chat]
        BMP[build_memory_context]
        BSP[build_system_prompt]
    end

    subgraph "Memory System"
        MS[PostgresMemoryStore]
        MC[MemoryCompiler]
        MSE[MemorySearch]
        IDX[Indexer]
    end

    subgraph "Database"
        MD[(MemoryDocument)]
        MCH[(MemoryChunk)]
        ME[(MemoryEvent)]
    end

    subgraph "Background Tasks"
        PTMT[process_thread_memory_task]
        IMDT[index_memory_document_task]
    end

    subgraph "Agent Runtime"
        AG[pydantic-ai Agent]
        MT[memory_toolset]
    end

    UI -->|chat request| CH
    CH --> BMP
    BMP --> MS
    BMP --> MC
    BMP --> MSE
    MC -->|read| MD
    MSE -->|vector search| MCH
    BMP --> BSP
    BSP -->|system prompt with memory| AG
    AG -->|memory_view, memory_search| MT
    MT --> MS
    MT --> MSE
    CH -->|on_complete| PTMT
    PTMT -->|generate changes| AG
    PTMT -->|apply changes| MS
    PTMT --> IMDT
    IMDT --> IDX
    IDX -->|chunk + embed| MCH
    MS -->|CRUD| MD
    MS -->|audit| ME

Data Models

MemoryDocument

The primary storage for memory content.

class MemoryDocument(TimeStampedModel, UUIDModel, TenantModelIDMixin):
    path = models.CharField(max_length=1024, db_index=True)  # e.g., /memories/project/abc/index.md
    scope = models.CharField(max_length=32)  # base, tenant, project, user
    scope_id = models.CharField(max_length=128, blank=True)  # project_id, user_id, etc.
    title = models.CharField(max_length=256, blank=True)
    markdown = models.TextField(blank=True)  # The actual content
    etag = models.CharField(max_length=64)  # For optimistic concurrency
    metadata = models.JSONField(default=dict)

MemoryChunk

Indexed chunks for semantic search.

class MemoryChunk(TimeStampedModel, UUIDModel, TenantModelIDMixin):
    document = models.ForeignKey(MemoryDocument, on_delete=models.CASCADE)
    path = models.CharField(max_length=1024)
    chunk_index = models.IntegerField()
    heading = models.CharField(max_length=512, blank=True)  # Section heading
    text = models.TextField()  # Chunk content
    token_count = models.IntegerField(default=0)
    embedding = VectorField(dimensions=1536, null=True)  # pgvector
    embedding_model = models.CharField(max_length=64, blank=True)

MemoryEvent

Audit trail for all memory operations.

class MemoryEvent(TimeStampedModel, UUIDModel, TenantModelIDMixin):
    event_type = models.CharField(max_length=32)  # write, delete, rename
    path = models.CharField(max_length=1024)
    payload = models.JSONField(default=dict)
    created_by = models.ForeignKey(User, null=True, on_delete=models.SET_NULL)

Memory Scoping

Memories are organized in a hierarchical path structure with different access levels:

graph TD
    subgraph "Memory Path Structure"
        ROOT["/memories"]
        ROOT --> BASE["/base"]
        ROOT --> TENANT["/tenant/{tenant_id}"]
        ROOT --> PROJECT["/project/{project_id}"]
        ROOT --> USER["/user/{user_id}"]
    end

    subgraph "Access Rules"
        A1["Base: Read-only, shared across all"]
        A2["Tenant: All users in workspace"]
        A3["Project: All users with project access"]
        A4["User: Only the specific user"]
    end

    BASE -.-> A1
    TENANT -.-> A2
    PROJECT -.-> A3
    USER -.-> A4

Scope Precedence

When building memory context, scopes are merged with this precedence (highest to lowest):

1. User - Personal preferences and notes
2. Project - Project-specific knowledge
3. Tenant - Workspace-wide settings
4. Base - System defaults

Example Paths

/memories/base/index.md                    # System-wide defaults
/memories/tenant/tenant-123/index.md       # Workspace settings
/memories/project/proj-456/index.md        # Project knowledge base
/memories/project/proj-456/competitors.md  # Project sub-document
/memories/user/user-789/index.md           # User preferences
/memories/user/user-789/writing-style.md   # User sub-document

Recursive Includes

Memory documents can include other documents using a special syntax:

# Project Memory

This is the main project memory.




## Additional Notes

More content here...

Include Resolution

sequenceDiagram
    participant C as Compiler
    participant S as Store
    participant D as Documents

    C->>S: read("/memories/project/p1/index.md")
    S->>D: fetch document
    D-->>S: markdown with includes
    S-->>C: content

    Note over C: Parse includes

    C->>S: read("/memories/project/p1/competitors.md")
    S->>D: fetch document
    D-->>S: content
    S-->>C: content

    C->>S: read("/memories/project/p1/content-calendar.md")
    S->>D: fetch document
    D-->>S: content
    S-->>C: content

    Note over C: Merge all content
    C-->>C: Return compiled output

Include Features

• Cycle detection: Prevents infinite loops from circular includes
• Depth limiting: Configurable maximum nesting depth (default: 10)
• Budget limiting: Configurable maximum characters/tokens
• Source annotations: Optional markers showing where content came from
• Deduplication: Same document included twice is only expanded once

Semantic Search

The system provides semantic search over memory content using vector embeddings.

flowchart LR
    subgraph "Indexing Pipeline"
        DOC[Document] --> CHUNK[Chunker]
        CHUNK --> EMB[Embeddings API]
        EMB --> STORE[(pgvector)]
    end

    subgraph "Search Pipeline"
        Q[Query] --> QEMB[Embed Query]
        QEMB --> COS[Cosine Distance]
        STORE --> COS
        COS --> RESULTS[Ranked Results]
    end

Chunking Strategy

Documents are split into chunks based on:

1. Headings - Each heading starts a new chunk
2. Token limits - Chunks are capped at ~500 tokens
3. Paragraph boundaries - Natural text breaks

Search Methods

1. Vector Search (primary)

   • Query is embedded via OpenRouter API
   • Cosine similarity against stored embeddings
   • Returns top-k most relevant chunks

2. Full-Text Search (fallback)

   • PostgreSQL trigram similarity
   • Used when embeddings unavailable
   • Keyword-based matching

Search Scoping

Searches are automatically scoped to accessible memories:

# Only searches memories the user can access
results = MemorySearch(tenant_id).search(
    query="competitor pricing",
    project_id="proj-456",
    user_id="user-789",
    limit=5
)

Automatic Memory Processing

After every N user messages (configurable), the system automatically processes conversations to extract and update memories.

sequenceDiagram
    participant CH as Chat Handler
    participant TH as Thread
    participant TASK as Celery Task
    participant LLM as Memory Processor LLM
    participant STORE as MemoryStore

    CH->>TH: Check message count
    TH-->>CH: count % N == 0

    CH->>TASK: enqueue process_thread_memory_task

    TASK->>TH: Load messages after last_processed_id
    TH-->>TASK: New messages only

    TASK->>STORE: Get current memory excerpt
    STORE-->>TASK: memory_main

    TASK->>LLM: Generate memory changes
    Note over LLM: Analyzes conversation<br/>Returns JSON operations

    LLM-->>TASK: MemoryChangeSet

    loop For each operation
        TASK->>STORE: Apply write/update/delete
        STORE-->>TASK: doc_id
        TASK->>TASK: Queue indexing task
    end

    TASK->>TH: Update last_memory_processed_message_id

Memory Processor Prompt

The LLM is given this system prompt for memory extraction:

You are a memory maintenance agent.

Goal: maintain a small, high-signal Markdown memory tree for an SEO/project assistant.

Rules:
- DO NOT store raw chat logs or full conversation history.
- Prefer stable facts: project details, constraints, recurring preferences, conventions.
- Avoid personal data unless explicitly provided and useful.
- Keep memories concise; update existing docs rather than creating many new ones.
- Only output JSON matching the provided schema.

Memory Operations

The processor can output these operations:

class MemoryOperation(BaseModel):
    op: Literal["write", "str_replace", "insert", "delete"]
    path: str  # Must be within allowed prefixes
    content: str | None = None
    old_str: str | None = None  # For str_replace
    new_str: str | None = None  # For str_replace
    insert_line: int | None = None  # For insert

Incremental Processing

To avoid reprocessing entire conversations:

1. Thread stores last_memory_processed_message_id in metadata
2. Task only fetches messages created after that timestamp
3. After processing, updates the tracking ID

Agent Toolset

Agents have access to memory via the memory_toolset:

memory_view

List directories or read files:

# List root memories
memory_view(path="/memories")
# Returns: ["base/", "project/", "user/"]

# Read a specific file
memory_view(path="/memories/project/p1/index.md")
# Returns: "# Project Memory\n\nContent here..."

memory_search

Semantic search across accessible memories:

memory_search(query="competitor pricing strategy", limit=5)
# Returns: [
#   {"path": "/memories/project/p1/competitors.md", "heading": "Pricing", "snippet": "..."},
#   ...
# ]

Configuration

Settings in config/settings.py:

# Embeddings model (OpenRouter format)
AI_MEMORY_EMBEDDINGS_MODEL = env(
    "AI_MEMORY_EMBEDDINGS_MODEL",
    default="openai/text-embedding-3-small"
)

# LLM for memory processing
AI_MEMORY_PROCESSOR_MODEL = env(
    "AI_MEMORY_PROCESSOR_MODEL",
    default="openai:gpt-4.1-mini"
)

# Process memory every N user messages
AI_MEMORY_PROCESS_EVERY_N_USER_MESSAGES = env.int(
    "AI_MEMORY_PROCESS_EVERY_N_USER_MESSAGES",
    default=5
)

Prompt Integration

Memory is injected into the system prompt via a template:

{% if memory_enabled %}
## Memory

You have access to a persistent memory system. Use `memory_view` to browse
and `memory_search` to find relevant information.

{% if memory_main %}
### Current Memory

{{ memory_main }}
{% endif %}

{% if memory_snippets %}
### Relevant Context

{% for snippet in memory_snippets %}
**{{ snippet.path }}** ({{ snippet.heading }}):
{{ snippet.text }}

{% endfor %}
{% endif %}
{% endif %}

Usage Examples

Creating a Memory Document

from apps.ai.memory.store import PostgresMemoryStore

store = PostgresMemoryStore(tenant_id="tenant-123")

# Write a new document
doc_id = store.write_file(
    path="/memories/project/proj-456/competitors.md",
    content="# Competitors\n\n- Competitor A: Focus on enterprise\n- Competitor B: SMB market",
    created_by_id="user-789"
)

Searching Memories

from apps.ai.memory.search import MemorySearch

search = MemorySearch(tenant_id="tenant-123")
results = search.search(
    query="What are the main competitors?",
    project_id="proj-456",
    limit=3
)

for hit in results:
    print(f"{hit.path} [{hit.heading}]: {hit.snippet}")

Compiling Memory Context

from apps.ai.memory.context import build_memory_context

context = build_memory_context(
    tenant_id="tenant-123",
    project_id="proj-456",
    user_id="user-789",
    query="competitor analysis"
)

# context = {
#     "memory_enabled": True,
#     "memory_main": "# Project Memory\n...",
#     "memory_snippets": [{"path": "...", "heading": "...", "text": "..."}]
# }

Database Schema

erDiagram
    MemoryDocument {
        uuid id PK
        uuid tenant_id FK
        string path
        string scope
        string scope_id
        string title
        text markdown
        string etag
        json metadata
        datetime created
        datetime modified
    }

    MemoryChunk {
        uuid id PK
        uuid tenant_id FK
        uuid document_id FK
        string path
        int chunk_index
        string heading
        text text
        int token_count
        vector embedding
        string embedding_model
        datetime created
        datetime modified
    }

    MemoryEvent {
        uuid id PK
        uuid tenant_id FK
        string event_type
        string path
        json payload
        uuid created_by FK
        datetime created
    }

    Thread {
        uuid id PK
        json metadata
    }

    MemoryDocument ||--o{ MemoryChunk : "has chunks"
    MemoryDocument ||--o{ MemoryEvent : "has events"
    Thread ||--o| MemoryDocument : "triggers processing"

Best Practices

Memory Organization

1. Keep index.md files small - Use includes for detailed content
2. Use descriptive paths - /memories/project/p1/seo-strategy.md not /memories/project/p1/doc1.md
3. Update rather than create - Prefer modifying existing docs over creating new ones

Content Guidelines

1. Store facts, not conversations - Extract insights, not chat logs
2. Be concise - Memory is injected into prompts, affecting token usage
3. Use structure - Headings and lists are easier to search and parse

Performance

1. Limit includes - Deep nesting increases compilation time
2. Index important documents - Ensure key docs have embeddings
3. Scope searches - Always provide project_id/user_id to limit search space