Content Database System

The wiki uses a multi-layer storage architecture. There is no single database — different kinds of data live in the storage layer best suited for them.

Storage Layers

flowchart TD
  subgraph Durable["Durable Storage"]
      PG[("PostgreSQL
(wiki-server)")]
      KB[("KB YAML
packages/kb/data/things/")]
      YAML[("YAML Files
data/")]
      MDX[("MDX Pages
content/docs/")]
  end

  subgraph Transient["Transient / Session"]
      LRU["In-Memory LRU Cache
(500 entries, per-process)"]
      SOURCES[".cache/sources/
(fetched HTML/text)"]
      HASHES[".cache/content-hashes.json
(change detection)"]
  end

  subgraph Build["Build Artifacts"]
      JSON[("database.json
YAML + MDX → JSON")]
  end

  KB --> JSON
  YAML --> JSON
  MDX --> JSON
  PG -.->|"Hono RPC API"| LRU
  LRU -.->|"cache miss"| PG

1. PostgreSQL (wiki-server)

The wiki-server runs a PostgreSQL database that stores all structured data requiring durability and cross-machine access. This replaced the earlier local SQLite database (.cache/knowledge.db), which was retired in February 2026.

What it stores:

Access pattern: All access goes through the wiki-server's Hono RPC API. CLI tools use apiRequest() from crux/lib/wiki-server/. The frontend uses typed RPC clients (e.g., getFactsRpcClient()).

# Example CLI commands that read/write PostgreSQL
pnpm crux citations verify <page-id>    # Verify citations → writes audit results
pnpm crux query entity <id>             # Read entity data
pnpm crux query search "topic"          # Full-text search

2. In-Memory LRU Cache

Source fetching uses a session-scoped in-memory cache (crux/lib/citation-content-cache.ts) to avoid redundant network requests and database lookups within a single process.

When fetching a URL, the system checks:

1. In-memory LRU cache (fastest)
2. PostgreSQL citation_content table (durable)
3. Network fetch via Firecrawl or built-in fallback (slowest)

Results are written back to both the LRU cache and PostgreSQL.

3. KB YAML (packages/kb/)

The Knowledge Base package (packages/kb/) is the authoritative source for structured entity facts — valuations, revenue, headcounts, founding dates, and other typed properties. As of March 2026, 9+ entities have been migrated here from the older data/facts/ system.

KB facts are rendered on wiki pages via <FBF> and <FBFactValue> components, and computed values via <Calc>. See Data System Authority RulesE1018Defines which data system is authoritative for each entity type, resolving overlaps between KB YAML and old YAML facts.Quality: 47/100 for which system is authoritative for which entities.

4. YAML Files (data/)

Human-editable YAML files are the source of truth for content metadata:

YAML files are checked into git and are the canonical source for everything they contain. PostgreSQL mirrors some of this data for API access and full-text search.

5. File-System Caches (.cache/)

Temporary files for local development workflows:

These are gitignored and can be deleted without data loss.

6. Build Artifact (database.json)

The build pipeline (apps/web/scripts/build-data.mjs) compiles YAML + MDX frontmatter into apps/web/src/data/database.json. This single JSON file contains all entities, pages, relations, facts, search data, and statistics needed by the Next.js frontend.

pnpm build-data           # Full build (~2 min)
pnpm build-data:content   # Content-only rebuild (~15s)

The JSON is loaded at server startup with lazy-built indexes (see ArchitectureE734Technical architecture of the Longterm Wiki — data pipeline, clever design patterns, and the ideas that make it workQuality: 73/100).

Data Flow

flowchart LR
  subgraph Edit["Authoring"]
      AUTHOR["Human or AI
edits YAML/MDX"]
  end

  subgraph Pipeline["Build Pipeline"]
      BUILD["build-data.mjs"]
  end

  subgraph Serve["Runtime"]
      NEXT["Next.js
reads database.json"]
      API["Wiki-server API
reads PostgreSQL"]
  end

  AUTHOR -->|"git push"| BUILD
  BUILD -->|"database.json"| NEXT
  AUTHOR -->|"crux citations verify"| API
  API -->|"search, facts, claims"| NEXT

Source Fetching Flow

When verifying citations or fetching content for page improvement:

sequenceDiagram
  participant CLI as Crux CLI
  participant Cache as LRU Cache
  participant PG as PostgreSQL
  participant Net as Network

  CLI->>Cache: getCachedContent(url)
  alt Cache hit
      Cache-->>CLI: cached content
  else Cache miss
      CLI->>PG: query citation_content
      alt DB hit
          PG-->>CLI: stored content
          CLI->>Cache: setCachedContent(url)
      else DB miss
          CLI->>Net: fetch via Firecrawl / fallback
          Net-->>CLI: raw content
          CLI->>PG: saveFetchResultToPostgres(url)
          CLI->>Cache: setCachedContent(url)
      end
  end

CLI Commands

Limitations

1. No offline PostgreSQL access: CLI commands that query the wiki-server require network connectivity
2. LRU cache is session-scoped: Restarting a process loses cached content (by design — PostgreSQL is the durable tier)
3. database.json must be rebuilt: Changes to YAML or MDX frontmatter are not visible to the frontend until build-data runs
4. Citation content is append-mostly: Old fetched content is not automatically refreshed

Related

• ArchitectureE734Technical architecture of the Longterm Wiki — data pipeline, clever design patterns, and the ideas that make it workQuality: 73/100 — System overview and design patterns
• Automation ToolsE757Comprehensive technical documentation for wiki maintenance automation, covering page improvement workflows (Q5 standards requiring 10+ citations, 800+ words), content grading via Claude API (~$0.02...Quality: 41/100 — Full CLI reference
• Data System Authority RulesE1018Defines which data system is authoritative for each entity type, resolving overlaps between KB YAML and old YAML facts.Quality: 47/100 — Which data system is authoritative for each entity