Karpathy's LLM Base: Building Personal Knowledge Bases with LLMs

Karpathy's LLM Base: Building Personal Knowledge Bases with LLMs

1. Overview

This article covers Andrej Karpathy's approach to building personal knowledge bases using Large Language Models (LLMs). Instead of traditional RAG (Retrieval-Augmented Generation) systems, Karpathy has developed a pattern where LLMs incrementally build and maintain a structured, interlinked wiki from raw source documents.

This approach is designed for researchers, developers, and professionals who need to accumulate and organize knowledge over time. It's particularly valuable for anyone working with complex information that requires synthesis across multiple sources.

Prerequisites include familiarity with LLMs, basic knowledge of markdown, and access to an LLM with a sufficiently large context window for compilation tasks.

2. Core Concepts

The Core Idea

Most people's experience with LLMs and documents follows the RAG pattern: uploading a collection of files, having the LLM retrieve relevant chunks at query time, and generating answers. While this works, the LLM must rediscover knowledge from scratch on every question. For subtle questions requiring synthesis across multiple documents, the LLM must find and piece together relevant fragments repeatedly.

Karpathy's approach differs fundamentally. Instead of just retrieving from raw documents at query time, the LLM incrementally builds and maintains a persistent wiki—a structured, interlinked collection of markdown files that sits between you and the raw sources. When you add a new source, the LLM reads it, extracts key information, and integrates it into the existing wiki—updating entity pages, revising topic summaries, noting contradictions, and strengthening or challenging the evolving synthesis.

The key difference is that the wiki is a persistent, compounding artifact. Cross-references already exist, contradictions are flagged, and synthesis reflects everything you've read. The wiki grows richer with every source added and every question asked.

Three-Layer Architecture

The system consists of three distinct layers:

1. Raw sources - Your curated collection of source documents (articles, papers, images, data files). These are immutable—the LLM reads from them but never modifies them. This is your source of truth.

2. The wiki - A directory of LLM-generated markdown files containing summaries, entity pages, concept pages, comparisons, overviews, and synthesis. The LLM owns this layer entirely, creating pages, updating them when new sources arrive, maintaining cross-references, and keeping everything consistent.

3. The schema - A document (e.g., CLAUDE.md for Claude Code or AGENTS.md for Codex) that tells the LLM how the wiki is structured, what conventions to follow, and what workflows to use when ingesting sources, answering questions, or maintaining the wiki. This configuration file makes the LLM a disciplined wiki maintainer rather than a generic chatbot.

The Compiler Analogy

Think of this system like a code compiler: raw/ is the source code, the LLM is the compiler, the wiki is the executable, health checks are the test suite, and queries are the runtime. Karpathy describes it as "building GCC for knowledge."

3. Detailed Explanation

Operations

The system follows a continuous cycle of four main operations:

1. Ingest

You drop a new source into the raw collection and tell the LLM to process it. The LLM reads the source, discusses key takeaways with you, writes a summary page in the wiki, updates the index, updates relevant entity and concept pages across the wiki, and appends an entry to the log. A single source might touch 10-15 wiki pages.

You can ingest sources one at a time with involvement (reading summaries, checking updates, guiding the LLM on what to emphasize) or batch-ingest many sources at once with less supervision. The workflow should be documented in the schema for future sessions.

2. Query

You ask questions against the wiki. The LLM searches for relevant pages, reads them, and synthesizes an answer with citations. Answers can take different forms depending on the question—a markdown page, a comparison table, a slide deck (Marp), a chart (matplotlib), or a canvas.

Crucially, good answers can be filed back into the wiki as new pages. Comparisons, analyses, and discovered connections are valuable and shouldn't disappear into chat history. This way, explorations compound in the knowledge base just like ingested sources do.

3. Lint

Periodically, you ask the LLM to health-check the wiki. This involves:

• Looking for contradictions between pages
• Identifying stale claims that newer sources have superseded
• Finding orphan pages with no inbound links
• Noting important concepts mentioned but lacking their own page
• Identifying missing cross-references
• Spotting data gaps that could be filled with web search

The LLM is good at suggesting new questions to investigate and new sources to look for, keeping the wiki healthy as it grows.

4. Indexing and Logging

Two special files help navigate the wiki as it grows:

• index.md - A content-oriented catalog of everything in the wiki, with each page listed with a link, a one-line summary, and optional metadata like date or source count. Organized by category (entities, concepts, sources, etc.). The LLM updates it on every ingest.

• log.md - A chronological, append-only record of what happened and when—ingests, queries, lint passes. If each entry starts with a consistent prefix (e.g., ## [2026-04-02] ingest | Article Title), the log becomes parseable with simple unix tools.

Tools and Implementation

The implementation relies on several tools:

• Obsidian - Serves as the IDE and file viewer for the wiki. Its graph view is particularly useful for seeing the shape of the wiki—what's connected to what, which pages are hubs, and which are orphans.

• Obsidian Web Clipper - A browser extension that converts web articles to clean .md files with locally downloaded images.

• qmd - A local search engine for markdown files with hybrid BM25/vector search and LLM re-ranking, all on-device. It has both a CLI (so the LLM can shell out to it) and an MCP server (so the LLM can use it as a native tool).

• Marp - A markdown-based slide deck format. Obsidian has a plugin for it, useful for generating presentations directly from wiki content.

• Dataview - An Obsidian plugin that runs queries over page frontmatter. If the LLM adds YAML frontmatter to wiki pages (tags, dates, source counts), Dataview can generate dynamic tables and lists.

The wiki is essentially a git repo of markdown files, providing version history, branching, and collaboration for free.

Use Cases

Karpathy built his knowledge base for ML research, but the pattern applies to many contexts:

Personal: Tracking goals, health, psychology, and self-improvement by filing journal entries, articles, podcast notes, and building a structured picture over time.

Research: Going deep on a topic over weeks or months by reading papers, articles, reports, and incrementally building a comprehensive wiki with an evolving thesis.

Reading a book: Filing each chapter as you go, building pages for characters, themes, plot threads, and their connections—creating a rich companion wiki similar to fan wikis like Tolkien Gateway.

Business/team: Creating an internal wiki maintained by LLMs, fed by Slack threads, meeting transcripts, project documents, and customer calls—possibly with humans in the loop reviewing updates.

Other applications: Competitive analysis, due diligence, trip planning, course notes, hobby deep-dives—anything where knowledge accumulates over time and needs organization rather than being scattered.

4. Common Questions

How is this different from traditional RAG?

Traditional RAG systems retrieve relevant chunks from documents at query time and generate answers from scratch each time. Karpathy's approach builds a persistent, structured wiki that compiles knowledge once and keeps it current. The wiki already contains cross-references, flagged contradictions, and evolving synthesis, eliminating the need to rediscover connections on every query.

What scale can this approach handle?

Karpathy's research wiki has grown to approximately 100 articles and 400,000 words on a single topic. At this scale, the index files plus LLM context window are sufficient for retrieval, eliminating the need for embedding-based RAG infrastructure. However, as the wiki grows beyond this scale, additional tools like qmd may become necessary for efficient search.

How does the LLM maintain consistency?

The LLM maintains consistency through the lint operation, which periodically scans for contradictions between pages, stale claims that newer sources have superseded, orphan pages, missing cross-references, and data gaps. The LLM can also impute missing information via web search and discover connections between concepts that could become new articles.

5. Related Topics

Graph RAG Connection

Karpathy's wiki-with-backlinks approach is essentially a manual, markdown-based implementation of Graph RAG—a retrieval technique where knowledge is stored as a graph of interconnected nodes rather than flat documents. Tools like Neural Composer for Obsidian are building this into plugins, using Obsidian's 3D graph view to visualize connections that the AI discovers.

Personal Knowledge Management (PKM)

The personal knowledge management community—followers of Tiago Forte's "Building a Second Brain" methodology and the Zettelkasten method—recognizes Karpathy's workflow as a significant evolution. While traditional PKM requires manual note-taking, linking, and review, Karpathy's approach automates the most labor-intensive parts while keeping the human in the loop as curator and questioner.

Evolution of Karpathy's AI Approach

This represents the latest step in Karpathy's evolution of how he uses LLMs:

• February 2025: "Vibe Coding" - "Forget the code even exists" — talk to the AI, accept all, ship
• December 2025: "Never felt this behind" - Called it a "magnitude 9 earthquake" for programmers
• January 2026: Agentic Engineering - The disciplined successor: humans write <1% of code, orchestrate AI agents
• April 2026: LLM Knowledge Bases - Beyond code entirely — LLMs compile and maintain knowledge

The throughline is that markdown is becoming the programming language of the AI era, with plain text serving as the interface between humans and AI.