Lenny Rag MCP

🚀 Lenny RAG MCP Server

An MCP server that offers hierarchical RAG over 299 Lenny Rachitsky podcast transcripts. It facilitates product development brainstorming by retrieving relevant insights, real - world examples, and full transcript context.

🚀 Quick Start

Clone the Repository

# Clone the repository (includes pre-built index via Git LFS)
git clone git@github.com:mpnikhil/lenny-rag-mcp.git
cd lenny-rag-mcp

Create and Activate Virtual Environment

# Create and activate virtual environment
python -m venv venv
source venv/bin/activate

Install the Package

# Install the package
pip install -e .

Claude Code

claude mcp add lenny --scope user -- /path/to/lenny-rag-mcp/venv/bin/python -m src.server

Or add to ~/.claude.json:

{
  "mcpServers": {
    "lenny": {
      "type": "stdio",
      "command": "/path/to/lenny-rag-mcp/venv/bin/python",
      "args": ["-m", "src.server"],
      "cwd": "/path/to/lenny-rag-mcp"
    }
  }
}

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "lenny": {
      "command": "/path/to/lenny-rag-mcp/venv/bin/python",
      "args": ["-m", "src.server"],
      "cwd": "/path/to/lenny-rag-mcp"
    }
  }
}

Cursor

Add to .cursor/mcp.json in your project or ~/.cursor/mcp.json globally:

{
  "mcpServers": {
    "lenny": {
      "command": "/path/to/lenny-rag-mcp/venv/bin/python",
      "args": ["-m", "src.server"],
      "cwd": "/path/to/lenny-rag-mcp"
    }
  }
}

⚠️ Important Note

Replace /path/to/lenny-rag-mcp with your actual clone location in all configs.

✨ Features

MCP Tools

search_lenny

Semantic search across the entire corpus. Returns pointers for progressive disclosure.

Returns: Ranked results with relevance scores, episode references, and topic IDs for drilling down.

get_chapter

Load a specific topic with full context. Use after search_lenny to get details.

Returns: Topic summary, all insights, all examples, and raw transcript segment.

get_full_transcript

Load complete episode transcript with metadata.

Returns: Full transcript (10 - 40K tokens), episode metadata, and topic list.

list_episodes

Browse available episodes, optionally filtered by expertise.

Returns: List of 299 episodes with guest names and expertise tags.

🔧 Technical Details

Data Curation Approach

Hierarchical Extraction

Each transcript is processed into a 4 - level hierarchy enabling progressive disclosure:

Episode
├── Topics (10-20 per episode)
│   ├── Insights (2-4 per topic)
│   └── Examples (1-3 per topic)

This allows Claude to start with lightweight search results and drill down only when needed, keeping context windows efficient.

Extraction Schema

{
  "episode": {
    "guest": "Guest Name",
    "expertise_tags": ["growth", "pricing", "leadership"],
    "summary": "150-200 word episode summary",
    "key_frameworks": ["Framework 1", "Framework 2"]
  },
  "topics": [{
    "id": "topic_1",
    "title": "Searchable topic title",
    "summary": "Topic summary",
    "line_start": 1,
    "line_end": 150
  }],
  "insights": [{
    "id": "insight_1",
    "text": "Actionable insight or contrarian take",
    "context": "Additional context",
    "topic_id": "topic_1",
    "line_start": 45,
    "line_end": 52
  }],
  "examples": [{
    "id": "example_1",
    "explicit_text": "The story as told in the transcript",
    "inferred_identity": "Airbnb",
    "confidence": "high",
    "tags": ["marketplace", "growth", "launch strategy"],
    "lesson": "Specific lesson from this example",
    "topic_id": "topic_1",
    "line_start": 60,
    "line_end": 85
  }]
}

Implicit Anchor Detection

Many guests reference companies without naming them ("at my previous company..."). The extraction prompt instructs the model to infer identities based on the guest's background:

• Brian Chesky saying "when we started" → Airbnb (high confidence)
• A marketplace expert saying "one ride - sharing company" → likely Uber/Lyft (medium confidence) This surfaces examples that wouldn't be found by keyword search alone.

Quality Thresholds

Each transcript extraction is validated against minimum thresholds:

Models & Tech Stack

Why Claude Haiku for Preprocessing?

• Quality: Haiku follows complex extraction prompts reliably.
• Cost: ~$0.02 - 0.03 per transcript (~$6 - 9 total for 299 episodes).
• Speed: ~30 seconds per transcript.

Why bge - small - en - v1.5 for Embeddings?

• Performance: Top - tier retrieval quality for its size.
• Efficiency: 384 dimensions, fast inference.
• Local: Runs entirely on CPU, no API calls needed.

📚 Documentation

Corpus Statistics

Rebuilding the Index

The repo includes a pre - built ChromaDB index. To rebuild from scratch:

Reprocess Transcripts (requires Claude CLI)

# Process all unprocessed transcripts
python scripts/preprocess_haiku.py

# Process specific file
python scripts/preprocess_haiku.py --file "Brian Chesky.txt"

# Parallel processing (4 batches of 50)
python scripts/preprocess_haiku.py --limit 50 --offset 0 &
python scripts/preprocess_haiku.py --limit 50 --offset 50 &
python scripts/preprocess_haiku.py --limit 50 --offset 100 &
python scripts/preprocess_haiku.py --limit 50 --offset 150 &

Rebuild Embeddings

# Incremental (only new files)
python scripts/embed.py

# Full rebuild
python scripts/embed.py --rebuild

Project Structure

lenny-rag-mcp/
├── transcripts/           # 299 raw .txt podcast transcripts
├── preprocessed/          # Extracted JSON hierarchy (one per episode)
├── chroma_db/             # Vector embeddings (Git LFS)
├── prompts/
│   └── extraction.md      # Haiku extraction prompt
├── src/
│   ├── server.py          # MCP server & tool definitions
│   ├── retrieval.py       # LennyRetriever class (ChromaDB wrapper)
│   └── utils.py           # File loading utilities
├── scripts/
│   ├── preprocess_haiku.py  # Claude CLI preprocessing
│   └── embed.py             # ChromaDB embedding pipeline
└── pyproject.toml

📄 License

MIT