Token Optimizer MCP

🚀 Token Optimizer MCP

Intelligent token optimization through caching, compression, and smart tooling for Claude Code and Claude Desktop

🚀 Quick Start

Token Optimizer MCP is a Model Context Protocol (MCP) server that reduces context window usage by 60 - 90% through intelligent caching, compression, and smart tool replacements. By storing compressed content externally in SQLite and providing optimized alternatives to standard tools, the server helps you maximize your available context window.

Production Results: Achieved 60 - 90% token reduction across 38,000+ operations in real - world usage.

✨ Features

• Smart Tool Replacements: Automatic optimization for Read, Grep, Glob, and more.
• Context Window Optimization: Store content externally to free up context space.
• High Compression: Brotli compression (2 - 4x typical, up to 82x for repetitive content).
• Persistent Caching: SQLite - based cache that persists across sessions.
• Accurate Token Counting: Uses tiktoken for precise token measurements.
• 61 Specialized Tools: File operations, API caching, database optimization, monitoring, and more.
• Zero External Dependencies: Completely offline operation.
• Production Ready: Built with TypeScript for reliability.

📦 Installation

Quick Install (Recommended)

Windows

# Run PowerShell as Administrator, then:
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

# Install globally (hooks install automatically!)
npm install -g @ooples/token-optimizer-mcp

macOS / Linux

# Install globally (hooks install automatically!)
npm install -g @ooples/token-optimizer-mcp

That's it! The postinstall script will automatically:

1. ✅ Install token-optimizer-mcp globally via npm.
2. ✅ Auto-detect and configure all installed AI tools (Claude Desktop, Cursor, Cline, etc.).
3. ✅ Set up automatic token optimization on every tool call.
4. ✅ Configure workspace trust and execution permissions.

Result: 60 - 90% token reduction across all operations!

Note: If automatic installation is skipped (e.g., in CI environments), you can manually run the installer:

• Windows: powershell -ExecutionPolicy Bypass -File install-hooks.ps1
• macOS/Linux: bash install-hooks.sh

Manual Configuration

For detailed platform-specific installation instructions, see docs/HOOKS-INSTALLATION.md.

💻 Usage Examples

Basic Caching

// Cache large content to remove from context window
const result = await optimize_text({
  text: "Large API response or file content...",
  key: "cache-key",
  quality: 11
});
// Result: Original tokens removed, only cache key remains (~50 tokens)

// Retrieve later
const cached = await get_cached({ key: "cache-key" });
// Result: Full original content restored

Smart File Reading

// First read: full content
await smart_read({ path: "/src/app.ts" });

// Subsequent reads: only changes (80% reduction)
await smart_read({ path: "/src/app.ts" });

API Caching

// First request: fetch and cache
await smart_api_fetch({
  method: "GET",
  url: "https://api.example.com/data",
  ttl: 300
});

// Subsequent requests: cached (95% reduction)
await smart_api_fetch({
  method: "GET",
  url: "https://api.example.com/data"
});

Session Analysis

// View token usage for current session
await get_session_stats({});
// Result: Breakdown by tool, operation, and savings

// Analyze entire project
await analyze_project_tokens({
  projectPath: "/path/to/project"
});
// Result: Cost estimation and optimization opportunities

📚 Documentation

• Detailed Tool Reference - Complete documentation for all 61 tools.
• Installation Guide - Platform-specific installation instructions.
• Contributing Guide - Development setup and contribution guidelines.

🔧 Technical Details

Available Tools (65 Total)

Core Caching & Optimization (8 tools)

// Cache large content to remove it from context window
optimize_text({
  text: "Large API response or file content...",
  key: "api-response-key",
  quality: 11
})
// Result: 60 - 90% token reduction

Smart File Operations (10 tools)

// Read a file with automatic caching
smart_read({ path: "/path/to/file.ts" })
// First read: full content
// Subsequent reads: only diff (80% reduction)

API & Database Operations (10 tools)

// Fetch API with automatic caching
smart_api_fetch({
  method: "GET",
  url: "https://api.example.com/data",
  ttl: 300
})
// Cached responses: 95% token reduction

Build & Test Operations (10 tools)

// Run tests with caching
smart_test({
  onlyChanged: true,  // Only test changed files
  coverage: true
})

Advanced Caching (10 tools)

// Configure multi-tier cache
smart_cache({
  operation: "configure",
  evictionStrategy: "LRU",
  l1MaxSize: 1000,
  l2MaxSize: 10000
})

Monitoring & Dashboards (7 tools)

// Create an alert
alert_manager({
  operation: "create-alert",
  alertName: "high-cpu-usage",
  channels: ["slack", "email"],
  threshold: { type: "above", value: 80 }
})

System Operations (6 tools)

// View session token usage
get_session_stats({})
// Result: Detailed breakdown of token usage by tool

How It Works

Token Analytics (4 tools)

// Get per-hook analytics
get_hook_analytics({
  startDate: "2025-01-01T00:00:00Z",
  endDate: "2025-12-31T23:59:59Z"
})
// Result: Shows which hooks consume the most tokens

// Get per-action analytics
get_action_analytics({})
// Result: Shows which tools use the most tokens

// Export analytics as CSV
export_analytics({
  format: "csv",
  hookPhase: "PreToolUse"
})
// Result: CSV export filtered by PreToolUse hook

Global Hooks System (7-Phase Optimization)

When global hooks are installed, token-optimizer-mcp runs automatically on every tool call:

┌─────────────────────────────────────────────────────────────┐
│ Phase 1: PreToolUse - Tool Replacement                      │
│ ├─ Read   → smart_read   (80% token reduction)             │
│ ├─ Grep   → smart_grep   (80% token reduction)             │
│ └─ Glob   → smart_glob   (75% token reduction)             │
└─────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────┐
│ Phase 2: Input Validation - Cache Lookups                   │
│ └─ get_cached checks if operation was already done          │
└─────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────┐
│ Phase 3: PostToolUse - Output Optimization                  │
│ ├─ optimize_text for large outputs                          │
│ └─ compress_text for repeated content                       │
└─────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────┐
│ Phase 4: Session Tracking                                   │
│ └─ Log all operations to operations-{sessionId}.csv         │
└─────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────┐
│ Phase 5: UserPromptSubmit - Prompt Optimization             │
│ └─ Optimize user prompts before sending to API              │
└─────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────┐
│ Phase 6: PreCompact - Pre-Compaction Optimization           │
│ └─ Optimize before Claude Code compacts the conversation    │
└─────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────┐
│ Phase 7: Metrics & Reporting                                │
│ └─ Track token reduction metrics and generate reports       │
└─────────────────────────────────────────────────────────────┘

Production Performance

Based on 38,000+ operations in real-world usage:

Per-Session Savings: 300K - 700K tokens (worth $0.90 - $2.10 at $3/M tokens).

Technology Stack

• Runtime: Node.js 20+
• Language: TypeScript
• Database: SQLite (better-sqlite3)
• Token Counting: tiktoken (GPT - 4 tokenizer)
• Compression: Brotli (built-in Node.js)
• Caching: Multi-tier LRU/LFU/FIFO caching
• Protocol: MCP SDK (@modelcontextprotocol/sdk)

Supported AI Tools

The automated installer detects and configures token-optimizer-mcp for:

• ✅ Claude Code - CLI with global hooks integration
• ✅ Claude Desktop - Native desktop application
• ✅ Cursor IDE - AI-first code editor
• ✅ Cline - VS Code extension (formerly Claude Dev)
• ✅ GitHub Copilot - VS Code with MCP support
• ✅ Windsurf IDE - AI-powered development environment

No manual configuration needed - the installer automatically detects and configures all installed tools!

Performance Characteristics

• Compression Ratio: 2 - 4x typical (up to 82x for repetitive content).
• Context Window Savings: 60 - 90% average across all operations.
• Cache Hit Rate: >80% in typical usage.
• Operation Overhead: <10ms for cache operations (optimized from 50 - 70ms).
• Compression Speed: ~1ms per KB of text.
• Hook Overhead: <10ms per operation (7x improvement from in-memory optimizations).

Performance Optimizations

The PowerShell hooks have been optimized to reduce overhead from 50 - 70ms to <10ms through:

• In-Memory Session State: Session data kept in memory instead of disk I/O on every operation.
• Batched Log Writes: Operation logs buffered and flushed every 5 seconds or 100 operations.
• Lazy Persistence: Disk writes only occur when necessary (session end, optimization, reports).

Environment Variables

Control hook behavior with these environment variables:

Performance Controls

• TOKEN_OPTIMIZER_USE_FILE_SESSION (default: false)

  • Set to true to revert to file-based session tracking (legacy mode).
  • Use if you encounter issues with in-memory session state.
  • Example: $env:TOKEN_OPTIMIZER_USE_FILE_SESSION = "true"

• TOKEN_OPTIMIZER_SYNC_LOG_WRITES (default: false)

  • Set to true to disable batched log writes.
  • Forces immediate writes to disk (slower but more resilient).
  • Use for debugging or if logs are being lost.
  • Example: $env:TOKEN_OPTIMIZER_SYNC_LOG_WRITES = "true"

• TOKEN_OPTIMIZER_DEBUG_LOGGING (default: true)

  • Set to false to disable DEBUG-level logging.
  • Reduces log file size and improves performance.
  • INFO/WARN/ERROR logs still written.
  • Example: $env:TOKEN_OPTIMIZER_DEBUG_LOGGING = "false"

Development Path

• TOKEN_OPTIMIZER_DEV_PATH
  • Path to local development installation.
  • Automatically set to ~/source/repos/token-optimizer-mcp if not specified.
  • Override for custom development paths.
  • Example: $env:TOKEN_OPTIMIZER_DEV_PATH = "C:\dev\token-optimizer-mcp"

Performance Impact: Using in-memory mode (default) provides a 7x improvement in hook overhead:

• Before: 50 - 70ms per hook operation.
• After: <10ms per hook operation.
• 85% reduction in hook latency.

Monitoring Token Savings

Real-Time Session Monitoring

To view your actual token SAVINGS, use the get_session_stats tool:

// View current session statistics with token savings breakdown
await get_session_stats({});

Output includes:

• Total tokens saved (this is the actual savings amount!)
• Token reduction percentage (e.g., "60% reduction")
• Cache hit rate and compression ratios
• Breakdown by tool (Read, Grep, Glob, etc.)
• Top 10 most optimized operations with before/after comparison

Example Output:

{
  "sessionId": "abc-123",
  "totalTokensSaved": 125430,  // ← THIS is your savings!
  "tokenReductionPercent": 68.2,
  "originalTokens": 184000,
  "optimizedTokens": 58570,
  "cacheHitRate": 72.0,
  "byTool": {
    "smart_read": { "saved": 45000, "percent": 80 },
    "smart_grep": { "saved": 32000, "percent": 75 }
  }
}

Session Tracking Files

All operations are automatically tracked in session data files:

Location: ~/.claude-global/hooks/data/current-session.txt

Format:

{
  "sessionId": "abc-123",
  "sessionStart": "20251031-082211",
  "totalOperations": 1250,      // ← Number of operations
  "totalTokens": 184000,         // ← Cumulative token COUNT
  "lastOptimized": 1698765432,
  "savings": {                   // ← Auto-updated every 10 operations (Issue #113)
    "totalTokensSaved": 125430,  // Tokens saved by compression
    "tokenReductionPercent": 68.2,  // Percentage of tokens saved
    "originalTokens": 184000,    // Original token count before optimization
    "optimizedTokens": 58570,    // Token count after optimization
    "cacheHitRate": 42.5,        // Cache hit rate percentage
    "compressionRatio": 0.32,    // Compression efficiency (lower is better)
    "lastUpdated": "20251031-092500"  // Last savings update timestamp
  }
}

New in v1.x: The savings object is now automatically updated every 10 operations, eliminating the need to manually call get_session_stats() for real-time monitoring. This provides instant visibility into token optimization performance.

How it works:

• Every 10 operations, the PowerShell hooks automatically call get_cache_stats() MCP tool.
• Savings metrics are calculated from cache performance data (compression ratio, original vs compressed sizes).
• The session file is atomically updated with the latest savings data.
• If the MCP call fails, the update is skipped gracefully without blocking operations.

Note: For detailed per-operation analysis, use get_session_stats(). The session file provides high-level aggregate metrics.

Project-Wide Analysis

Analyze token usage across your entire project:

// Analyze project token costs
await analyze_project_tokens({
  projectPath: "/path/to/project"
});

Provides:

• Total token cost estimation.
• Largest files by token count.
• Optimization opportunities.
• Cost projections at current API rates.

Cache Performance

Monitor cache hit rates and storage efficiency:

// View cache statistics
await get_cache_stats({});

Metrics:

• Total entries.
• Cache hit rate (%).
• Average compression ratio.
• Total storage saved.
• Most frequently accessed keys.

Troubleshooting

Common Issues and Solutions

Issue: "Invalid or malformed JSON" in Claude Code Settings

Symptom: Claude Code shows "Invalid Settings" error after running install-hooks.

Cause: UTF - 8 BOM (Byte Order Mark) was added to settings.json files.

Solution: Upgrade to v3.0.2+ which fixes the BOM issue:

npm install -g @ooples/token-optimizer-mcp@latest

If you're already on v3.0.2+, manually remove the BOM:

# Windows: Remove BOM from settings.json
$content = Get-Content "~/.claude/settings.json" -Raw
$content = $content -replace '^\xEF\xBB\xBF', ''
$content | Set-Content "~/.claude/settings.json" -Encoding utf8NoBOM

# Linux: Remove BOM from settings.json
sed -i '1s/^\xEF\xBB\xBF//' ~/.claude/settings.json

# macOS: Remove BOM from settings.json (BSD sed requires empty string after -i)
sed -i '' '1s/^\xef\xbb\xbf//' ~/.claude/settings.json

Issue: Hooks Not Working After Installation

Symptom: Token optimization not occurring automatically.

Diagnosis:

1. Check if hooks are installed:

   # Windows
   Get-Content ~/.claude/settings.json | ConvertFrom-Json | Select-Object -ExpandProperty hooks
   

   # macOS/Linux
   cat ~/.claude/settings.json | jq .hooks
   

2. Verify dispatcher.ps1 exists:

   # Windows
   Test-Path ~/.claude-global/hooks/dispatcher.ps1
   

   # macOS/Linux
   [ -f ~/.claude-global/hooks/dispatcher.sh ] && echo "Exists" || echo "Missing"
   

Solution: Re-run the installer:

# Windows
powershell -ExecutionPolicy Bypass -File install-hooks.ps1

# macOS/Linux
bash install-hooks.sh

Issue: Low Cache Hit Rate (<50%)

Symptom: Session stats show cache hit rate below 50%.

Causes:

1. Working with many new files (expected).
2. Cache was recently cleared.
3. TTL (time-to-live) is too short.

Solutions:

1. Warm up the cache before starting work:

   await cache_warmup({
     paths: ["/path/to/frequently/used/files"],
     recursive: true
   });
   

2. Increase TTL for stable APIs:

   await smart_api_fetch({
     url: "https://api.example.com/data",
     ttl: 3600  // 1 hour instead of default 5 minutes
   });
   

3. Check cache size limits:

   await smart_cache({
     operation: "configure",
     l1MaxSize: 2000,  // Increase from default 1000
     l2MaxSize: 20000  // Increase from default 10000
   });
   

Issue: High Memory Usage

Symptom: Node.js process using excessive memory.

Cause: Large cache in memory (L1/L2 tiers).

Solution: Configure cache limits:

await smart_cache({
  operation: "configure",
  evictionStrategy: "LRU",  // Least Recently Used
  l1MaxSize: 500,  // Reduce L1 cache
  l2MaxSize: 5000  // Reduce L2 cache
});

Or clear the cache:

await clear_cache({});

Issue: Slow First-Time Operations

Symptom: Initial Read/Grep/Glob operations are slow.

Cause: Cache is empty, building indexes.

Solution: This is expected behavior. Subsequent operations will be 80 - 90% faster.

To pre-warm the cache:

await cache_warmup({
  paths: ["/src", "/tests", "/docs"],
  recursive: true,
  schedule: "startup"  // Auto-warm on every session start
});

Issue: "Permission denied" Errors on Windows

Symptom: Cannot write to cache or log files.

Cause: PowerShell execution policy or file permissions.

Solution:

1. Set execution policy:

   Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
   

2. Check file permissions:

   icacls "$env:USERPROFILE\.token-optimizer"
   

3. Re-run installer as Administrator if needed

Issue: Cache Files Growing Too Large

Symptom: ~/.token-optimizer/cache.db is >1GB.

Cause: Caching very large files or many API responses.

Solution:

1. Clear old entries:

   await clear_cache({ olderThan: 7 });  // Clear entries older than 7 days
   

2. Reduce cache retention:

   await smart_cache({
     operation: "configure",
     defaultTTL: 3600  // 1 hour instead of 7 days
   });
   

3. Manually delete cache (nuclear option):

   rm -rf ~/.token-optimizer/cache.db
   

Getting Help

If you encounter issues not covered here:

1. Check the hook logs: ~/.claude-global/hooks/logs/dispatcher.log
2. Check session data: ~/.claude-global/hooks/data/current-session.txt
3. File an issue: GitHub Issues
   • Include debug logs.
   • Include your OS and Node.js version.
   • Include the output of get_session_stats

Limitations

• Small Text: Best for content >500 characters (cache overhead on small snippets).
• One-Time Content: No benefit for content that won't be referenced again.
• Cache Storage: Automatic cleanup after 7 days to prevent disk usage issues.
• Token Counting: Uses GPT - 4 tokenizer (approximation for Claude, but close enough).

📄 License

MIT License - see LICENSE for details

Author

Built for optimal Claude Code token efficiency by the ooples team.