Tokenscope

🚀 Token Scope

TokenScope is a tool for efficiently analyzing codebases and interacting with Large Language Models (LLMs). It combines FastMCP and tiktoken to perform efficient context analysis of codebases through LLMs.

🚀 Quick Start

Here is a simple usage example:

from tokenscope import create_client

# Create a client
client = create_client()

# Scan the directory structure
structure = client.scan_directory_structure("path/to/directory")

# Extract file content
content = client.extract_file_content("path/to/file.py")

# Analyze token usage
analysis = client.analyze_token_usage("path/to/directory")

print(structure)
print(content)
print(analysis)

✨ Features

• Directory Structure Scanning: Scan and return the directory structure in a token-efficient manner.
• File Content Extraction: Extract the content of specific files, considering token limits and formats.
• File Search: Search for files in the directory structure according to specified patterns.
• Token Usage Analysis: Analyze the token usage of directories or files and estimate LLM processing requirements.
• Report Generation: Generate a comprehensive Markdown report containing directory statistics.
• File Copying: Copy files from the source path to the destination path.

📦 Installation

Install via pip

pip install token-scope

Clone the repository and install

git clone https://github.com/yourusername/token-scope.git
cd token-scope
python -m pip install .

💻 Usage Examples

Basic Usage

The basic usage has been shown in the "Quick Start" section above.

Advanced Usage

Here are detailed descriptions of each API:

scan_directory_structure

Scan a directory and return its structure, considering token limits.

scan_directory_structure(
    path: str, 
    depth: int = 3,
    max_tokens: int = 10000,
    ignore_patterns: list[str] | None = None,
    include_gitignore: bool = True,
    include_default_ignores: bool = True
)

extract_file_content

Extract the content of a specific file, considering token limits and formats.

extract_file_content(
    file_path: str, 
    max_tokens: int = 10000,
    sample_only: bool = False
)

search_files_by_pattern

Search for files in a directory according to specified patterns.

search_files_by_pattern(
    directory: str,
    patterns: list[str],
    max_depth: int = 5,
    include_content: bool = False,
    max_files: int = 100,
    max_tokens_per_file: int = 1000,
    sample_only: bool = False,
    ignore_patterns: list[str] | None = None,
    include_gitignore: bool = True,
    include_default_ignores: bool = True
)

analyze_token_usage

Analyze the token usage of a specified path.

analyze_token_usage(
    path: str,
    include_file_details: bool = False,
    ignore_patterns: list[str] | None = None,
    include_gitignore: bool = True,
    include_default_ignores: bool = True
)

generate_directory_report

Generate a Markdown report containing statistics.

generate_directory_report(
    directory: str, 
    depth: int = 3,
    include_file_content: bool = True,
    max_files_with_content: int = 5,
    max_tokens_per_file: int = 1000,
    sample_only: bool = False,
    ignore_patterns: list[str] | None = None,
    include_gitignore: bool = True,
    include_default_ignores: bool = True
)

copy_file_to_destination

Copy a file to a specified path.

copy_file_to_destination(
    source_path: str,
    destination_path: str
)

📚 Documentation

Default Ignore Patterns

TokenScope automatically ignores the following common directories and files:

DEFAULT_IGNORE_PATTERNS = [
    ".git/",
    ".venv/",
    "venv/",
    "__pycache__/",
    "node_modules/",
    ".pytest_cache/",
    ".ipynb_checkpoints/",
    ".DS_Store",
    "*.pyc",
    "*.pyo",
    "*.pyd",
    "*.so",
    "*.dll",
    "*.class",
    "build/",
    "dist/",
    "*.egg-info/",
    ".tox/",
    ".coverage",
    ".idea/",
    ".vscode/",
    ".mypy_cache/",
]

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Combined with FastMCP
• Uses tiktoken for tokenization