Website To Markdown Converter

🚀 Website Scraping Tool

A command-line tool and MCP server for scraping websites and converting HTML to Markdown.

✨ Features

• Extracts meaningful content from web pages using Mozilla's Readability library (the same engine used in Firefox's Reading View).
• Converts clean HTML to high-quality Markdown using TurndownService.
• Handles HTML securely by removing potentially harmful script tags.
• Can be run as a command-line tool or an MCP server.
• Supports direct conversion of local HTML files to Markdown.

📦 Installation

# Install dependencies
npm install

# Build the project
npm run build

# Optional: Install globally
npm install -g .

💻 Usage Examples

Basic Usage

# Output results in the console
scrape https://example.com

# Save results to a file
scrape https://example.com output.md

# Directly convert a local HTML file to Markdown
scrape --html-file input.html

# Convert a local HTML file and save the results
scrape --html-file input.html output.md

# Display help information
scrape --help

# Or run via npm script
npm run start:cli -- https://example.com

Advanced Usage

This tool can be used as a Model Context Protocol (MCP) server:

# Start in MCP server mode
npm start

📚 Documentation

Code Structure

📄 API

The tool exports the following functions:

// Scrape a website and convert it to Markdown
import { scrapeToMarkdown } from './build/index.js';

// Directly convert an HTML string to Markdown
import { htmlToMarkdown } from './build/data_processing.js';

async function example() {
  // Website scraping
  const markdown = await scrapeToMarkdown('https://example.com');
  console.log(markdown);
  
  // Direct HTML conversion
  const html = '<h1>Hello, world!</h1><p>This is <strong>bold</strong> text.</p>';
  const md = htmlToMarkdown(html);
  console.log(md);
}

📄 License

ISC