Vggt Mps

🚀 VGGT-MPS: 3D Vision Agent for Apple Silicon

VGGT-MPS is a 3D vision agent optimized for Apple Silicon with Metal Performance Shaders (MPS). It can transform single or multi - view images into rich 3D reconstructions using the VGGT model.

🍎 VGGT (Visual Geometry Grounded Transformer) optimized for Apple Silicon with Metal Performance Shaders (MPS)

Transform single or multi - view images into rich 3D reconstructions using Facebook Research's VGGT model, now accelerated on M1/M2/M3 Macs.

🚀 Quick Start

Installation Options

Option A: Install from PyPI (Coming Soon)

# Install from PyPI (when published)
pip install vggt-mps

# Download model weights (5GB)
vggt download

Option B: Install from Source with UV (Recommended for Development)

git clone https://github.com/jmanhype/vggt-mps.git
cd vggt-mps

# Install with uv (10 - 100x faster than pip!)
make install

# Or manually with uv
uv pip install -e .

Option C: Traditional pip install from Source

git clone https://github.com/jmanhype/vggt-mps.git
cd vggt-mps

# Create virtual environment
python -m venv vggt-env
source vggt-env/bin/activate

# Install dependencies
pip install -r requirements.txt

2. Download Model Weights

# Download the 5GB VGGT model
vggt download

# Or if running from source:
python main.py download

Or manually download from Hugging Face

3. Test MPS Support

# Test MPS acceleration
vggt test --suite mps

# Or from source:
python main.py test --suite mps

Expected output:

✅ MPS (Metal Performance Shaders) available!
   Running on Apple Silicon GPU
✅ Model weights loaded to mps
✅ MPS operations working correctly!

4. Setup Environment (Optional)

# Copy environment configuration
cp .env.example .env

# Edit .env with your settings
nano .env

✨ Features

🎉 Release v2.0.0

Major Update: Complete packaging overhaul with unified CLI, PyPI - ready distribution, and production - grade tooling!

🎯 Major Changes in v2.0.0

• Unified CLI: New vggt command with subcommands for all operations
• Professional Packaging: PyPI - ready with pyproject.toml, proper src layout
• Web Interface: Gradio UI for interactive 3D reconstruction (vggt web)
• Enhanced Testing: Comprehensive test suite with MPS and sparse attention tests
• Modern Tooling: UV support, Makefile automation, GitHub Actions CI/CD

🚀 Core Features

• MPS Acceleration: Full GPU acceleration on Apple Silicon using Metal Performance Shaders
• ⚡ Sparse Attention: O(n) memory scaling for city - scale reconstruction (100x savings!)
• 🎥 Multi - View 3D Reconstruction: Generate depth maps, point clouds, and camera poses from images
• 🔧 MCP Integration: Model Context Protocol server for Claude Desktop integration
• 📦 5GB Model: Efficient 1B parameter model that runs smoothly on Apple Silicon
• 🛠️ Multiple Export Formats: PLY, OBJ, GLB for 3D point clouds

🎯 What VGGT Does

VGGT reconstructs 3D scenes from images by predicting:

• Depth Maps: Per - pixel depth estimation
• Camera Poses: 6DOF camera parameters
• 3D Point Clouds: Dense 3D reconstruction
• Confidence Maps: Reliability scores for predictions

💻 Usage Examples

Process Images

from src.tools.readme import vggt_quick_start_inference

result = vggt_quick_start_inference(
    image_directory="./tmp/inputs",
    device="mps",  # Use Apple Silicon GPU
    max_images=4,
    save_outputs=True
)

Extract Video Frames

from src.tools.demo_gradio import vggt_extract_video_frames

result = vggt_extract_video_frames(
    video_path="input_video.mp4",
    frame_interval_seconds=1.0
)

Create 3D Scene

from src.tools.demo_viser import vggt_reconstruct_3d_scene

result = vggt_reconstruct_3d_scene(
    images_dir="./tmp/inputs",
    device_type="mps",
    confidence_threshold=0.5
)

📚 Documentation

CLI Commands (v2.0.0)

All functionality is accessible through the unified vggt command:

# Quick demo with sample images
vggt demo

# Demo with kitchen dataset (4 images)
vggt demo --kitchen --images 4

# Process your own images
vggt reconstruct data/*.jpg

# Use sparse attention for large scenes
vggt reconstruct --sparse data/*.jpg

# Export to specific format
vggt reconstruct --export ply data/*.jpg

# Launch interactive web interface
vggt web

# Open on specific port with public link
vggt web --port 8080 --share

# Run comprehensive tests
vggt test --suite all

# Test sparse attention specifically
vggt test --suite sparse

# Benchmark performance
vggt benchmark --compare

# Download model weights
vggt download

From Source (Development)

If running from source without installation:

python main.py demo
python main.py reconstruct data/*.jpg
python main.py web
python main.py test --suite mps
python main.py benchmark --compare

🔧 MCP Server Integration

Add to Claude Desktop

1. Edit ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "vggt-agent": {
      "command": "uv",
      "args": [
        "run",
        "--python",
        "/path/to/vggt-mps/vggt-env/bin/python",
        "--with",
        "fastmcp",
        "fastmcp",
        "run",
        "/path/to/vggt-mps/src/vggt_mps_mcp.py"
      ]
    }
  }
}

2. Restart Claude Desktop

Available MCP Tools

• vggt_quick_start_inference - Quick 3D reconstruction from images
• vggt_extract_video_frames - Extract frames from video
• vggt_process_images - Full VGGT pipeline
• vggt_create_3d_scene - Generate GLB 3D files
• vggt_reconstruct_3d_scene - Multi - view reconstruction
• vggt_visualize_reconstruction - Create visualizations

⚡ Sparse Attention - NEW!

City - scale 3D reconstruction is now possible! We've implemented Gabriele Berton's research idea for O(n) memory scaling.

🎯 Key Benefits

• 100x memory savings for 1000 images
• No retraining required - patches existing VGGT at runtime
• Identical outputs to regular VGGT (0.000000 difference)
• MegaLoc covisibility detection for smart attention masking

🚀 Usage

from src.vggt_sparse_attention import make_vggt_sparse

# Convert any VGGT to sparse in 1 line
sparse_vggt = make_vggt_sparse(regular_vggt, device="mps")

# Same usage, O(n) memory instead of O(n²)
output = sparse_vggt(images)  # Handles 1000+ images!

📊 Memory Scaling

See full results: docs/SPARSE_ATTENTION_RESULTS.md

🔧 Technical Details

MPS Optimizations

• Device Detection: Auto - detects MPS availability
• Dtype Selection: Uses float32 for optimal MPS performance
• Autocast Handling: CUDA autocast disabled for MPS
• Memory Management: Efficient tensor operations on Metal

Model Architecture

• Parameters: 1B (5GB on disk)
• Input: Multi - view images
• Output: Depth, camera poses, 3D points
• Resolution: 518x518 (VGGT), up to 1024x1024 (input)

🐛 Troubleshooting

MPS Not Available

# Check PyTorch MPS support
python -c "import torch; print(torch.backends.mps.is_available())"

Model Loading Issues

# Verify model file
ls -lh repo/vggt/vggt_model.pt
# Should show ~5GB file

Memory Issues

• Reduce batch size
• Lower resolution
• Use CPU fallback

📚 References

• VGGT Paper
• Facebook Research VGGT
• Hugging Face Model

📚 Documentation

• Development Guide - Setting up your dev environment
• Publishing Guide - PyPI release process
• Contributing Guide - How to contribute
• API Documentation - Detailed API reference
• Examples - Code examples and demos

🚀 Release Notes

v2.0.0 (Latest)

• ✨ Unified CLI with vggt command
• 📦 Professional Python packaging (PyPI - ready)
• 🌐 Gradio web interface
• 🧪 Comprehensive test suite
• 🛠️ Modern tooling (UV, Makefile, GitHub Actions)
• 📝 Complete documentation overhaul

See full changelog

🤝 Contributing

We follow a lightweight Git Flow:

• main holds the latest stable release and is protected.
• develop is the default integration branch for day - to - day work.

When contributing:

1. Create your feature branch from develop (git switch develop && git switch -c feature/my - change).
2. Keep commits focused and include tests or documentation updates when relevant.
3. Open your pull request against develop; maintainers will promote changes to main during releases.

Please open issues for bugs or feature requests before starting large efforts. Full details, testing expectations, and the release process live in .

📄 License

MIT License - See LICENSE file for details

🙏 Acknowledgments

• Facebook Research for VGGT
• Apple for Metal Performance Shaders
• PyTorch team for MPS backend

Made with 🍎 for Apple Silicon by the AI community