Lilith Emails

🚀 Lilith Email System

A Gmail sync daemon combined with Lilith agent tools for semantic email search, featuring privacy - aware classification and PII sanitization.

🚀 Quick Start

1. Database (shared Postgres)

This project utilizes a shared PostgreSQL server. The database name for this application is lilith_emails.

Ensure that the shared Postgres (with pgvector) is up and running. First, clone the lilith - compose project.

2. Run migrations

uv run alembic upgrade head

3. Add a Gmail account

Download OAuth client secrets from Google Cloud Console, then:

uv run python main.py add-account path/to/client_secrets.json

4. Sync (download only)

uv run python main.py sync 1

The logs will display the progress, including the number of pages fetched, messages stored, and the total so far.

5. Transform (classify + sanitize + embed)

Run this after the sync process to generate privacy_tier, body_redacted, and multi - level embeddings (subject, body or chunks) from the stored data. You can re - run this anytime you change the models or logic (no need to re - download).

uv run python main.py transform 1

To clean all derived columns added by the transform command:

uv run python main.py reset-transform 1

6. Run the sync daemon (Pub/Sub webhook)

uv run python main.py serve

When the daemon receives a Gmail Pub/Sub push, it will automatically perform an incremental sync and then a transform for that account.

Without a public URL (local dev): Use pull instead of push. Create a pull subscription, set PUBSUB_SUBSCRIPTION in .env, then run:

gcloud auth application-default login
uv run python main.py watch 1
# In another terminal, poll for notifications (same sync+transform as webhook):
uv run python main.py pull

Create the pull subscription (in the same project as the topic):
gcloud pubsub subscriptions create lilith-emails-pull --topic=gmail-topic --project=lilithsync

With a public URL: Use a push subscription (endpoint = your public /webhook/gmail URL) and run the daemon with uv run python main.py serve. Register the watch once: uv run python main.py watch <account_id> (requires GOOGLE_CLOUD_PROJECT and PUBSUB_TOPIC in .env).

If watch returns 403: Grant Gmail permission to publish to your topic:

gcloud pubsub topics add-iam-policy-binding gmail-topic \
  --member="serviceAccount:gmail-api-push@system.gserviceaccount.com" \
  --role="roles/pubsub.publisher" \
  --project=lilithsync

Testing the webhook locally

You can trigger the same path without Gmail by POSTing a simulated Pub/Sub payload. First, run a full sync so the account has last_history_id, then start the daemon and send:

# Start daemon in another terminal: uv run python main.py serve --port 8000

# Replace YOUR_EMAIL and HISTORY_ID (e.g. from DB: email_accounts.last_history_id)
# Portable (any OS):
python3 -c "
import base64, json, urllib.request
d = base64.b64encode(json.dumps({'emailAddress':'YOUR_EMAIL','historyId':'HISTORY_ID'}).encode()).decode()
urllib.request.urlopen(urllib.request.Request('http://localhost:8000/webhook/gmail', data=json.dumps({'message':{'data':d}}).encode(), headers={'Content-Type':'application/json'}, method='POST'))
print('OK')
"

Or with curl (Linux: use base64 -w0; macOS: use base64):

B64=$(echo -n '{"emailAddress":"YOUR_EMAIL","historyId":"HISTORY_ID"}' | base64)
curl -s -X POST http://localhost:8000/webhook/gmail -H "Content-Type: application/json" -d "{\"message\":{\"data\":\"$B64\"}}"

The daemon will perform an incremental sync and then a transform for that account. Use get-email or MCP tools to verify new or updated rows.

📚 Documentation

Configuration

Environment variables (.env or shell):

Transform uses capabilities.json: run uv run python main.py capabilities before transform so the file exists and has embedding.max_tokens, vllm.model_id, spacy_api.available, and fasttext_langdetect.available. There is no env fallback for transform. Emails with transform_completed_at set are skipped unless you use --force (which prompts for confirmation); if transform fails mid - run, those emails are retried next time.

MCP Server (Agent Tools)

The Lilith Email MCP server exposes your transformed Gmail.

uv run mcp
uv run mcp --transport streamable-http --port 6201

MCP Tools

All responses use external privacy: SENSITIVE content is redacted, PERSONAL content is shown sanitized.