PT Emails — Real-time Email Pipeline
Event-driven streaming pipeline that classifies, creates drafts, and enriches every customer email in real-time
In Action
Authorized Mailboxes
Per Mailbox Statistics
Email Agent Tasks
Pipeline Overview
Ingest & Process
Fetch, Enrich, Classify, Draft
6 stages
AI Classification
Category, Priority, Sentiment, Action
3 AI models
Output & Distribute
Labels, Drafts, Pinecone, Monday
4 destinations
RedPanda · Event-driven · Real-time
Trigger — Gmail Push Notifications
GCP Pub/Sub Webhook
POST /internal/produce-raw
Gmail watches push real-time notifications via GCP Pub/Sub. Webhook checks pipeline_enabled per mailbox before forwarding — disabled mailboxes are silently dropped. Serializes to Avro and publishes to RedPanda.
Gmail Gateway
FastAPI · port 8507
Internal microservice that handles Gmail API access, thread content fetching, content cleaning, summarization, and metadata tracking via PostgreSQL.
Mailbox Management
Multi-Mailbox OAuth
Add any number of Gmail mailboxes via the admin dashboard — each completes Google OAuth2 flow independently. Tokens encrypted at rest (Fernet) and auto-refreshed on expiry.
care@
sales@
support@
Smart Filtering
Exclusion Layers
Excluded senders per mailbox (exact + @domain)
Corporate & internal emails auto-detected
Gmail SPAM auto-skipped
Unsent DRAFTs silently filtered
Pipeline Stages
1
History
Processes Gmail push notifications via Gateway history endpoint. Filters for new messages, routes SENT messages to fast-path, and produces INBOX messages to the fetch stage.
2
Fetch
Fetches full thread content from Gmail Gateway with summarization. Cleans HTML, filters SPAM/DRAFT, detects corporate and excluded senders, extracts all sender emails.
3
Enrich
hubspot-api
Looks up each customer in HubSpot via unified /lookup endpoint. Generates LLM customer summary with deal history, collects suggested labels from orders, RDM cases, and payment status.
4
Classify
gpt-4.1-mini
Classifies email using OpenAI Structured Outputs: category, priority, sentiment, action required, and follow-up type. Merges programmatic labels with HubSpot suggested labels.
5
Draft
gpt-5-mini
Composes context-aware draft replies using customer history, classification, and HubSpot data. Creates threaded Gmail drafts with proper In-Reply-To headers. Skips when no reply is needed.
6
Complete
Applies Gmail labels (4–13 per thread), handles conflicting label removal, creates labels with color coding via Gmail API, and marks the pipeline as done.
Parallel Outputs
Pinecone
text-embedding-3-small
Generates embeddings and upserts to Pinecone for semantic email search. Filters spam and low-quality emails. Updates vectors when threads evolve or replies are sent.
Monday Tasks
monday-api
Creates follow-up tasks on Monday.com boards with customer details, HubSpot links, Gmail thread links, and RDM case references. Auto-marks tasks Done when reply is sent.
RedPanda · Avro · Schema Registry · 7 topics
Label System — Classification Categories
Programmatic Labels
4 labels always applied — one from each dimension
Category
order
delivery
support
complaint
inquiry
feedback
spam
other
Priority
High
Normal
Low
Sentiment
Angry
Neutral
Positive
Action
Reply Required
Follow-up
FYI
Completed
System Labels
Gmail native labels preserved from API
INBOX
SENT
IMPORTANT
STARRED
SPAM
TRASH
DRAFT
Customer Data Labels
0–9 contextual labels based on CRM customer data
Customer
Existing
Prospect
High-Value
Repeat
Lead Type
Pre-Sale
Post-Sale
Complaint
Delivery
Active Sales
Support
RDM
Damaged
Missing
Incorrect
Transit
Quality
Mixed
Investigation
Status
Delivery/Late
Payment/Pending
Order/Refunded
up to 42 labels · 4 minimum · 6–8 typical · 13 maximum per thread
Sent Message Fast-Path
SENT Detection
stage-sent-mail-complete
When a team member sends a reply, Gmail push notification triggers. History consumer detects the SENT label and routes directly to the fast-path topic — bypassing the full pipeline.
Auto-Resolution
Monday consumer marks the follow-up task as Done. Pinecone consumer replaces the draft text with actual sent content. Complete consumer applies the Action/Completed label to the Gmail thread.
External Services & Infrastructure
Gmail API
email
GCP Pub/Sub
trigger
OpenAI
AI models
HubSpot
CRM
Pinecone
vector DB
Monday.com
project mgmt
RedPanda
streaming
PostgreSQL
metadata
OpenTelemetry
observability
Streaming & Resilience
RedPanda
Kafka-compatible
7 pipeline topics + 6 dead-letter queues. Avro serialization with Schema Registry. Manual offset commits — failed messages retry with exponential backoff (max 3, 5s–300s). Poison messages route to per-stage DLQ.
7 topics
6 DLQ
Avro
3 retries
Backfill & Reprocessing
Idempotent
Reprocess historical emails by date range, mailbox, and starting stage. Force mode clears existing metadata for full re-run. Per-stage idempotency — re-runs safely skip completed messages unless forced.
date range
per-stage
force mode
Deployment & Infrastructure
AWS EC2
Docker
GitHub Actions
Runs alongside the agent platform and RedPanda cluster on the same machine. Docker container with Python 3.11 runtime. CI/CD via GitHub Actions with auto-deploy on push.
t3.xlarge
16 GB
4 vCPU
PostgreSQL Metadata
Per-Stage Tracking
Tracks every email through each pipeline stage with processing times, error messages, and retry counts. Customer context caching, classification results, and vector metadata for analytics.
timber_gmail
postgres:5432
Pipeline Summary
3
mailboxes
8
pipeline stages
3
AI models
42
Gmail labels
13
topics + DLQ
real-time
event-driven