Technical Architecture

Dependency Layers

┌─────────────┐
│   Frontend  │  Next.js + React
├─────────────┤
│  API Layer  │  FastAPI routes (thin handlers)
├─────────────┤
│  Services   │  Business logic, follow-up engine, scoring
├─────────────┤
│Integrations │  Gmail, Telegram, Twitter, LinkedIn, Apollo
├─────────────┤
│   Models    │  SQLAlchemy ORM + Pydantic schemas
├─────────────┤
│    Core     │  Config, auth, database, Redis, logging
└─────────────┘

Allowed Import Direction

api/ → services/ → integrations/ → models/ → core/
Never: integrations/ → api/ or services/ → api/

Module Ownership

Directory	Responsibility
`api/`	HTTP request handling, validation, response formatting
`services/`	Business logic, orchestration, no HTTP concerns
`integrations/`	External API clients, protocol adapters
`models/`	Database schema, relationships
`schemas/`	Pydantic request/response models
`core/`	Cross-cutting: config, auth, database, logging, middleware
`task_jobs/`	Celery task definitions, organized by domain

Key Files by Size

File	Lines	Role
`integrations/telegram.py`	~300	Telegram DM/bio sync
`integrations/telegram_transport.py`	~85	Telethon client lifecycle
`integrations/telegram_helpers.py`	~150	Contact resolution helpers
`integrations/telegram_groups.py`	~250	Group member sync
`integrations/twitter.py`	~470	Twitter bio and activity polling, OAuth token refresh
`integrations/twitter_dms.py`	~435	Twitter DM sync
`integrations/twitter_auth.py`	~200	OAuth 2.0 PKCE token management
`services/followup_engine.py`	~490	AI follow-up suggestion generation
`services/scoring.py`	~220	Relationship score calculation

Telegram sync logic is also split across telegram_bio_sync.py, telegram_chat_sync.py, and telegram_chat_batch.py.

System Overview

PingCRM has three tiers. A Next.js frontend talks to a FastAPI backend, which stores data in PostgreSQL and uses Redis for caching, task brokering, and short-lived state. Celery workers run background sync, scoring, and AI suggestion generation.

┌─────────────────┐       ┌──────────────────────┐       ┌────────────────┐
│   Next.js 16    │──────>│   FastAPI (async)     │──────>│  PostgreSQL    │
│   React 19      │ REST  │   Pydantic schemas    │  ORM  │  UUID PKs      │
│   Tailwind v4   │<──────│   JWT auth            │<──────│  GIN indexes   │
└─────────────────┘       └──────────┬───────────┘       └────────────────┘
                                     │
                          ┌──────────▼───────────┐
                          │   Redis              │
                          │   - Celery broker    │
                          │   - Rate gates       │
                          │   - Tweet cache      │
                          │   - OAuth nonces     │
                          └──────────┬───────────┘
                                     │
                          ┌──────────▼───────────┐
                          │   Celery Workers     │
                          │   - Gmail sync       │
                          │   - Telegram sync    │
                          │   - Twitter polling  │
                          │   - Score refresh    │
                          │   - AI suggestions   │
                          └──────────────────────┘

The backend also talks to the Gmail and Google Calendar APIs (OAuth), Telegram MTProto (Telethon), Twitter/X (the Bird CLI with per-user cookies, plus the Twitter API v2 with OAuth 2.0 PKCE), the Anthropic Claude API for event classification and message composition, a WhatsApp sidecar, and a Chrome extension that pushes LinkedIn data.

Backend Architecture

The FastAPI application lives under backend/app/ and follows a layered structure:

backend/app/
├── main.py              # FastAPI app, CORS, middleware, router mounting, MCP mount
├── api/                 # Route handlers (REST endpoints)
│   ├── contacts_routes/ # Contacts: CRUD, imports, sync, taxonomy, messaging
│   ├── interactions.py  # Interaction timeline
│   ├── suggestions.py   # Follow-up suggestions, snooze, dismiss
│   ├── identity.py      # Identity resolution scan, review, merge
│   ├── organizations.py # Company records and stats
│   ├── notifications.py # In-app notification feed
│   ├── auth.py          # Login, register
│   ├── auth_google.py   # Google OAuth connect and callback
│   ├── telegram.py      # Connect, sync, send messages
│   ├── twitter.py       # OAuth PKCE flow, polling
│   ├── twitter_cookies.py # Per-user Bird CLI cookies
│   ├── whatsapp.py      # WhatsApp sidecar session and sync
│   ├── map.py           # Contact map data
│   ├── sync_history.py  # Sync run history
│   ├── version.py       # Update check
│   ├── linkedin.py      # Extension data push (profiles + messages)
│   ├── extension.py     # Chrome extension pairing
│   ├── activity.py      # Recent activity feed
│   └── settings.py      # User preferences
├── models/              # SQLAlchemy 2.x async ORM models
├── schemas/             # Pydantic request/response schemas
├── services/            # Business logic layer
├── integrations/        # Third-party API clients
└── core/                # Config, auth, database, encryption, Redis

API Response Envelope

All endpoints return a typed generic envelope, which gives the frontend one consistent contract:

from typing import Generic, TypeVar
from pydantic import BaseModel

T = TypeVar("T")

class Envelope(BaseModel, Generic[T]):
    data: T | None = None
    error: str | None = None
    meta: dict | None = None

Endpoints declare response_model=Envelope[SomePayloadType], which keeps the types consistent from backend to frontend. For example:

@router.get("/contacts/{contact_id}", response_model=Envelope[ContactRead])
async def get_contact(contact_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
    ...
    return Envelope(data=contact)

Services Layer

Service	Responsibility
`followup_engine`	Two-pool suggestion generator (Pool A: active relationships, Pool B: dormant revival; candidate selection lives in `followup_pools`)
`identity_resolution`	Tier 1 deterministic + Tier 2 probabilistic duplicate detection and merge
`scoring`	Relationship score calculation (0-10 scale)
`meeting_prep`	Pre-meeting briefs built from Calendar events
`event_classifier`	LLM-based classification of tweets and bio changes into structured events
`message_composer`	Context-aware AI draft generation with tone matching
`auto_tagger`	LLM-driven contact tagging with taxonomy management
`organization_service`	Company grouping, domain extraction, stats materialized view
`contact_search`	Case-insensitive substring search across contact names, company, title, handles, and interaction text
`telegram_service`	Higher-level Telegram orchestration with caching
`bio_refresh`	Twitter/Telegram bio change detection and tracking

Integrations

Integration	Protocol	Auth
Gmail	Gmail API (REST)	OAuth 2.0 (one token per linked Google account via the `GoogleAccount` model; sync covers every linked account)
Telegram	MTProto via Telethon	Phone number + session string (encrypted at rest)
Twitter/X	Bird CLI for bios and tweets, X API v2 for DMs	Per-user browser cookies (`auth_token` + `ct0`) pushed by the user, no server-wide cookies / OAuth 2.0 PKCE
Google Calendar	Calendar API (REST)	Same Google OAuth tokens as Gmail, for every linked account
LinkedIn	Chrome extension (browser-side scrape)	Pairing code; `CHROME_EXTENSION_ID` adds the extension origin to CORS. No LinkedIn API key required
WhatsApp	Sidecar service (`WHATSAPP_SIDECAR_URL`)	Session paired through the sidecar

Database Schema

PostgreSQL with UUID primary keys throughout. Key models and their relationships:

┌──────────┐     ┌───────────┐     ┌──────────────┐
│   User   │────<│  Contact   │────<│  Interaction  │
│          │     │            │     │  (email/tg/x) │
│          │     │ org_id ──────>┌──┴──────────────┘
│          │     └─────┬──────┘ │
│          │           │        │  ┌──────────────────┐
│          │           ├───────<│  DetectedEvent      │
│          │           │        │  (job_change, etc.) │
│          │           │        └──────────────────────┘
│          │     ┌─────▼──────┐
│          │────<│FollowUp    │
│          │     │Suggestion  │
│          │     └────────────┘
│          │
│          │────<│ Notification │
│          │────<│ GoogleAccount │
└──────────┘
           ┌────────────────┐
           │ IdentityMatch  │  (contact_a_id, contact_b_id, score, status)
           │ ContactMerge   │  (audit trail for merged contacts)
           │ Organization   │  (company records with domain, industry)
           │ TagTaxonomy    │  (hierarchical tag categories)
           └────────────────┘

Key Model Fields

Contact: full_name, emails[] (PostgreSQL ARRAY with GIN index), phones[], company, organization_id, twitter_handle, twitter_bio, telegram_username, telegram_bio, linkedin_url, relationship_score (0-10), interaction_count, priority_level (high/medium/low/archived), tags[], last_interaction_at, last_followup_at, birthday.

Interaction: contact_id, platform (email/telegram/twitter/linkedin/whatsapp/meeting/manual), direction (inbound/outbound), content_preview, occurred_at, call_type (call category, e.g. voice/video), duration_seconds (call length).

FollowUpSuggestion: contact_id, trigger_type (time_based/event_based/scheduled/birthday/dormant_deep/dormant_mid/dormant_event), suggested_message, suggested_channel, status (pending/snoozed/dismissed/sent), pool (A/B), scheduled_for (the snooze end; the API accepts it as snooze_until or scheduled_for), dismissed_by (user/system).

DetectedEvent: contact_id, event_type (job_change/fundraising/product_launch/promotion/milestone/event_attendance), confidence (0.0-1.0), summary, source_url.

Materialized View

organization_stats_mv aggregates per-organization metrics (contact count, total interactions, average score, last interaction time) over non-archived contacts. A Celery beat task refreshes it hourly.

Notable Indexes

ix_contacts_emails_gin: GIN index on contacts.emails for @> (array containment) queries
ix_contacts_relationship_score: B-tree for score-based sorting and filtering
ix_contacts_interaction_count: B-tree for count-based sorting
ix_contacts_full_name: B-tree for name search

Task Queue

Celery uses Redis as its broker. Tasks use JSON serialization and run in UTC.

Beat Schedule

Task	Schedule	Description
`sync_gmail_all`	Every 6 hours (:00)	Sync email threads for every linked Google account of each connected user
`sync_google_calendar_all`	Daily 06:00 UTC	Sync calendar events from every linked Google account for meeting detection
`sync_telegram_all`	Daily 03:00 UTC	Sync Telegram chat history via MTProto
`poll_twitter_all`	Daily 04:00 UTC	For users with Twitter connected, poll contact bio changes and sync DMs
`update_relationship_scores`	Daily 02:00 UTC	Recalculate all contact relationship scores
`generate_suggestions_all`	Daily 08:00 UTC	Run follow-up engine for all users
`refresh_org_stats`	Hourly (:30)	Refresh `organization_stats_mv` materialized view
`reactivate_snoozed_suggestions`	Hourly (:00)	Un-snooze suggestions past their `snooze_until`
`send_weekly_digests`	Monday 09:00 UTC	Email weekly networking digest
`recheck_telegram_bios_all`	05:00 UTC on days 1, 4, 7 ... 28 of each month	Re-check Telegram bios for changes
`cleanup_stale_telegram_locks`	Hourly (:15)	Remove stale Telegram rate-limit locks
`scan_meeting_preps`	Every 10 minutes	Scan upcoming Calendar meetings and email pre-meeting prep briefs
`check_whatsapp_sessions`	Daily 01:00 UTC	Verify WhatsApp sidecar sessions are still connected
`check_for_updates`	Every 6 hours (:15)	Check GitHub for new PingCRM releases

Task Safety

Default soft time limit is 5 minutes and hard limit is 10 minutes. Long tasks such as sync_gmail_for_user and poll_twitter_activity override this with 15 and 20 minutes
Sync failure tasks (notify_sync_failure) create in-app notifications when retries are exhausted
Celery workers are synchronous, so each task runs its own async event loop through a _run() helper

@shared_task(name="app.services.tasks.sync_gmail_all")
def sync_gmail_all() -> dict:
    return _run(_sync_gmail_all_async())

AI Pipeline

The AI features all use the Anthropic Claude API.

Event Classification

Tweets and bio changes are classified into structured event types:

Tweet/Bio text
    │
    ▼
┌────────────────────┐     ┌─────────────────────────────┐
│ classify_tweet()   │────>│ JSON response:              │
│ classify_bio_      │     │ { event_type, confidence,   │
│   change()         │     │   summary }                 │
└────────────────────┘     └─────────────┬───────────────┘
                                         │
                                         ▼
                              confidence >= 0.7?
                              ├── yes ──> DetectedEvent record
                              └── no ───> discarded

Valid event types: job_change, fundraising, product_launch, promotion, milestone, event_attendance, none.

An asyncio.Semaphore caps LLM calls at 5 concurrent requests, and transient errors (429, 500, 529) are retried with exponential backoff and jitter.

Message Composition

The message composer builds each draft from these inputs:

Contact profile: name, company, title, bios, relationship score
Conversation history: last 5 interactions with direction labels
Tone analysis: analyze_conversation_tone() classifies past messages as formal/casual based on language indicators
Twitter context: recent tweets fetched via bird CLI (cached 12h in Redis) for time-based and revival triggers
Trigger context: reason for follow-up (time gap, detected event, birthday, dormant revival)

The prompt asks for at most 3 sentences in the detected tone that refer to the trigger. If the API call fails, the composer falls back to a template message.

Auto-Tagging

LLM-generated tags are managed through a TagTaxonomy with hierarchical categories. Tags can be generated for one contact or applied in bulk by background tasks.

Rate Limiting

Telegram Rate Gate

Telegram's MTProto API enforces strict rate limits and signals them with FloodWaitError. PingCRM coordinates all Telegram operations through a Redis rate gate:

┌─────────────────┐     FloodWaitError(seconds=N)
│ Any Telegram    │────────────────┐
│ operation       │                ▼
│ (sync, send,    │     ┌──────────────────────┐
│  bio refresh)   │     │ Redis SET            │
└────────┬────────┘     │ tg_flood:{user_id}   │
         │              │ EX = N seconds       │
         │              └──────────────────────┘
         │
         ▼
┌─────────────────┐     ┌──────────────────────┐
│ _check_rate_    │────>│ Redis TTL            │
│ gate(user_id)   │     │ tg_flood:{user_id}   │
└────────┬────────┘     └──────────────────────┘
         │
    TTL > 0?
    ├── yes ──> skip operation / return error with wait time
    └── no ───> proceed

Key behaviors:

Cross-operation coordination: When any Telegram call triggers FloodWaitError, the gate is set for that user. Later operations (sync, send, bio refresh, group member fetch) check the gate before connecting.
Skip unchanged dialogs: The sync skips a dialog when its latest message is already stored, which saves API calls.
7-day bio freshness filter: Bio refresh skips contacts whose bio was checked within the last 7 days (telegram_bio_checked_at).
Max contacts per bio sync: Capped at 100 contacts per run.

Frontend Rate Limit UX

The MessageEditor component detects 429 responses on send and blocks sending until the wait ends. The wait comes from meta.retry_after in the response body (the API also sets a Retry-After header) and defaults to 3600 seconds when absent.

Identity Resolution

Duplicate detection runs in three tiers, with guards against false positives:

Contacts
    │
    ▼
┌────────────────────────────────────┐
│ Tier 1: Deterministic              │
│ - Same email in two contacts       │
│ - Same normalized phone number     │
│ - Email found in Twitter bio       │
│                                    │
│ Result: auto-merge immediately     │
└───────────────┬────────────────────┘
                ▼
┌────────────────────────────────────┐
│ Tier 2: Probabilistic (weighted)   │
│                                    │
│ Signal          Base Weight        │
│ ─────────────── ───────────        │
│ Email domain    0.40               │
│ Name similarity 0.20               │
│ Company match   0.20               │
│ Username sim.   0.10               │
│ Mutual signals  0.10               │
│                                    │
│ Adaptive: weights redistribute     │
│ when signals are unavailable       │
│                                    │
│ Score > 0.85  ──> auto-merge       │
│ 0.70 - 0.85  ──> pending review   │
│ Score < 0.70  ──> ignored          │
└───────────────┬────────────────────┘
                ▼
┌────────────────────────────────────┐
│ Tier 3: Manual Review              │
│ - UI lists each pending match      │
│ - User merges or rejects the match │
└────────────────────────────────────┘

Blocking Keys

To avoid comparing every pair, contacts are only compared if they share a blocking key:

Name prefix (first 3 characters)
Name tokens (words of 3 or more characters)
Company (exact, lowercase)
Email domain
Email local part tokens (e.g., pengcheng.chen@gmail.com generates tokens pengcheng, chen)
Twitter handle, Telegram username, LinkedIn profile ID

Blocks larger than 500 contacts are skipped, such as the gmail.com domain block. Contacts from the same platform source are never compared with each other, since their platform identifiers already differ.

Colleague Guard

When two contacts share a company but have clearly different names (similarity below 0.5), the score is capped at 0.35, below the 0.70 review threshold. The same cap applies when they share an email domain and the names differ (similarity below 0.7). Coworkers at one company are therefore not flagged as duplicates.

Additional guards:

Single-word names: When either contact has a one-word name and there is no email, company, or username signal, the score is multiplied by 0.4 and capped at 0.50. A company mismatch caps it at 0.35, and a weak username match caps it at 0.45.
Name-only cap: When the name is the only available signal, the score is capped at 0.70.

Merge Behavior

When contacts are merged, the richer contact (by non-null field count) becomes the primary:

List fields (emails, phones, tags) are unioned with order-preserving, case-insensitive dedup
Scalar fields are filled from the secondary contact where the primary has nulls
Interactions are reassigned to the surviving contact
A ContactMerge audit record is created
An IdentityMatch record tracks the merge (secondary's FK becomes NULL after deletion)

Frontend Architecture

Stack

Next.js 16 with App Router (file-based routing under src/app/)
React 19 with server and client components
Tailwind CSS v4 for styling
TanStack React Query for server state management (via QueryClientProvider wrapper)
openapi-fetch for a typed API client generated from the backend OpenAPI spec

API Client

The typed client is generated from the backend's OpenAPI schema. Interceptors handle auth and errors (simplified here):

import createClient from "openapi-fetch";
import type { paths } from "./api-types";

const client = createClient<paths>({ baseUrl: "" });

client.use({
  async onRequest({ request }) {
    const token = localStorage.getItem("access_token");
    if (token) {
      request.headers.set("Authorization", `Bearer ${token}`);
    }
    return request;
  },
  async onResponse({ response }) {
    if (response.status === 401) {
      localStorage.removeItem("access_token");
      window.location.href = "/auth/login";
    }
    if (response.status >= 500) {
      throw new Error(`API error: ${response.status}`);
    }
    return response;
  },
});

Key Pages

Route	Component	Purpose
`/dashboard`	`DashboardPage`	Overview with stats, recent activity, suggestions
`/contacts`	`ContactsPage`	Contact list with search, filters, CSV import
`/contacts/[id]`	`ContactDetailPage`	Full contact profile with interaction timeline
`/suggestions`	`SuggestionsPage`	Follow-up suggestions with inline message composer
`/identity`	`IdentityPage`	Duplicate review queue for contacts and organizations
`/organizations`	`OrganizationsPage`	Company directory with aggregated stats
`/notifications`	`NotificationsPage`	In-app notification feed
`/map`	`MapPage`	Map of geocoded contacts
`/settings`	`SettingsPage`	Tabs: Integrations, Import, Follow-up Rules, Tags, Account

Error Handling

The app uses the Next.js error boundary files error.tsx and global-error.tsx. The root boundary shows the error message with a retry button and reports the error to POST /api/v1/errors.

Custom Hooks

Hooks live in frontend/src/hooks/, one file per domain.

Hook	Purpose
`useAuth`	JWT token management, login/logout, user state
`useContacts`, `useContact`, `useUpdateContact`	Contact list, detail, and mutations with React Query
`useSuggestions`, `useUpdateSuggestion`, `useSendMessage`	Follow-up suggestion fetching, status changes, and sending
`useDashboardStats`, `useDashboardActivity`	Dashboard stats and recent activity
`useNotifications`, `useUnreadCount`	Notification feed and unread count
`useIdentityMatches`, `useMergeMatch`, `useRejectMatch`	Identity match review and resolution
`useTelegramSyncProgress`	Monitor Telegram sync progress

Dependency Layers​

Allowed Import Direction​

Module Ownership​

Key Files by Size​

System Overview​

Backend Architecture​

API Response Envelope​

Services Layer​

Integrations​

Database Schema​

Key Model Fields​

Materialized View​

Notable Indexes​

Task Queue​

Beat Schedule​

Task Safety​

AI Pipeline​

Event Classification​

Message Composition​

Auto-Tagging​

Rate Limiting​

Telegram Rate Gate​

Frontend Rate Limit UX​

Identity Resolution​

Blocking Keys​

Colleague Guard​

Merge Behavior​

Frontend Architecture​

Stack​

API Client​

Key Pages​

Error Handling​

Custom Hooks​