Data Flow

This document describes how data moves through the UniPulse platform, covering both synchronous request handling and asynchronous job processing.

---

Synchronous Request Flow

Step-by-Step Breakdown

Step	Component	Purpose
1	React SPA	Makes API call via TanStack Query to /api/v1/*
2	Express Router	Routes request to the correct handler
3	auth.ts	Validates JWT from Authorization: Bearer <token> header
4	requireWorkspace.ts	Attaches workspace context, enforces RBAC role check
5	requireFeature.ts	Checks if the feature is enabled for the workspace's plan
6	requireQuota.ts	Ensures workspace hasn't exceeded usage quotas
7	validate.ts	Validates request body/params against Zod schema from @unipulse/shared
8	rateLimiter.ts	Enforces per-IP rate limits (separate limits for auth and AI routes)
9	Service Layer	Executes business logic
10	Prisma ORM	Runs database queries (always scoped by workspaceId)
11	Response	Returns JSON payload to client

Middleware Order Matters

The middleware chain runs in strict order. Authentication runs first so that unauthorized requests are rejected before any business logic executes. Validation runs after auth so that validation errors include the user context for logging.

---

Asynchronous Processing Flow

Heavy operations are offloaded to BullMQ queues backed by Redis. This keeps API response times fast while processing happens in the background.

Queue Job Lifecycle

State	Description
waiting	Job is in the queue, waiting for a worker
active	Worker has picked up the job and is processing it
completed	Job finished successfully
failed	Job failed (will retry based on backoff config)
delayed	Job is scheduled for future execution

---

All 27 Queue Flows

Queue Name	Trigger	Processing	Output
publish	User publishes post	Posts to Facebook/Instagram/TikTok APIs	PostPublication records
schedule	Cron or scheduled time	Triggers publish queue at scheduled time	Moves post to publish queue
analytics-sync	Periodic / manual	Fetches metrics from platform APIs	PostMetric records
ai-suggestions	Background generation	Calls Gemini API for content suggestions	AiSuggestion records
workflow-engine	Workflow trigger event	Evaluates conditions, executes actions	WorkflowRun records
prediction-eval	After analytics sync	Runs engagement/revenue prediction models	EngagementPrediction records
competitor-scan	Periodic schedule	Scrapes competitor social profiles	CompetitorSnapshot records
ecommerce-sync	Webhook / periodic	Syncs products and orders from stores	EcommerceProduct/CommerceOrder records
calendar-generate	User request	Generates AI content calendar	CalendarEntry records
trend-scan	Periodic schedule	Detects trending topics via Gemini	Trend data
ad-sync	Periodic schedule	Syncs ad performance from Meta/TikTok	AdCampaign metrics
ab-test-eval	Test duration elapsed	Evaluates A/B test results	ABTest results
audience-sync	After interaction	Updates audience graph and scores	AudienceNode records
benchmark-compute	Periodic schedule	Computes industry benchmarks	IndustryBenchmark records
ice-process	Incoming message	Runs 3-step LLM pipeline (classify, context, respond)	ConversationMessage reply
ice-escalation	Escalation rule match	Routes thread to human agent	Escalation records
ice-experiment-eval	Experiment duration elapsed	Evaluates reply experiments	ReplyExperiment results
order-sync	Webhook / periodic	Syncs orders for revenue attribution	CommerceOrder records
payment-webhook	Payment provider webhook	Processes subscription changes	Subscription/Transaction records
post-classify	After post creation	Classifies post content type and topic	PostClassification records
revenue-attribution	After order sync	Matches orders to posts via UTM params	RevenueAttribution records
segment-eval	After audience update	Re-evaluates audience segments	AudienceSegment membership
token-refresh	Before token expiry	Refreshes social platform OAuth tokens	Updated SocialAccount tokens
webhook-process	External webhook received	Processes incoming webhook payloads	Various
workspace-classify	Periodic	Classifies workspace industry/niche	Workspace metadata

Queue Failure Handling

All queues are configured with exponential backoff retry. Default: 3 attempts with 5-second base delay. Failed jobs after all retries are retained in Redis for debugging and are visible in the admin dashboard.

---

End-to-End Data Flow Examples

Post Publishing Flow

Conversation Engine Flow

Revenue Attribution Flow

---

Cross-Reference

• Queue System -- detailed queue configuration and code patterns
• Middleware -- full middleware stack documentation
• Redis & Queues Infrastructure -- Redis configuration
• Conversation Engine -- ICE pipeline details
• Auth Flow -- authentication details