Skip to main content

Redis & Queues

Redis 7 serves as the backbone for BullMQ job queues, rate limiting counters, and optional caching in UniPulse.

Redis Roles​

RoleDescription
BullMQ job storePersists all 27 queue jobs, states, and metadata
Rate limitingStores request counters for authLimiter and aiLimiter
Token refresh trackingTracks which social tokens are being refreshed to prevent duplicates
Session cachingOptional session data caching

Connection​

// apps/api/src/lib/redis.ts
import Redis from 'ioredis';

export const redis = new Redis(process.env.REDIS_URL, {
  maxRetriesPerRequest: null, // Required by BullMQ
  enableReadyCheck: false,
});
EnvironmentDefault URL
Developmentredis://localhost:6379
Productionredis://redis:6379 (Docker service name)

BullMQ Queue Configuration​

Standard Queue Pattern​

// apps/api/src/queues/publish.queue.ts
import { Queue } from 'bullmq';
import { redis } from '../lib/redis';

export const publishQueue = new Queue('publish', {
  connection: redis,
  defaultJobOptions: {
    attempts: 3,
    backoff: { type: 'exponential', delay: 5000 },
    removeOnComplete: 100,    // Keep last 100 completed
    removeOnFail: 500,        // Keep last 500 failed
  },
});

Default Job Options​

OptionValuePurpose
attempts3Number of retry attempts
backoff.typeexponentialExponential backoff between retries
backoff.delay5000Base delay in ms (5s, 10s, 20s)
removeOnComplete100Keep last N completed jobs for monitoring
removeOnFail500Keep last N failed jobs for debugging

Queue-Specific Overrides​

QueueAttemptsBackoff DelayNotes
publish35000msCritical path -- must succeed
token-refresh510000msMore retries for token refresh
analytics-sync35000msStandard
ice-process33000msLower delay for conversation responsiveness
ab-test-eval1-One-shot evaluation
ice-experiment-eval1-One-shot evaluation

All 27 Queues​

Queue NameWorker ConcurrencyJob Type
ab-test-eval2One-shot
ad-sync3Periodic
ai-suggestions5Background
analytics-sync5Periodic + on-demand
audience-sync3Event-driven
benchmark-compute2Periodic
calendar-generate3On-demand
competitor-scan2Periodic
ecommerce-sync3Webhook + periodic
ice-escalation5Event-driven
ice-experiment-eval2One-shot
ice-process10Real-time (high priority)
order-sync3Webhook + periodic
payment-webhook5Webhook
post-classify5After post creation
prediction-eval3After analytics sync
publish5On-demand
revenue-attribution3After order sync
schedule5Cron-driven
segment-eval3After audience update
token-refresh3Proactive (before expiry)
trend-scan2Periodic
webhook-process5Webhook
workflow-engine5Event-driven
workspace-classify2Periodic

Redis Memory Management​

Key Patterns in Redis​

PatternPurpose
bull:{queue}:*BullMQ queue data
bull:{queue}:idJob counter
bull:{queue}:waitingWaiting job list
bull:{queue}:activeActive jobs
bull:{queue}:completedCompleted jobs
bull:{queue}:failedFailed jobs
rate:{limiter}:{key}Rate limiting counters

Memory Monitoring​

# Check Redis memory usage
redis-cli INFO memory

# Monitor real-time commands
redis-cli MONITOR

# List all queue keys
redis-cli KEYS "bull:*" | head -20

# Check specific queue size
redis-cli LLEN "bull:publish:waiting"
Redis Persistence

In production, ensure Redis has persistence configured (RDB snapshots or AOF logging) so that queued jobs survive Redis restarts. The redis:7-alpine image has RDB persistence enabled by default.

Monitoring​

MethodDescription
Admin API endpointsQueue job counts (waiting, active, completed, failed)
Control Center dashboardVisual queue health monitoring
Redis CLIDirect inspection of queue data
BullMQ eventsWorker event handlers for logging
Cross-Reference