Webhooks, Queues & Caching

The Big Picture

Webhooks, queues, and caching work together to build backends that are responsive, resilient, and fast. Webhooks bring data in, queues smooth out the processing, and caching speeds up the delivery.

Webhooks

A webhook is an HTTP callback — when something happens in an external service, it sends an HTTP POST request to a URL you've registered. Instead of your code constantly asking "did anything change?", the service tells you the moment something happens.

Push vs Poll

How Webhooks Work

1. You register a callback URL with the external service (e.g., https://yourapp.com/webhooks/stripe).
2. When an event happens (payment completed, PR merged, message sent), the service creates a JSON payload describing the event.
3. The service sends an HTTP POST request to your URL with the payload.
4. Your server receives it, verifies the signature, and processes the event.
5. You return a 200 OK immediately. If you don't, the service will retry.

Webhook Payload Example

// Stripe payment webhook payload
{
  "id": "evt_1NqFbSD2eZvKYlo2C3LPaXnf",
  "type": "payment_intent.succeeded",
  "data": {
    "object": {
      "id": "pi_3NqFbSD2eZvKYlo20kL2LvZ",
      "amount": 2000,
      "currency": "usd",
      "status": "succeeded",
      "customer": "cus_9s6XKzkNRiz8i3",
      "metadata": {
        "order_id": "order_12345"
      }
    }
  },
  "created": 1694123456
}

// GitHub push event webhook payload
{
  "ref": "refs/heads/main",
  "repository": {
    "full_name": "user/repo",
    "html_url": "https://github.com/user/repo"
  },
  "pusher": {
    "name": "developer",
    "email": "[email protected]"
  },
  "commits": [
    {
      "id": "abc123",
      "message": "Fix login bug",
      "timestamp": "2026-04-04T10:30:00Z"
    }
  ]
}

Building a Webhook Receiver

// Express.js webhook receiver
const express = require('express');
const crypto = require('crypto');
const app = express();

// IMPORTANT: Use raw body for signature verification
app.post('/webhooks/stripe',
  express.raw({ type: 'application/json' }),
  (req, res) => {
    const signature = req.headers['stripe-signature'];
    const endpointSecret = process.env.STRIPE_WEBHOOK_SECRET;

    // 1. Verify the signature (HMAC-SHA256)
    let event;
    try {
      event = stripe.webhooks.constructEvent(
        req.body, signature, endpointSecret
      );
    } catch (err) {
      console.error('Signature verification failed:', err.message);
      return res.status(400).send('Invalid signature');
    }

    // 2. Return 200 immediately (process async later)
    res.status(200).json({ received: true });

    // 3. Process the event asynchronously
    processWebhookAsync(event).catch(err => {
      console.error('Webhook processing failed:', err);
    });
  }
);

async function processWebhookAsync(event) {
  switch (event.type) {
    case 'payment_intent.succeeded':
      await fulfillOrder(event.data.object.metadata.order_id);
      break;
    case 'payment_intent.payment_failed':
      await notifyCustomer(event.data.object.customer);
      break;
    default:
      console.log('Unhandled event type:', event.type);
  }
}

Verifying Webhook Signatures

Never trust an incoming webhook without verifying its signature. Anyone can send a POST request to your endpoint. Webhook providers sign payloads using HMAC-SHA256 with a shared secret.

# Manual HMAC-SHA256 verification (Python)
import hmac
import hashlib

def verify_webhook(payload_body, signature_header, secret):
    """Verify that a webhook came from the expected sender."""
    expected_signature = hmac.new(
        key=secret.encode('utf-8'),
        msg=payload_body,
        digestmod=hashlib.sha256
    ).hexdigest()

    # Use constant-time comparison to prevent timing attacks
    return hmac.compare_digest(
        f"sha256={expected_signature}",
        signature_header
    )

Idempotency: Handling Duplicate Deliveries

Webhook providers retry on failures, which means you may receive the same event multiple times. Your handler must be idempotent — processing the same event twice should have the same result as processing it once.

# Idempotency with a processed events table
async def handle_webhook(event):
    event_id = event['id']

    # Check if we already processed this event
    if await db.events.find_one({'event_id': event_id}):
        return  # Already processed, skip

    # Process the event
    await process_event(event)

    # Mark as processed
    await db.events.insert_one({
        'event_id': event_id,
        'processed_at': datetime.utcnow()
    })

Retry Logic and Failure Handling

Most webhook providers use exponential backoff for retries:

• Attempt 1: Immediately
• Attempt 2: After 1 minute
• Attempt 3: After 5 minutes
• Attempt 4: After 30 minutes
• Attempt 5: After 2 hours
• After all retries fail, the event is dropped or sent to a dead letter queue

Common Webhook Providers

Message Queues

A message queue is a buffer that sits between producers (services that send messages) and consumers (services that process them). The producer drops a message into the queue and moves on. The consumer picks it up when it's ready. They don't need to be online at the same time — the queue holds messages until they're consumed.

Why Use Queues?

• Decouple services — The API server doesn't need to know (or wait for) the email service. It just drops "send welcome email" onto the queue.
• Handle traffic spikes — 10,000 orders in a flash sale? The queue absorbs the spike and workers process them at their own pace.
• Retry failures — If a consumer crashes, the message goes back on the queue and another worker picks it up.
• Async processing — Move slow work (PDF generation, image processing, API calls) out of the request-response cycle.

Comparison: Message Queue Technologies

Key Concepts

Publishing and Consuming with Python

# RabbitMQ with pika (Python)
import pika
import json

# --- Publisher (sends messages) ---
connection = pika.BlockingConnection(
    pika.ConnectionParameters('localhost')
)
channel = connection.channel()

# Declare a durable queue (survives broker restart)
channel.queue_declare(queue='email_tasks', durable=True)

# Publish a message
message = {
    'to': '[email protected]',
    'subject': 'Welcome!',
    'template': 'welcome_email'
}
channel.basic_publish(
    exchange='',
    routing_key='email_tasks',
    body=json.dumps(message),
    properties=pika.BasicProperties(
        delivery_mode=2  # Make message persistent
    )
)
print("Message sent to queue")
connection.close()


# --- Consumer (processes messages) ---
connection = pika.BlockingConnection(
    pika.ConnectionParameters('localhost')
)
channel = connection.channel()
channel.queue_declare(queue='email_tasks', durable=True)

# Process only 1 message at a time (fair dispatch)
channel.basic_qos(prefetch_count=1)

def callback(ch, method, properties, body):
    task = json.loads(body)
    try:
        send_email(task['to'], task['subject'], task['template'])
        # Acknowledge: remove from queue
        ch.basic_ack(delivery_tag=method.delivery_tag)
        print(f"Email sent to {task['to']}")
    except Exception as e:
        # Reject and requeue for retry
        ch.basic_nack(
            delivery_tag=method.delivery_tag,
            requeue=True
        )
        print(f"Failed, requeued: {e}")

channel.basic_consume(
    queue='email_tasks',
    on_message_callback=callback
)
print("Waiting for messages...")
channel.start_consuming()

Common Use Cases

• Email sending — Drop email tasks into a queue; a worker sends them without blocking the API response.
• Image processing — User uploads a photo; a queue job resizes, crops, and generates thumbnails in the background.
• Payment processing — Webhook arrives from Stripe; push it onto a queue for reliable, ordered processing.
• Event sourcing — Every state change is an event on a Kafka topic; replay events to rebuild state.
• Data pipelines — ETL jobs that extract, transform, and load data between systems.

Background Jobs & Task Queues

A background job is any work that happens outside the HTTP request-response cycle. Instead of making the user wait while your server generates a PDF, resizes an image, or calls a slow third-party API, you push the work to a task queue and respond immediately.

Popular Task Queue Libraries

When to Move Work to the Background

• Sending emails — SMTP calls take 1–5 seconds. Don't make the user wait.
• Generating reports — A complex CSV or PDF export might take 30 seconds.
• Image/video processing — Resizing, transcoding, and thumbnail generation are CPU-heavy.
• Third-party API calls — External services can be slow or unreliable.
• Data aggregation — Computing analytics, leaderboards, or recommendations.

Celery Task Example

# tasks.py - Define background tasks with Celery
from celery import Celery
from datetime import timedelta

app = Celery('tasks', broker='redis://localhost:6379/0')

# Configure retries and timeouts
app.conf.update(
    task_serializer='json',
    result_serializer='json',
    accept_content=['json'],
    task_time_limit=300,       # Hard kill after 5 minutes
    task_soft_time_limit=240,  # Raise exception after 4 minutes
)

@app.task(
    bind=True,
    max_retries=3,
    default_retry_delay=60,  # Wait 60s between retries
)
def send_welcome_email(self, user_id):
    """Send a welcome email to a new user."""
    try:
        user = get_user(user_id)
        email_service.send(
            to=user.email,
            subject="Welcome!",
            template="welcome",
            context={"name": user.name}
        )
    except EmailServiceError as exc:
        # Retry with exponential backoff
        raise self.retry(exc=exc, countdown=60 * (2 ** self.request.retries))

@app.task(bind=True, max_retries=2)
def generate_report(self, report_id):
    """Generate a PDF report in the background."""
    try:
        report = build_report(report_id)
        upload_to_s3(report)
        notify_user(report_id, status="complete")
    except Exception as exc:
        notify_user(report_id, status="failed")
        raise self.retry(exc=exc)

# views.py - Dispatch tasks from your API handler
from tasks import send_welcome_email, generate_report

def register_user(request):
    user = create_user(request.data)

    # Fire and forget: push to background queue
    send_welcome_email.delay(user.id)

    # Return immediately - email sends in background
    return {"id": user.id, "message": "Account created!"}

def request_report(request):
    report = create_report_record(request.data)

    # Dispatch background job
    generate_report.delay(report.id)

    return {"report_id": report.id, "status": "processing"}

Monitoring Background Jobs

• Flower (Celery) — Web dashboard showing active workers, task history, success/failure rates.
• Bull Board (BullMQ) — UI to inspect queues, retry failed jobs, view job data.
• Key metrics to track: queue depth (are jobs piling up?), processing time, failure rate, retry count.

Caching

Caching stores copies of frequently accessed data in a faster storage layer so you don't have to recompute or re-fetch it every time. It's the single most impactful optimization for read-heavy applications.

Why Caching Matters: Latency by Storage Layer

Cache Layers

1. Browser cache — The user's browser stores static assets (images, CSS, JS) locally. Controlled by Cache-Control and ETag headers.
2. CDN (Content Delivery Network) — Servers at the network edge (Cloudflare, CloudFront) cache static and even dynamic content close to users worldwide.
3. Reverse proxy (Nginx/Varnish) — Sits in front of your app server and caches full HTTP responses. Great for public pages that don't change often.
4. Application cache (Redis/Memcached) — Your code explicitly caches data (database results, API responses, computed values) in a shared in-memory store.
5. Database query cache — Some databases cache recent query results automatically. Useful but limited — invalidated on every write to the table.

Caching Strategies

Redis as a Cache

# Redis CLI basics
redis-cli

# SET a value with TTL (expire after 300 seconds)
SET user:123:profile '{"name":"Alice","email":"[email protected]"}' EX 300

# GET a cached value
GET user:123:profile

# Check TTL remaining
TTL user:123:profile

# Delete a cached key
DEL user:123:profile

# Set only if key doesn't exist (cache miss pattern)
SET user:123:profile '{"name":"Alice"}' NX EX 300

Caching in Python with Redis

import redis
import json

r = redis.Redis(host='localhost', port=6379, db=0)

def get_user_profile(user_id):
    """Cache-aside pattern: check cache first, fallback to DB."""
    cache_key = f"user:{user_id}:profile"

    # 1. Check cache
    cached = r.get(cache_key)
    if cached:
        return json.loads(cached)  # Cache hit!

    # 2. Cache miss: fetch from database
    profile = db.query("SELECT * FROM users WHERE id = %s", user_id)

    # 3. Store in cache with TTL (5 minutes)
    r.setex(cache_key, 300, json.dumps(profile))

    return profile

def update_user_profile(user_id, data):
    """Update DB and invalidate cache."""
    db.execute(
        "UPDATE users SET name=%s, email=%s WHERE id=%s",
        data['name'], data['email'], user_id
    )
    # Invalidate the cached version
    r.delete(f"user:{user_id}:profile")

Caching in Node.js with Redis

const Redis = require('ioredis');
const redis = new Redis();

async function getUserProfile(userId) {
  const cacheKey = `user:${userId}:profile`;

  // 1. Check cache
  const cached = await redis.get(cacheKey);
  if (cached) {
    return JSON.parse(cached); // Cache hit!
  }

  // 2. Cache miss: fetch from database
  const profile = await db.query(
    'SELECT * FROM users WHERE id = $1', [userId]
  );

  // 3. Store in cache with TTL (5 minutes)
  await redis.setex(cacheKey, 300, JSON.stringify(profile));

  return profile;
}

async function updateUserProfile(userId, data) {
  await db.query(
    'UPDATE users SET name=$1, email=$2 WHERE id=$3',
    [data.name, data.email, userId]
  );
  // Invalidate the cached version
  await redis.del(`user:${userId}:profile`);
}

Cache Invalidation

Cached data goes stale. The three main invalidation strategies are:

• TTL-based (Time To Live) — Set an expiration time. After 5 minutes (or whatever you choose), the cache entry is automatically deleted. Simple and good enough for most cases.
• Event-based — When data changes, actively delete or update the cache entry. More complex but ensures freshness. Works well with webhooks and message queues.
• Manual purge — An admin or deployment script clears the cache. Used for deployments or emergency fixes.

CDN Caching

A CDN caches your content on servers around the world (edge locations). Users in Tokyo get served from a Tokyo edge server instead of your origin in Virginia — cutting latency from 200ms to 20ms.

# Cache-Control headers for different content types

# Static assets (CSS, JS, images): cache for 1 year
Cache-Control: public, max-age=31536000, immutable

# HTML pages: cache briefly, revalidate
Cache-Control: public, max-age=3600, must-revalidate

# API responses: don't cache by default
Cache-Control: no-store

# Private data (user-specific): never cache on CDN
Cache-Control: private, no-cache

• Static assets — Cache aggressively (1 year) with content hashing in filenames (style.abc123.css) so new deploys get new URLs.
• HTML pages — Short TTL (1 hour) with must-revalidate so users get fresh content without waiting too long.
• API responses — Generally no-store unless the data is public and rarely changes (e.g., product catalog).

When to Use What

These patterns often work together, but here's a quick decision guide for when to reach for each one: