Real-Time Presence Telemetry at Scale
Stateless heartbeats + Valkey, not WebSockets + Redis
Options on the table
Architecture at a glance
Background
Our live-streaming telemetry platform supports events of 200 to 3,000 concurrent viewers. The core requirement sounds simple: track exactly how many users are actively watching a stream at any moment, and show that number to moderators on a real-time Mission Control dashboard.
The standard industry pattern points to AWS API Gateway WebSockets paired with a transactional database like DynamoDB, listening for $connect and $disconnect events. Stress-testing and an architectural audit revealed that pattern would fail us in two specific, fatal ways under real live-event traffic.
The challenge
- The 500/sec connection wall (thundering herd): API Gateway WebSockets allow ~500 new handshakes per second by default. Live events are a thundering herd — thousands join the exact second it starts. With 3,000 simultaneous joiners, AWS would throttle and drop ~2,500 with 429 errors, and raising the quota needs historical metrics you don't have yet on a new platform.
- The mobile 'ghost user' problem: WebSockets are persistent, stateful connections. When a phone loses 5G, enters an elevator, or dies, it's a dirty disconnect — no goodbye packet. API Gateway can take up to 10 minutes (idle timeout) to tear down the zombie connection, so ghost users falsely inflate the live count by roughly 10% and the data stops being trustworthy for stakeholders.
The 'Pulse' architecture
Instead of fighting stateful connections, we moved viewers to a stateless 'text-message' pulse pattern — mirroring how giants like YouTube and Twitch handle presence telemetry.
- Stateless HTTP heartbeats: The video player runs a background timer; every 30 seconds it sends a lightweight sendHeartbeat GraphQL mutation over standard HTTPS.
- Infinite-scale ingestion: HTTPS mutations are stateless — they open, deliver, and close — so they bypass the 500/sec WebSocket wall entirely and let AWS absorb thousands of requests per second out of the box.
- Self-cleaning state engine (Valkey): Each heartbeat triggers a Lambda that runs ZADD on a Valkey sorted set: key event:{eventId}:live_users, score = current timestamp, member = userId (from the AppSync authorizer context).
- Hybrid delivery: WebSockets stay where they earn their keep — the dashboard. The handful of moderators open one reliable AppSync WebSocket subscription; a background service periodically queries Valkey for users active in the last 60s and pushes the unified count.
Why Valkey over Redis
The live count is a hot, high-frequency workload: thousands of heartbeats per second, each a tiny write and a windowed read. Writing those straight into a transactional disk database (DynamoDB) means astronomical Write Capacity Unit costs.
An in-memory cache is the right tool — and we chose Valkey over traditional Redis for better price/performance and a serverless-friendly fit, keeping the same sorted-set model without the cost. Historical telemetry still lands durably and cheaply in S3 via Kinesis Firehose for post-event analytics.
The impact
- Zero throttling, infinite scale: Routing check-ins through stateless AppSync mutations unlocked a default ~10,000 req/sec tier — 3,000 users can hit 'Join' in the same millisecond with no errors and no custom quota increase.
- Ghost users eliminated: A deliberate 60-second grace window lets heartbeats lapse naturally; Valkey's sorted set ages absent users out on the next cycle. Max inaccuracy dropped from a 10-minute delay to a tight, predictable 60-second ceiling.
- Drastic cost optimization: High-frequency writes moved from expensive transactional WCUs to ultra-low-latency, low-cost memory, with history streamed to S3 — real-time presence at a fraction of the cost.