Socket.io at Scale: Lessons from 5,000 Concurrent Users
AutoBazaar was a real-time vehicle auction platform — dozens of live auctions running simultaneously, each with hundreds of bidders watching the price tick up in real time. We hit 5,000 concurrent WebSocket connections within six months of launch.
Here's what the architecture looked like, what broke, and what we changed.
The Naive Start
The first version was straightforward:
@WebSocketGateway({ cors: { origin: '*' } })
export class AuctionGateway {
@WebSocketServer() server: Server;
@SubscribeMessage('join-auction')
handleJoin(client: Socket, auctionId: string) {
client.join(`auction:${auctionId}`);
}
async broadcastBid(auctionId: string, bid: Bid) {
this.server.to(`auction:${auctionId}`).emit('new-bid', bid);
}
}This worked perfectly — on a single server. The problem arrived when we scaled to two Node.js instances behind a load balancer.
The Multi-Instance Problem
Socket.io keeps a connection state in memory. Instance A knows about its connected clients. Instance B knows nothing about them. When broadcastBid runs on Instance A, none of Instance B's clients receive the event.
The fix is the Redis adapter, which uses Redis pub/sub to broadcast events across all instances:
npm install @socket.io/redis-adapter ioredisasync function bootstrap() {
const app = await NestFactory.create(AppModule);
const httpServer = app.getHttpServer();
const io = Server(httpServer);
// Connect Redis adapter — all instances share event state
const pubClient = new Redis({ host: 'redis', port: 6379 });
const subClient = pubClient.duplicate();
io.adapter(createAdapter(pubClient, subClient));
await app.listen(3000);
}With the Redis adapter, a server.to('auction:123').emit('new-bid', bid) on any instance reaches all clients in that room, regardless of which instance they're connected to.
Room Architecture
We used a consistent room naming convention:
const ROOMS = {
auction: (id: string) => `auction:${id}`,
auctionAdmins: (id: string) => `auction:${id}:admins`,
user: (id: string) => `user:${id}`,
};auction:{id}— all bidders watching an auctionauction:{id}:admins— auctioneers managing the auction (separate room for control events)user:{id}— private channel for notifications to a specific user
On join, a client subscribes to the relevant rooms:
@SubscribeMessage('join-auction')
async handleJoin(
@ConnectedSocket() client: Socket,
@MessageBody() { auctionId }: { auctionId: string },
) {
const user = await this.authService.validateSocket(client);
if (!user) return client.disconnect();
await client.join(ROOMS.auction(auctionId));
await client.join(ROOMS.user(user.id)); // private notifications
if (user.role === 'auctioneer') {
await client.join(ROOMS.auctionAdmins(auctionId));
}
// Send current auction state immediately on join
const state = await this.auctionService.getCurrentState(auctionId);
client.emit('auction-state', state);
}Authentication Over WebSockets
Never trust a socket connection without verifying the token. Use the NestJS WsGuard pattern with a middleware on the handshake:
export function wsAuthMiddleware(
authService: AuthService,
): (socket: Socket, next: NextFn) => void {
return async (socket, next) => {
const token = socket.handshake.auth?.token
?? socket.handshake.headers?.authorization?.replace('Bearer ', '');
if (!token) return next(new Error('Unauthorized'));
try {
const user = await authService.verifyToken(token);
socket.data.user = user;
next();
} catch {
next(new Error('Unauthorized'));
}
};
}@WebSocketGateway()
export class AuctionGateway implements OnGatewayInit {
afterInit(server: Server) {
server.use(wsAuthMiddleware(this.authService));
}
}Handling Reconnection
Clients disconnect — mobile networks, browser tabs going to sleep, network hiccups. When they reconnect, they need to re-sync state without missing events.
@SubscribeMessage('reconnect-auction')
async handleReconnect(
@ConnectedSocket() client: Socket,
@MessageBody() { auctionId, lastEventId }: { auctionId: string; lastEventId: string },
) {
await client.join(ROOMS.auction(auctionId));
// Send missed events since lastEventId
const missed = await this.auctionService.getEventsSince(auctionId, lastEventId);
client.emit('missed-events', missed);
// Send current state
const state = await this.auctionService.getCurrentState(auctionId);
client.emit('auction-state', state);
}Store events in Redis with a short TTL (30 minutes) so reconnecting clients can catch up without a database query.
What Broke at Scale
1. Memory leaks from lingering event listeners
We were adding event listeners inside handleJoin without cleaning them up. At 5,000 connections this became significant. Fix: always remove listeners in handleDisconnect:
@SubscribeMessage('disconnect')
handleDisconnect(client: Socket) {
client.removeAllListeners();
this.activeConnections.delete(client.id);
}2. Broadcast storms
When a popular auction closed, we emitted auction-ended to 300+ clients simultaneously. The Redis pub/sub broadcast + 300 individual socket writes created a spike. Fix: batch the emission with a small stagger using setImmediate:
async broadcastToRoom(room: string, event: string, data: unknown) {
const sockets = await this.server.in(room).fetchSockets();
for (const socket of sockets) {
await new Promise<void>((resolve) => setImmediate(resolve));
socket.emit(event, data);
}
}3. Missing connection limits
Without limits, a single user could open dozens of tabs and hold hundreds of connections. Add a per-user connection limit:
@SubscribeMessage('connection')
async handleConnection(client: Socket) {
const userId = client.data.user?.id;
const userSockets = await this.server
.in(ROOMS.user(userId))
.fetchSockets();
if (userSockets.length > 5) {
client.emit('error', { message: 'Too many connections' });
client.disconnect();
}
}If you don't use the Redis adapter, Socket.io requires sticky sessions (consistent hashing) at your load balancer level — every request from the same client must go to the same server. With the Redis adapter, sticky sessions are optional but still recommended to reduce cross-instance pub/sub traffic.
Monitoring
Track these metrics in production:
socket.ioconnected clients (gauge, per instance)- Events emitted per second (rate)
- Room sizes — alert if any room exceeds your expected max
- Handshake auth failures — sudden spikes indicate bot activity
The Redis adapter exposes connection counts via io.of('/').adapter.rooms.
The two things that mattered most at 5,000 connections: the Redis adapter for multi-instance state, and per-user connection limits to prevent resource exhaustion. The room architecture and reconnection handling came after, but those two were the foundation everything else built on.
Arif Iqbal
Senior Backend Engineer with 10+ years building high-traffic platforms. NestJS · Node.js · Laravel · AWS · PostgreSQL. Open to remote & relocation.
Enjoyed this post?
Get my technical deep-dives in your inbox. No spam, unsubscribe anytime.
Discussion