Scaling Slack and GitHub with Maude Lemaire

Slack and GitHub hit the limits of hand-built sharding

Slack's data model treated the team as the natural unit, which held until enterprise customers arrived. Teams got databases sized to them until Slack ran out of bigger ones to buy, and a single enterprise row sent every IBM user to one shard. The fix was resharding by channel and rebuilding load testing around real clients. GitHub later ran into the same wall, with 16 shards it could not add to.

Takeaways

• Slack outgrew sharding by team. Teams got databases sized to their headcount until Slack ran out of larger instances to buy, and one enterprise row sent every IBM user to a single shard. Slack resharded data by channel ID on Vitess.
• The membership merge was Slack's riskiest fix. Sidebars were one large join union because DMs, public channels, private channels and group DMs sat in separate tables. The merge touched 400-plus raw SQL call sites, and one flag flip left everyone with blank sidebars for half an hour.
• Shedding load is a product decision. One CEO post in a 100,000-user channel turns a single custom emoji into 100,000 sent to everyone connected. Nobody owned custom emojis, so nobody capped them. Deciding what to drop under load belongs to product, not engineering.
• GitHub's 16 shards could not grow. Pull requests and issues lived on 16 Vitess shards, and scatter queries without the primary key hit every shard, so adding shards was not an option. The files page ran over 100 MySQL queries across three clusters.
• LLMs have not helped performance work. At Cursor she instrumented time to first token by hand, annotating a blueprint-paper sequence diagram against the code. Claude Opus suggested parallelising tasks, but the longest task was always the bottleneck and the other three served about 1% of users.
• Platform guardrails matter more as teams speed up. Product teams now ship faster, so the platform has to enforce quality rather than hope for it. She wants engineers rotating between product and product platform teams, like the embedded SRE model, because the two groups otherwise drift apart.

Rebuilding Slack's load testing

1. The first tester ran Slack as headless Chrome instances in Kubernetes, attempting 10,000 per pod, with no real WebSocket connections.
2. More than 50,000 instances at 37 cents each per five-hour tranche cost about $150,000 a test, with half the time spent restarting crashed pods.
3. The replacement used small Go processes holding real WebSocket connections, plus a recorder tool started by a slash command that captured every API request and WebSocket message.
4. The recordings fed Slack's Koi Pond system, and product managers were invited to record their own flows.