How Engadget survives Apple iDevice events

Friday morning at LISA, people are tired but sessions still go on. Engineers from Engadget gave a presentation on how they survive Apple product announcements. It’s a very hard problem. The 4.4 billion requests during the two hour announcement is more than most of AOL’s sites get in a week.

There are a few methods for how they survive, but the single biggest is to build caching into every layer and avoid cache-busters. They’ve done so well at this that a single MySQL server per datacenter is enough to feed the thundering hordes.

The infrastructure that just survived the iPhone5 announcement is very different from the ones that lived through earlier announcements. At the very top, browsers poll the Engadget stack for updates written in JSON, and the browser rerenders based on what they get. At the bottom, when a front-end server gets a request for an update that isn’t in its local memcached instance it requests it from the MySQL server, and then updates the other Memcached processes in it’s server-group. The load-balancers then cache these JSON objects.

They also do some fairly normal optimizations on the webservers themselves. Turning down the TIME WAIT recycle time to a very small number allows their webservers to not run out of connections, as does turning on the available TCP port range. They also do some fairly normal Apache optimizations for handling their PHP code.

There are a few more steps, but this cache-focused architecture has survived!

There is a wild-card, though: actual articles. When the major announcements are made, the editors push full articles to the main site. All of the syndication traffic that sparks causes it’s own problems in a different stack, so they’ve built a cache/proxy system just for handling that traffic.

Happily, they know in advance when to start scaling up for these events! And every time they live through it, they learn something new. They’re doing well enough that they can tell when other tech-sites fall over due to loading; their own traffic spikes as people come to them.

It doesn’t help that every iDevice announcement has more traffic than the previous one.