I recently attended the first annual QCon held in New York City. I was fortunate enough to attend some excellent presentations from folks who design, build and operate some of the most heavily visited sites on the Internet: Twitter, Facebook, Etsy, and Netflix to name a few.
While the companies at QCon might all be in different businesses – from social networking to e-commerce to entertainment – they all have a few things in common: fanatical attention to detail, rapid delivery of enhancements, intolerance of wasted effort, a genuine understanding and appreciation of their users, and an intense focus on constant improvement. The end result is that all of their businesses are well regarded by not only their users but also by fellow software developers.
Twitter: It’s all about speed
Twitter is one of the most fascinating companies in the world. In a small amount of time they’ve tackled one of the most fundamentally challenging tasks humans have faced: scaling the delivery of information to a virtually unlimited audience. The challenge of sharing ideas with a wide audience was historically logistical in nature and tended to boil down to the cost of delivery; sharing a message with a few people was easy, but sharing a message with millions of people was difficult, time consuming, and expensive. Today, a message can be shared in virtually real-time with millions of people around the world just as easily as sharing the same idea with a few friends in your living room. This is a serious game-changer.
So how does Twitter deliver a tweet from Lady Gaga to 20 million of her followers almost instantly?
Rather than use a single approach for all tweets and a single set of technologies, they define sensible boundaries within the overall system and optimize within those boundaries. Timeline, searching, and delivery are all built with completely different sets of technologies and considered separate concerns within the overall Twitter system. They use a blend of technologies such as Ruby of Rails, Java, and Scala to accomplish this. The use of multiple languages is commonly referred to as polyglot programming and is becoming a much more widely-used technique. Applications are becoming more complex, more heavily used, and more depended on being available 24/7.
Twitter also uses a few other very simple techniques like storing data in the same format that it’s read to avoid unnecessary processing at runtime. They also do something most engineers haven’t quite mastered the knack of; they handle traffic outliers like Lady Gaga differently than regular users like me. When I send a tweet to a handful of people, my tweet is processed using a general-purpose delivery mechanism that is good enough for average users and doesn’t consume too many resources. But a tweet from Lady Gaga is considered a special event from a technology perspective, because after all, it needs to be delivered to millions of followers.
Web applications are often implemented with a one-size fits all approach. For sites as heavily visited as Twitter, fanatically measuring usage is critical, and being able to adapt to unusual usage patterns like a tweet from Lady Gaga is critical to keeping Twitter fast and stable.
Etsy: Optimizing for happiness
Etsy is a remarkable company. Not only do they sell fantastic products from very creative craftspeople, but they also regard the act of writing code for their own site as an act of craftsmanship. Their blog, Code as Craft, is one of my personal favourites.
Rather than use a customized general purpose e-commerce platform, Etsy built their own platform in-house using a set of simple technologies. Simplicity is what makes Etsy both remarkable and profitable. Rather than falling down the rabbit-hole of complexity, by keeping things simple they can rapidly enhance, maintain, and support a site that does some serious business: their merchants sold a combined $538 million of crafts in 2011. Just how rapidly do they deploy enhancements to the site? Over 20 times per day. In a single month, Etsy performs 300-600 upgrades. Rather than waiting months for the next quarterly release, at Etsy an idea can be planned, designed, built, and delivered in an afternoon.
How do they manage such a rapid evolution and keep both users and engineers happy?
* Part 2 in this series will be published next week. Stay tuned!