Klout had an incredible 2011, and as the end of the year approached I felt so proud to be part of such an amazing team! We accomplished a lot, so the holidays came at a great time; it was a much-needed opportunity for the team to catch a break and have some fun with our families. At the same time, I was excited about all the incredible things we were gonna have a blast building. So when January 2nd came along, I was back at Klout HQ ready to go; as is customary at Klout, there was a fun challenge ahead. We’ve been hard at work for the past few months and we’re excited to share with you what we’ve built.
The challenge was to re-engineer Klout’s API to sustain its rapid growth and massive traffic—ten billion API calls a month and growing—and also to empower our clients with all the features currently available on Klout.com.
So I started by concentrating on the problem at hand; I find it much more productive than jumping into the solution or any implementation detail, going against our instinct as engineers. There’s a small select group of companies serving that type of massive API traffic but a peculiar number screams out of the infographic above. Klout is the only one with less than seven hundred employees; we actually only have about 76 troopers! And what does that number represent? That number tells me that as fast as our API should be, we need to be moving even faster as a team–we need killer productivity! Besides, everyone likes to go fast!
As seems to be a recurring fact in my life—see more on Why Did I Fall in Love with Play! Framework?—the technology stack chosen was the always awesome, lean-and-mean, super-duper productive Play! Framework. As I mentioned on my previous blog post, Find Your Klout, Play was designed to provide a powerful, easy to extend infrastructure, it uses fast non-blocking IO, and it uses a stateless model that makes horizontal scaling a cakewalk. There is such a joy that comes every time my terminal sings “play new klout-* –with scala”. And with this type of traffic, our API might just be the most heavily used Play! Framework application to date!
Our API has Swagger
We were obviously going to create a RESTful API, but there are some components of SOAP (Simple Object Access Protocol) that have a lot of value. Even though I wanted no part of that business, in any shape or form, new or old, and its verbose XML format (remember the productivity factor) the WSDL (Web Service Definition Language) does provides features that aren’t commonly seen in REST. The WSDL allows client proxies to be automatically generated (wsdl2java comes to mind); it also allows developers to create client interfaces easily. How could we accomplish that without deep-diving into thousands of lines of XML? Swagger son! Here’s how Wordnik defines their marvel: “Swagger is a specification and complete framework implementation for describing, producing, consuming, and visualizing RESTful web services.” Swagger allows our API to have a well-defined contract in JSON, the same format we are using on every single endpoint.
“Talk is cheap. Show me the code.” – Linus Torvalds
- First of all, add the dependency on Swagger Play to your conf/dependencies.yml and run “play deps”. Then go ahead and define your data transfer class.
- Now it’s time to define the controller. All the @Api* annotations are provided by Swagger. The code you are about to see doesn’t do much other instantiating the Score class and setting its value to 100 (I wish that was my Klout score).
- We created an API trait to define all the business logic used on every endpoint. This trait provides an api method which we wrap each controller action with; this method does the validation, JSON transformation, and it tracks number of calls and response times with StatsD using David Ross’ super useful StatsD module. Stay tuned in our blog for another post on StatsD and our monitoring infrastructure soon.
- Now it’s time to define our REST-friendly route for our endpoint, a walk-in-the-park with my beloved Play! Framework. To do that add the following line to conf/routes.
- Caching is being done using Memcached, re-using Play’s support for it. Jay Taylor, Klout Perks’ lead engineer, wrote a nice wrapper for it.
We’re excited that the Play framework is now commercially supported by Typesafe, along with Akka and the Scala, all of which are firing on all cylinders at Klout. Building on a modern foundation like the Typesafe Stack makes it much easier for our development team to punch above its weight!
Unlock Your Klout!