[Note: What you're reading is the first installment of a new series I'm writing at Thinky.org. I'm frustrated that the cutting edge academic and industry research from technology domains isn't getting into the hands of product managers and front-line workers. That's because the conferences are expensive and the papers are hard to read. I'm going to boil down cutting edge research in user experience design, interaction design, infovis, and other fields and create short (500 word) posts to help make them relevant to other product designers and product owners. Should be fun, right?!]
Paper: Using Visualizations to Monitor Changes and Harvest Insights from a Global-Scale Logging Infrastructure at Twitter [ PDF ]
Authors: Krist Wongsuphasawat and Jimmy Lin
Tags: New System, Infovis, Big Data, Product Management
Web analytics is about getting the right data to make decisions about your product, your website, or your marketing program. Big data means that you have too much data. So what do we do? Learn from the experts — few companies have got bigger big data than Twitter. Just a note about the scale of the problem — Twitter has thousands of Hadoop clusters which ingest more than 100TB of log files daily (that's not to mention read more...