THE BEGINNING OF SOMETHING BIG

Today was my last day at Yahoo!, where I have worked for the past 6.5 years.  It's been a wild ride, and (so far) the best career choice I've ever made.  When I started working at Yahoo! I had already been working in the area of analytics, first at Broadbase (now KANA) and then at Enkata.  At both companies we worked with ever increasing database sizes - first Gigabytes at Broadbase, and then Terabytes at Enkata.  I thought I knew what it meant to work with big data sets.

But at Yahoo! I learned that what I'd been doing so far was child's play.  The first application I worked on, an internal tool for measuring the reach and engagement of Yahoo! properties, processed over 4 billion events a day and stored that data in a database that was 5 times larger than anything I had worked with to date.  During my time at Yahoo! I worked on data pipelines and analytical applications that routinely process tens of Terabytes a day of raw data, and data marts that easily break the 100 Terabyte mark.  And for a while, I think that Yahoo! was one of the few companies in the world that needed to and had the capacity to work with such data sizes.

What I see today is that times have changed: what was once the rarefied air of the few is becoming the daily Big Data challenge of the many.  It's no longer sufficient to expect that Big Data systems need to be run by a select few who know how to effectively deploy and manage Hadoop clusters or manually tune and maintain an MPP database.  In the emerging world of Big Data the companies that survive and win will be those that can get past the hurdles of just "managing" Big Data, but can rapidly iterate and learn using Big Analytics.
So, next week I will embark on a new journey, taking what I've learned over the past 12 years and applying it to the industries first real Big Data and Big Analytics platform.  I am sure it will be a fun ride, and a chance to do something, well...  BIG.