This Big Data thing Is…
Well… Big!
Early this year I made a big career move: after
almost 7 years working at Yahoo! I joined Greenplum as our VP of Product Management. The excitement of the new job has been exhilarating – new
industries to understand, Big Data challenges to solve, and the fast moving
pace of a “start-up-like” company. I’ve always enjoyed learning
new things – it was what I liked best about working in the central
data team at Yahoo!. Greenplum is a place where I have continued to learn.
That said, there are a
few patterns that I experienced at Yahoo! that I continue to see as I meet with
Greenplum customers and prospects who want to tackle the world of Big
Data. The first is an ever-growing need for analytic agility.
Organizations are constantly challenged by the time and efforts required to
extract insights from existing data assets and then convert these into actions
(in the form of well-informed business decisions or data-driven
applications). A portion of this challenge is solved by having the right
platform – one of my favorite parts of being at Greenplum is when I can share
with a customer how Greenplum’s Unified Analytics Platform supports an agile analytics environment.
But the right platform is only part of the solution. The
other consistent query I get from prospects and customers is: “What is the right
organizational model for me to be successful with Big Data?” And while
it’s fun for me to talk about our platform, I think it’s this second aspect of
the Big Data challenge that may be the toughest to solve. And the most
critical aspect to success.
Strategic Data Solutions
at Yahoo!
In mid-2005 I joined a
newly formed group at Yahoo! called Strategic Data Solutions (SDS). I
actually sought out a role in the group after reading an article in Information Week about the appointment of Usama Fayyad as
Yahoo!’s Chief Data Officer. My persistence paid off and I was lucky
enough to get hired, joining a group of other data-loving professionals
(including my current Greenplum colleague Annika Jimenez.
Yahoo! was among the earlier companies to realize that its data assets were in
fact a strategic asset, and to make a big bet (in the form of
what grew to a 500+ person organization) to extract the most value from this
data. It turns out that the bet Yahoo! made back in 2005 is similar to
what I see many non-Internet companies starting to do today. In
industries that people may consider to be old-school when it comes to Big Data
– insurance, manufacturing, utilities – we now see executives starting to
anoint their own Chief Data Officers and rolling out strategic data
initiatives.
So, when these customers
ultimately ask, “What is the right organizational model for me to be successful
with Big Data?” – I look back to the way that the Strategic Data Solutions
organization was set up, and I see a lot of things that we did right. Of
course there is no single cookie-cutter approach to organizational design that
works in all situations, but I believe that the core philosophy that drove how
SDS was set up can help give other companies a strong framework for how to
think about their strategic data initiatives. I’ve listed below the core
components of the SDS organization – you can view these both as “product lines”
as well as organizations within the group. My assertion: when
building out your strategic data organization and its capabilities, think in
terms of the big functional areas below:
- Data Platform: at the core of any strategic data initiative is
establishing a strong data platform that meets the core data provisioning
needs of the organizations data consumers. Be careful not to confuse a data platform initiative
with a more traditional “data warehouse” initiative. While one of the functions of the data platform may be to host or
integrate with a data warehouse, the data platform also needs to support
data sets that may not typically be in a data warehouse (documents,
machine-generate logs, etc) and also needs to support workloads that
aren’t well suited to a traditional data warehouse (sandbox-based
analytics, feed provisioning to production systems, non-SQL data analysis,
etc). At Yahoo! we made a big investment in
building out a core data platform (originally a home-grown file-based
system, and ultimately a combination of Hadoop and relational databases) to
support the broad range of data consumers we needed to support.
- Business Intelligence: one of the mistakes we made early on in SDS was to
abdicate responsibility for the delivery of the core business intelligence
needs of the various Yahoo! business units. It was a convenient
decision to make initially: it was an area with demanding consumers,
difficult-to-prove ROI, and was frankly not as “sexy” as the other more
advanced work that we wanted to do. Over time, however, we realized
that supporting the business intelligence needs of our business
stakeholders needed to be one of our core offerings. There were
benefits in terms of data re-use, stakeholder relationships, and other
economies of scale that made this the right thing for SDS to do. By
successfully supporting the BI needs of our business partners we were able
to “earn the right” to engage with them on the more advanced analytics and
data services we had to offer. The key to success here was to
(appropriately) view our BI investment as a cost center. We avoided
getting caught up in the losing battle of trying to show the ROI of our BI
efforts by instead focusing our ROI-based initiatives in areas where we
could, in fact, show true returns (see below).
- Data Science Services: within SDS we worked hard to enable customers (the
various Yahoo! product lines & business units) to derive “actionable
insights” from the data asset we created with our Data Platform.
Often the data science skills required for anything other than traditional
reporting and BI weren’t resident in the various lines of business.
(In fact, we continue to see this challenge today, and are working to help
solve the Data Scientist skills shortage through things like our innovative partnership with
Kaggle ) So SDS built out a consultancy-oriented group
to help our customers move to the next level of analysis. The
ultimate goal of our engagements with the business was twofold: first, the
Data Science team was devoted to solving data-driven problems that
resulted in a measureable ROI (increasing ad clickthrough rates, reducing
churn, improving customer acquisition); second, we wanted to train our
internal business customers on how to use the Data Platform and associated
tools to do subsequent Data Science projects on their own.
- Data Driven Applications: the ultimate goal of a lot of our Data Science
initiatives at Yahoo! was to spur the creation of data-driven applications
that could measurably impact the bottom or top line. As the
name implies, these applications leveraged the results of some underlying
data science efforts (scoring algorithms, recommendation models, pricing
optimizations) to drive actions taken in Yahoo!s customer and
internal-facing applications. The team was structured to work on a
commissioned project basis: business unites would request support to build
specific applications and back up their requests with detailed business
cases. The Data Driven Applications team would then prioritize the
long list of incoming requests and methodically tackle the highest-value
projects. This model turned out to be a win-win for both SDS and our
internal customers – the business units received value-enhancing
data-driven applications; and SDS was able to effectively show how the
investment in data as a strategic asset was driving true ROI for Yahoo!.
- Data Distribution:
a final and important aspect of the strategic data organization is an
understanding that in addition to supporting the analytical needs (either
via BI support or data science projects) there is also the need to support
data distribution. For example, at Yahoo! the core data platform was
used to generate segment membership information for billions of users
(browser cookies) each day. These profiles needed to be distributed
out to the operational systems that consumed them – the ad targeting
platforms – so it was important to have the appropriate infrastructure and
APIs to allow the consumers of these large data sets to access and move
them. Additionally, there were consistent demands to provision
subsets of the data in the core data platform to other consumers both
inside and outside of Yahoo!. The Data Distribution challenge is one
that many of our Greenplum customers today are started to struggle with as
well, and it’s important to think about it when scoping out a big data
strategy.
Dive Right In. The
Water’s Warm!
Now I can’t guarantee
that the structure described above is perfect for every organization – there
are likely variations of this perspective that have worked for other successful
data groups. However, I do think the emerging themes are consistent, and
that if you consider the above elements while diving in to the Strategic Data
Organization waters, you’ll be more likely to achieve success.
At the end of the day,
there is a bit of a leap of faith
required to make a strategic bet on big data. But the data shows that
it’s worth it. A recent article in the Harvard
Business Review revealed: “In
particular, companies in the top third of their industry in the use of
data-driven decision making were, on average, 5% more productive and 6% more
profitable than their competitors.”
Good luck!
