Is Big Data Gaining Momentum?

“Without Big Data, you are blind and deaf and in the middle of a freeway”– Geoffrey Moore

Absolutely yes. Last week was big for big data. On June 12 the Churchill Club hosted “The Elephant in the Enterprise: What Role Will Hadoop Play” Leaders from Cloudera, MapR, Facebook and Oracle offered their perspectives about big data's evolution and current inroads into the enterprise. It’s fair to say that Hadoop is no longer a science project as it’s quickly appearing on the radar of CIOs and CTOs. Big data is also poised to be big business with a total addressable market of about $100B (Hadoop’s share is currently about $14B and growing quickly), according to Shaun Connolly, VP of Strategy at Hortonworks.

The Churchill event was followed by the Fifth (yes, the Fifth) Hadoop Summit on June 13 and 14 at the San Jose Convention Center. If you ever had any doubt about the level of interest and expectation of big data in the marketplace check out the following stats:

2000+ participants: I venture to say that most were Hadoop novices and early adopters eager to learn about Hadoop and real-life use cases and to network with like-minded peers.
265 speaking abstracts were submitted to the summit
There were 84 selected presentations, 6 tracks, and 2 full days of sessions. The best attended sessions were the ones in which actual Hadoop practitioners shared their lessons learned with Hadoop. Among the notables: Wal-Mart Labs, Adobe, NetApp, Facebook, Twitter, and comScore (comScore analyzes about 30 billion events a day or 90% of the web on a daily basis–wow!)
The 49 sponsors and vendors who were represented in the conference showroom are part of the growing ecosystem of companies that offer support, training, integration and consulting services to get enterprise Hadoop infrastructures up and running.


So, what was the core message at the summit?


Hadoop is here to stay. It’s entering the enterprise. It’s evolving fast and if done right can be a material game-changer in how companies seek to monetize their ever growing data. By the way, data continues to grow at 50 percent a year, more than doubling every two years, according to IDC, a technology research firm.


What were some comments/questions heard at both conferences?


“Does hardware make a difference when running Hadoop?” –question at session.

"What problem does Hadoop solve: I/O or scale of data ?" -question during break

“Hadoop is changing how businesses interact with customers from reactive and post-transaction-based to interactive and pro-active-based transactions” –statement by business executive
“Integrating Hadoop with existing IT infrastructure is critical” –top-of-mind for CIOs
“Hadoop 2.0 will be more robust and will focus on reliability, availability and scalability, clearly a focus on the enterprise” -comment at keynote
“Hadoop is a technology, not a solution” –heard at a table during lunch
“Hadoop is very inefficient; the cost of energy doesn’t go away –we’ll need to rethink the data replication factor” -comment during session
“How do I integrate Hadoop with my current IT infrastructure?" -question from session
“Do I need to hire new staff, data scientists, and Hadoop experts?” -IT manager at conference
“I've heard Hadoop is best when close to data sources, not a good idea to have Hadoop in the cloud, is this true?” -question at session
“Is Hadoop secure and reliable for the enterprise?" -comment in the hallway


What’s next for Hadoop?

According to Eric Baldeschwieler, CTO at Hortonworks, this is just the beginning. Hadoop 2.0 (estimated general availability is 2013) promises to have many features and functionalities that will make it more robust and enterprise-ready. Hadoop 2.0 will focus on scale, always-on cluster availability, federation and reliability.

How do you get started with Hadoop?

Well, as you can see, there are many new questions to consider. You can do it yourself, which can be very interesting and also time consuming or you can have a  company like NetApp and its partners help you. NetApp runs a robust Hadoop implementation to support key business processes; for more information, see the arcticle in the WSJ CIO Journal.
Whatever you decide to do regarding Hadoop, stay tuned for more and interesting innovation in big data, as it is clear as we are at the beginning of a great journey.