How To Choose The Best Tool For Your Big Data Project

By Angus Kidman

Trying to choose the right tool for a big data project? This chart (and three simple rules) can help guide you through the options.

This chart is based on one shown by Microsoft Research senior research program manager Wenming Ye during a presentation at Build 2014 last week.

“While choosing appropriate tools is important, skills remains the biggest challenge, Ye noted. “There’s a lot of talk about challenges in the tools and challenges in the data,” he said. “But what’s really important is actually the people. There’s a lack of understanding — we really have a lack of people who are able to understand and use these distributed tools. And it’s no-one’s fault — a lot of these tools are very difficult.”

Yen suggest three key rules when dealing with big data:

• Make sure that you’re using data to drive decisions, and not merely tracking it for its own sake.
• Continuously update and refine your metrics.
• Use automation to conduct more experiments and ask more questions.

The chart divides big data tasks into three areas: batch processing, interactive analysis and real-time stream processing.