big dataIs it time for your company to get into big data? Wrong question—you’re already in it. The real question is whether you will seize the opportunity or be overwhelmed by it.

The fact is that all businesses are on the big data adoption curve. Some are just getting started; others have well honed plans. The rewards will go to those that figure out how to manage the growing terabytes of data they collect, glean insights from that data, and capitalize on them. At this point, that’s a minority of companies.

Big data isn’t a “should we or shouldn’t we” proposition, nor is it a question of timing. All businesses are being swept into the world of big data, and it’s happening now.

A better way to think about big data is to understand where your company is on the implementation curve and develop a plan for moving up the curve as quickly as possible. With that in mind, here are eight best practices to get you there.

  1.     Data creation. With our smartphones, social media apps, online shopping, video streaming, and web surfing, we all create what Oracle ORCL +0.3% big data strategist Paul Sonderegger calls “data exhaust.”  In other words, your company’s customers and employees have become data output machines, leaving a digital trail wherever they go. These digital trails can be measured in terabytes, and when you multiply that by a thousand consumers or knowledge workers, you reach a petabyte. Voila—you’ve got big data.
  2.     Tiering.  Step 2 is all about data accumulation. IT departments invest in tape storage, disk storage, flash storage, desktop storage, and cloud storage as a way of taking it all in. Best practices involve tiered storage, where data is moved to the most cost-effective medium. As terabytes become petabytes, Darwinian principles kick in—you must have a well conceived storage strategy to survive.  Even so, capturing data isn’t the same as managing it, and managing it isn’t the same as applying it.
  3.     Optimization. Data management is the bedrock of corporate IT implementation, but the influx of a thousand times more data has upped the game. (A petabyte is a thousand terabytes, and an exabyte is a thousand petabytes.) Exponential growth requires IT teams to rethink what’s worked in the past and bring in new tools to optimize database workloads that are orders of magnitude greater than before. In-memory and other specialized databases, Hadoop, engineered systems, and data-integration middleware are all pieces of the puzzle.
  4.     Analytics. Once your big data house is in order, the next step is to run algorithms against the data in search of insights. Most companies have some experience with business intelligence, but here too new tools and techniques are an absolute necessity. Keep in mind that big data isn’t just more data—it’s new types of data, coming in faster and from new sources. For example, social media buzz is just white noise—and a lot of useless data–without the right tools for the job. Likewise with data from mobile devices and sensors. Your father’s data warehouse won’t cut it.
  5.     Share the wealth. Big data strategists can learn a thing or two from BI efforts of the past, including the incredible value of making this an enterprisewide initiative, not the exclusive realm of a few highly specialized data analysts. Put big data tools and access into the hands of as many employees as possible, and mobilize those capabilities in the same way that you have other enterprise apps. You may want to share big data even more broadly by using APIs to make data sets available outside your organization, as government agencies are doing on Data.gov. by John Foley read more