big dataIt seems every organisation has either jumped or is seriously contemplating jumping onto the Big Data bandwagon. In an industry where the hype is often followed by the despair, I feel somewhat ashamed that the IT Industry that I work in pushes the barrow often before the horse. As a result, organisations get hyped up around the advantages and the outcomes of the technology without putting in place the safeguards to ensure project success. Often when I talk to my customers about Big Data, I like to ask some basic questions to ensure that whatever we deliver is achievable and is not going to be just another piece of technology sitting on the shelf.

A survey by Infochimps.com asked IT professionals what are the challenges in a Big Data project and it delivered some interesting yet not to be unexpected results:

The following are some basic rules we should all be adopting if we are to embrace a Big Data Project:

  1. Don’t start a Big Data project without understanding the value- In my previous blog I talked about the complexities of putting a value on data from an asset point of view. But it should go without saying that when you go up to your CFO with cap in hand to kick start a Big Data Project, you should know exactly how to demonstrate a Return on Investment (ROI). Not only do you need to know intimately the ROI of your project, but be able to discuss in detail the benefits such as increased revenue. But don’t just say that revenue will increase by x%. You need to weave the benefits and tie them back to functionality. For example “The big data project will allow us to better understand our customers through the use of sentiment analytics. The results of which will allow us to respond quicker to customer’s needs resulting increased revenue”.
  2. Don’t ignore the wider Enterprise story- More often than not, we tend to pigeon hole Big Data projects into specific departments or functions. Big Data brings with it the power to draw upon data across an organisation and merge it together to create insights. HR, Sales and Customer data coming together to deliver value. Therefore don’t ignore the impact that your Big Data project will have across the organisation.  If you can tie in your ROI discussion in point 1 above to go beyond just your area by demonstrating value across the organisation, then the stronger your project will be and the more visibility it will receive.
  3. A big data project is not a technology project- A lot of the hype around Big Data is the products themselves and as such when we go to deliver a big data project it’s often about the technology. However a Big Data project should be seen as a business change project. Big Data changes the way we interact with our suppliers, customers and peers. Executive support from the CEO/CFO/CTO  is a must to ensure that the project garners the right focus and achieves the right organizational shift to make it a success.
  4. “A Big Data project is more about Business Change than the technology itself”.
  5. It pays to have the right type of Data Scientists onboard- The problem with the industry is that it seems that there are a lot more people calling themselves data scientists without any real qualification or experience. And most probably one of the reasons for this change is the higher pay grades that real data scientists demand because of their specialist skills. So for a project to be a success, you must build a strong delivery team which involves making sure you get a data scientist with the runs on the board. Don’t skimp lower and get a data/business analyst who now calls themselves a data scientist as you will get lower than expected results.
  6. Build a support  structure from the beginning- It’s very easy to get caught up in the moment and therefore overlook some of the less exciting components of a  Big Data project such as Data Quality, Data Governance and Metadata. However these components are often the building blocks to the success of the project. Get started early on these and results will come quicker and the output will be much more valuable to the end users.
  7. Build for tomorrow- The pace of change in this industry is extreme. New products, features and technology hit the market everyday. The problem this speed of change presents in a project is the issue of scope creep. If you design a Big Data solution with only the technology in mind today, then by the time it’s delivered it’s value will be diminished, or you’ll be making that many changes to the design during the project that it will never be finished. So be bold and become visionary. Understand the possibilities that big data can have on your organisation in the future and design a system that will deliver this.
  8. Involve the organisation. Often the vision of a Big Data project comes from highly paid consultants or someone from within the tech team. But Big Data is a pervasive technology and as such you should involve all parts of the organisation to contribute to the vision. Involve sales, marketing, HR and customer facing staff to bring ideas to the table. What do they want to see and be able to use to make their jobs easier? Not only may you get ideas not previously thought of, but subconsciously you are getting the organisation to embrace the concepts of Big Data as they have had the input. Read more