Big Data and the Role of Intuition
Many people have asked me over the years about whether intuition has a role in the analytics and data-driven organization. I have always reassured them that there are plenty of places where intuition is still relevant. For example, a hypothesis is an intuition about what’s going on in the data you have about the world. The difference with analytics, of course, is that you don’t stop with the intuition — you test the hypothesis to learn whether your intuition is correct.
Another place where intuition is found in analytical companies is in the choice of the business area where analytical initiatives are undertaken. Few companies undertake a rigorous analytical study of what areas need analytics the most! The choice of a target domain is typically based on the gut feelings of executives. For example, at Caesars Entertainment — an early and continuing user of analytics in its business — the initial focus was on analytics for customer loyalty and service. CEO Gary Loveman noted that he knew that Caesars (then Harrah’s) had low levels of customer loyalty across its nationwide network of casinos. He had also done work while at Harvard Business School on the “service profit chain” — a theory that companies that improve customer service can improve financial results. While the theory had been applied and tested in several industries, it hadn’t been applied to gaming firms. But Loveman’s intuition about the value of loyalty and service was enough to propel years of analytics projects in those areas.
Of course, as with hypotheses, it’s important to confirm that your intuitions about where to apply analytics are actually valid. Loveman insists on an ROI from each analytics project at Caesars. Intuition plays an important role at the early stages of analytics strategy, however. In short, intuition’s role may be more limited in a highly analytical company, but it’s hardly extinct.
But how about with big data? Surely intuition isn’t particularly useful when there are massive amounts of data available for analysis. The companies in the online business that were early adopters of big data — Google, Facebook, LinkedIn, and so forth — had so much clickstream data available that no one needed hunches any more, correct?
Well, no, as it turns out. Major big data projects to create new products and services are often driven by intuition as well. Google’s self-driving car, for example, is described by its leaders as a big data project. Sebastian Thrun, a Google Fellow and Stanford professor, leads the project. He had an intuition that self-driving cars were possible well before all the necessary data, maps, and infrastructure were available. Motivated in part by the death of a friend in a traffic accident, he said in an interview that he formed a team to address the problem at Stanford without knowing what he was doing.
At LinkedIn, one of the company’s most successful data products, the People You May Know (PYMK) feature, was developed by Jonathan Goldman (now at Intuit) based on an intuition that people would be interested in what their former classmates and colleagues are up to. As he put it in an interview with me, he was “playing with ideas about how to help people build their networks.” That certainly sounds like an intuitive process. Read more