Top 10 Trends in Big Data
Big data is no longer just a buzzword. Researchers at Forrester have “found that, in 2016, almost 40 percent of firms are implementing and expanding big data technology adoption. Another 30 percent are planning to adopt big data in the next 12 months.”
Similarly, the Big Data Executive Survey 2016 from NewVantage Partners found that 62.5 percent of firms now have at least one big data project in production, and only 5.4 percent of organizations have no big data initiatives planned or underway.
Researchers say the adoption of big data technologies is unlikely to slow anytime soon. IDC predicts that the big data and business analytics market will increase from $130.1 billion this year to more than $203 billion in 2020. “The availability of data, a new generation of technology, and a cultural shift toward data-driven decision making continue to drive demand for big data and analytics technology and services,” said Dan Vesset, group vice president, analytics and information management. “This market is forecast to grow 11.3 percent in 2016 after revenues reached $122 billion worldwide in 2015 and is expected to continue at a compound annual growth rate (CAGR) of 11.7 percent through 2020.”
While it’s clear that the big data market will grow, how organizations will be using their big data is a little less clear. New big data technologies are entering the market, while use of some older technologies continues to grow. This slideshow covers ten top trends that will likely shape the big data market in 2017 and beyond.
- Open Source
Open source applications like Apache Hadoop, Spark and others have come to dominate the big data space, and that trend looks likely to continue. One survey found that nearly 60 percent of enterprises expect to have Hadoop clusters running in production by the end of this year. And according to Forrester, Hadoop usage is increasing 32.9 percent per year.
Experts say that in 2017, many enterprises will expand their use of Hadoop and NoSQL technologies, as well as looking for ways to speed up their big data processing. Many will be seeking technologies that allow them to access and respond to data in real time.
- In-Memory Technology
One of the technologies that companies are investigating in an attempt to speed their big data processing is in-memory technology. In a traditional database, the data is stored in storage systems equipped with hard drives or solid state drives (SSDs). In-memory technology stores the data in RAM instead, which is many, many times faster. A report from Forrester Research forecasts that in-memory data fabric will grow 29.2 percent per year.
Several different vendors offer in-memory database technology, notably SAP, IBM, Pivotal.
- Machine Learning
As big data analytics capabilities have progressed, some enterprises have begun investing in machine learning (ML). Machine learning is a branch of artificial intelligence that focuses on allowing computers to learn new things without being explicitly programmed. In other words, it analyzes existing big data stores to come to conclusions which change how the application behaves.
- Predictive Analytics
Predictive analytics is closely related to machine learning; in fact, ML systems often provide the engines for predictive analytics software. In the early days of big data analytics, organizations were looking back at their data to see what happened and then later they started using their analytics tools to investigate why those things happened. Predictive analytics goes one step further, using the big data analysis to predict what will happen in the future.
- Intelligent Apps
Another way that enterprises are using machine learning and AI technologies is to create intelligent apps. These applications often incorporate big data analytics, analyzing users’ previous behaviors in order to provide personalization and better service. One example that has become very familiar is the recommendation engines that now power many ecommerce and entertainment apps.
- Intelligent Security
Many enterprises are also incorporating big data analytics into their security strategy. Organizations’ security log data provides a treasure trove of information about past cyberattack attempts that organizations can use to predict, prevent and mitigate future attempts. As a result, some organizations are integrating their security information and event management (SIEM) software with big data platforms like Hadoop. Others are turning to security vendors whose products incorporate big data analytics capabilities.
- Internet of Things (IoT)
The Internet of Things is also likely to have a sizable impact on big data. According to a September 2016 report from IDC, “31.4 percent of organizations surveyed have launched IoT solutions, with an additional 43 percent looking to deploy in the next 12 months.”
- Edge Computing
One new technology that could help companies deal with their IoT big data is edge computing. In edge computing, the big data analysis happens very close to the IoT devices and sensors instead of in a data center or the cloud. For enterprises, this offers some significant benefits. They have less data flowing over their networks, which can improve performance and save on cloud computing costs. It allows organizations to delete IoT data that is only valuable for a limited amount of time, reducing storage and infrastructure costs. Edge computing can also speed up the analysis process, allowing decision makers to take action on insights faster than before.
- High Salaries
For IT workers, the increase in big data analytics will likely mean high demand and high salaries for those with big data skills. According to IDC, “In the U.S. alone there will be 181,000 deep analytics roles in 2018 and five times that many positions requiring related skills in data management and interpretation.”
As a result of that scarcity, Robert Half Technology predicts that average compensation for data scientists will increase 6.5 percent in 2017 and range from $116,000 to $163,500. Similarly, big data engineers should see pay increases of 5.8 percent with salaries ranging from $135,000 to $196,000 for next year.
As the cost of hiring big experts rises, many organizations are likely to be looking for tools that allow regular business professionals to meet their own big data analytics needs. IDC has previously predicted “Visual data discovery tools will be growing 2.5 times faster than rest of the business intelligence (BI) market. By 2018, investing in this enabler of end-user self service will become a requirement for all enterprises.” Source