The market for software related to the Hadoop and MapReduce programming frameworks for large-scale data analysis will jump from $77 million in 2011 to $812.8 million in 2016, a compound annual growth rate of 60.2%, according to a new report by analyst firm IDC, IT World Canada says.
Hadoop is an open-source implementation of the MapReduce framework. It is hosted at the Apache Software Foundation along with a number of supporting software projects, including the Hadoop Distributed File System and Pig programming language.
MapReduce and Hadoop are based on the principle of splitting up large volumes of data and then processing the chunks in parallel across large numbers of nodes.
It’s closely associated with the industry buzzword ‘big data’, which refers to the ever-larger volumes of information, particularly of unstructured form, being generated by Web sites, social media, sensors and other sources.
Big data has received a lot of focus, as companies aim to crunch structured and unstructured data to see around corners, ZDNet notes.
The growth of big data software could also pose a threat to database incumbents, such as Oracle. IDC expects the Hadoop-MapReduce market to develop like Linux did. Linux began with a lot of attention and a small market, and then grew to be commonplace in most data centres.
The one wildcard for big data growth will be talent to crunch the figures as well as analyse them.
Together, Hadoop and MapReduce are “taking the software world by storm”, says Carl Olofson, an information management researcher at IDC, Information Age states.
“[They are] inspiring a wide range of projects that collect both structured and unstructured data and produce output that may be used to answer a single question,” he adds.
“They serve as the foundation for a range of questions, queries or searches; or [can] be loaded into a data warehouse for more systematic and repeatable query.”