The Challenge and Promise of Big Data

As the Internet has grown, so has the amount of data and the extent to which data is valued – most notably by the government and publicly traded companies, although even small businesses appreciate the insights they collect from Google Analytics and similar services. Stockpiling and scrutinizing data is considered an extraordinarily powerful way to deliver better products; understand how people use websites and applications; track employee performance; determine the most effective marketing language; and more.

According to projections released by International Data Corporation, the big data market will maintain a CAGR of 26% through 2018, when sales of big data services will hit $42 billion.

The value of the market tells us that businesses are investing large sums of money in companies that specialize in the collection, comprehension, and integration of vast pools of data – in other words, services that can turn the information into better results.

How big of a focus is big data? Well, the spaceship people are in the game.

Chris Mattman of NASA says of his employer’s research in this area, “NASA in total is probably managing several hundred petabytes, approaching an exabyte.”

Forbes contributor Joshua Steimle notes that an exabyte is an extraordinarily vast amount of information, 1 billion gigabytes.

Awash in a Sea of Numbers?

With the technology we now have available, data can accrue so rapidly for enterprises that it amounts to overload for traditional databases – both in terms of keeping it and making any sense out of it. Companies don’t have to build rockets in order to feel somewhat overwhelmed by the sheer enormity of the data at their fingertips, argues Steimle.

Chris Riquier, the Asia-Pacific head of Taylor Nelson Sofres, says he has seen the impact in another area: “Market research was founded on surveying and polling.” Now, instead, it is done in real time with information flowing in from social media, search engines, and numerous other sources. Thanks to the focus of technologists on data science and the continual refinement of analytics, Riquier believes that our “ability to react to the market and make decisions has changed drastically.”

Finding meaningful patterns within big data has been a mystery to many firms. However, data engineers are building systems so strong that the trend toward amassing and processing data as quickly as possible is all but inevitable, per a 2014 Harvard Magazine report. The article suggested that innovative “big algorithm” formulas will allow firms to benefit from connecting and integrating data stores, as well as converting the numbers into more robust and user-friendly imaging.

Drawing on long-established research techniques from the fields of physics and astronomy, technologists and cultural experts bring their different perspectives to bear on the topic for mutual benefit.

Big data authors Viktor Mayer-Schonberger and Kenneth Cukier note that data analytics isn’t just about individual choices but about completely altering the way that businesses make big-picture decisions – after all, predictive analytics is intended to live up to its name, giving those who use it a logically based, all-seeing crystal ball.

Big data successes include:

Google – Google hypothesized that certain search terms were probably indicators that people were beginning to have flulike symptoms. By applying their guesses to real-time searches, the company was able to figure out where the flu was spreading faster than the CDC could.

Canadian Bank – Across the border, Canadian Bank is “using Hadoop, an open source software framework created by Apache,” Steimle explains, “to roll out a program that enables the identification of money laundering and fraud.”

Data & everyday life

NASA and Google are high-profile instances of the focus on big data, but its potential is much broader, extending to any business wanting to get attention online. It is the fuel that powers search engine optimization and social media optimization. Entrepreneurs can use Google’s Webmaster Tools, combined with analytic data from social software, to develop a more successful online presence.

Joe Hall, who helps clients improve their Internet presence through his company Hall Analysis, comments that there are two different basic ways data can be analyzed to improve search engine results:

1. Exploring the patterns within the voluminous data that is directly related to a particular business.

One of Hall’s clients had more than 15 million backlinks – meaning that many, many webpages were sending their traffic to his client.

“At that level it changes the rules for backlink analysis,” Hall comments. Determining the patterns within the data become significantly more complex.

2. Garnering a more sophisticated understanding of their business’s current position and the dynamics in the market as a whole.

Looking at the relationship between different variables through analytics software allows businesses to better understand how to rank prominently and how to get more people to respond to a listing when it does show up in the search engine. Two relevant fields of study are correlation analysis and user behavior analysis, says Hall.

Revving your big data engines

One technology that has been fundamental to data scientists and companies utilizing their services is premium cloud computing. After all, turn-key VPS hosting solutions are based on optimizing speed, reliability, and efficiency through virtualized distribution of resources.

Computer scientist Geoffrey Fox of Indiana University has even noted that the cloud is often faster than a supercomputer.

By Kent Roberts