Big data: The critical ingredient

In 2011 a clever startup in Seattle called Decide.com opened its online doors with fantastically bold ambitions. It wanted to be a price-prediction engine for zillions of consumer products. But it planned to start relatively modestly: with every possible tech gadget, from mobile phones and flat-screen televisions to digital cameras. Its computers sucked down data feeds from e-commerce sites and scoured the web for any other price and product information they could find.

Prices on the web constantly change throughout the day, dynamically updating based on countless, intricate factors. So the company needed to collect pricing data at all times. It isn’t just big data but “big text” too, since the system had to analyse words to recognize when a product was being discontinued or a newer model was about to launch – information that consumers ought to know that affect prices.

A year later, Decide.com was analyzing four million products using over 25bn price observations. It identified oddities about retailing that people had never been able to “see” before, like the fact that prices might temporarily increase for older models once new ones are introduced. Most people would purchase the older one figuring it would be cheaper, but depending on when they clicked “buy”, they might pay more.

Big data mindset

As online stores increasingly use automated pricing systems, Decide.com can spot unnatural, algorithmic price spikes and warn consumers to wait. The company’s predictions, according to its internal measurements, are accurate 77 percent of the time and provide buyers with average potential savings of around $100 per product.

So confident is the company, that in cases where its predictions prove incorrect, Decide.com will reimburse the price difference to paying members of the service.

On the surface, Decide.com sounds like many promising startups that aim to harness information in new ways and earn an honest dollar for their effort. What makes Decide.com special isn’t the data: the company relies on information it licenses from e-commerce sites and scrapes off the web, where it is free for the taking. It also isn’t technical expertise: the company doesn’t do anything so complex that the only engineers in the world capable of pulling it off are the ones at its own office.

Rather, although collecting the data and technical skills are important, the essence of what makes Decide.com special is the idea: the company has a “big-data mindset”. It spied an opportunity and recognized that certain data could be mined to reveal valuable secrets.

Data is becoming a new source of value in large part because of its “option value,” that is, the myriad of secondary purposes to which it can be put. Companies are cropping up that fit into the information value chain. It will have big implications for organizations and for individuals, both in their careers and in their everyday lives.

Three types of big-data companies have cropped up, which can be differentiated by the value they offer. Think of it as the data, the skills, and the ideas.

First is the data. These are the companies that have the data or at the least have access to it. But perhaps that is not what they are in the business for. Or, they don’t necessarily have the right skills to extract the data’s value or to generate creative ideas about what is worth unleashing. The best example is Twitter, which obviously enjoys a massive stream of data flowing through its servers but turned to two independent firms, GNIP and Data Sift, to license it to others to use.

Second are skills. This group often includes the consultancies, technology vendors, and analytics providers who have special expertise and do the work, but probably do not have the data themselves nor the ingenuity to come up with the most innovative uses for it. In the case of Walmart, for example, the retailer turned to the specialists at Teradata, a data-analytics firm, to help tease out insights.

Third is the big-data mindset. For certain firms, the data and the know-how are not the main reasons for their success. What sets them apart is that their founders and employees have unique ideas about ways to tap data to unlock new forms of value. In the case of Decide.com, it needed to license the data from other sites and could pick up talent from the computer scientists coming out of the nearby University of Washington. But what set the site apart was creativity of its founder, Oren Etzioni, a professor at the school and serial big-data entrepreneur.

The critical ingredient

So far, the first two of these elements get the most attention: the skills, which today are scarce, and the data, which seems abundant. A new profession has emerged in recent years, called “data scientist,” which combines the skills of the statistician, software programmer, infographics designer, and storyteller. Instead of squinting into a microscope to unlock a mystery of the universe, the data scientist peers into databases to make a discovery.

The McKinsey Global Institute proffers dire predictions about the dearth of data scientists now and in the future (which today’s data scientists like to cite to feel special and to pump up their salaries).

Hal Varian, Google’s chief economist, famously calls a statistician the “sexiest” job around. “If you want to be successful, you want to be complementary and scarce to something that is ubiquitous and cheap,” he says. “Data is so widely available and so strategically important that the scarce thing is the knowledge to extract wisdom from it. That is why statisticians, and database managers and machine learning people, are really going to be in a fantastic position.”

However, all the focus on the skills and the downplaying of the importance of the data may prove to be short-lived. As the industry evolves, the paucity of skilled personnel will be overcome. Moreover, the much vaunted ideas may also become less important over time, just as happened with the web between 1994 and 2000.

Instead, there is a mistaken belief that just because there is so much data around, it is free for the taking or its value is meager. In fact, the very opposite is true: the data is the critical ingredient.

This is an excerpt from Big Data: A Revolution That Will Transform How We Live, Work, and Think by Viktor Mayer-Schönberger and Kenneth Cukier (Houghton Mifflin Harcourt, 2013)