In case you haven’t noticed Big Data (BD) has been a hot topic in business for some time. Governments and public institutions, not to mention economists, business commentators and technology companies have been promoting the idea of a Big Data Revolution. Technology companies, whether providers of Business Systems, CRM software or Analytics have been promoting BD for over a decade. In 2012 McAfee and Bryynjolfsson, writing in the Harvard Business Review (2012), said BD represented a ‘revolution in management’. Brown et al. (2011) asserted that it would “transform business processes and alter corporate ecosystems”. Manyika et al. (2011) described it as ‘the next frontier for innovation, competition and productivity’.
As computer applications capable of analysing huge data-sets become affordable and readily available there is a collective assumption that BD will benefit organisations, individuals and society. Looking at almost any aspect of our personal and professional lives, it is hard to ignore the impact that technology has had on how we do things. Mobile technologies and smart devices are essential for many of us and given all the hype it is hard not to imagine that this trend will continue.
Yet last year, perhaps surprisingly Gartner the global technology advisor suggested that the hype was over. Having gone over the peak of inflated expectations BD was disappearing into the trough of disillusionment. Perhaps mirroring Gartner, colleagues and I (Matthias et al. 2016) wanted to get beyond the hype and see what was actually going on. In a recent paper we highlighted the lack of credible empirical research in the area and questioned whether BD has fundamentally changed businesses and management. We argued that the challenge for organisations is to develop strategies that exploit BD to add value to the bottom line.
There is certainly money to be made from Big Data, but with a few notable exceptions, few prospectors have found the rich seams of gold promised. ITC continues to be increasingly important in automating business processes and the internet enables businesses to connect with customers in many varied ways. Yet arguably in the 21st Century it is still business as usual for many businesses. Service providers still provide services, manufacturers make stuff and retailers still sell. If anything technology is making the business arena a tougher place as businesses have to adapt to a volatile environment where consumer led innovation seems the norm.
Big Data
Arguably, just as in the gold rushes of the 19th century, it is those selling the dream who have cashed in so far. Prospectors need to know where the gold might be and access to the right tools. Fortunes are to be made selling the metaphorical claims (databases and datastreams ) shovels (BD software) and advice (consultancy) the prospectors are desperate for. Few prospectors turn a profit and those that do often have the foresight to sell the claim quickly at a huge profit and let someone else do the hard graft.
I started work in the 1980s and can remember the revolution that ERP systems running on isolated mainframe computers promised, long before we’d ever even heard of the World Wide Web. Even at the time it was evident that those working for the businesses selling “the dream” were doing rather better than the businesses desperate to improve their productivity. And perhaps, just as then we should not be surprised. In 1984 the economist Robert Solow pointed out that, “You can see the computer age everywhere but in the productivity statistics.”
Solow’s Paradox suggests that the introduction of information technology does not drive up productivity. Some argue that there is simply a time lag, and productivity improvements follow years/decades latter. Others suggest that the benefits simply “evaporate”: the promise of new capabilities that will create competitive advantage is never realised because the technology simply diffuses and becomes available to all. What seems clear to me is that information is in itself not going to revolutionize business. If management is about creating and maintaining an organization that can thrive in a changing environment.
Businesses operate in a complex changing environment and however much data we collect the dataset is always an historical record of what has happened. The more data we have, the more complex that environment becomes and so paradoxically we can never have sufficient data to know what will happen in the future. Developing the management skills to be able to work in an environment of seemingly limitless data is arguably the biggest challenge we face. Solow’s paradox and Gartner’s hype cycle suggest we probably don’t have the tools we need just yet, but after the Gold Rush, we may be waking up to a world in which business success is increasingly built on Small Data. Managers who are able to quickly see patterns and extract relatively simple meanings from an endless stream of data, in order to make decisions are those who will thrive in the future.