Human generated content is comprised of all the files and e-mails that we create every day, all the presentations, word processing documents, spread sheets, audio files and other documents our employers ask us to produce hour-by-hour. These are the files that take up the vast majority of digital storage space in most organisations—they are kept for significant amounts of time and they have huge amounts of metadata associated with them.

Human generated content is huge, and its metadata is even bigger. Metadata is the information about a file: who might have created it, what type of file it is, what folder it is stored in, who has been reading it and who has access to it. The content and metadata together make up human generated big data.

The problem is that most of us, meaning organisations and governments, are not yet equipped with the tools to exploit human generated big data. The conclusion of a recent survey of over 1000 Internet experts and other Internet users, published by the Pew Research Centre and the Imagining the Internet Center at Elon University, is that the world may not be ready to properly handle and understand Big Data.

These experts have come to the conclusion that the huge quantities of data, which they term "digital exhaust," which will be created by the year 2020 could very well enhance productivity, improve organisational transparency and expand the frontier of the "knowable future." However they are concerned about whose hands this information is in and whether government or corporates will use this information wisely.


The survey found that "…human and machine analysis of big data could improve social, political and economic intelligence by 2020. The rise of what is known as Big Data will facilitate things like real-time forecasting of events; the development of "inferential software" that assesses data patterns to project outcomes; and the creation of algorithms for advanced correlations that enable new understanding of the world."

Of those surveyed, 39% of the Internet experts asked agreed with the counter-argument to Big Data’s benefits, which posited that "Human and machine analysis of Big Data will cause more problems than it solves by 2020. The existence of huge data sets for analysis will engender false confidence in our predictive powers and will lead many to make significant and hurtful mistakes. Moreover, analysis of Big Data will be misused by powerful people and institutions with selfish agendas who manipulate findings to make the case for what they want."

As one of the study’s participants, entrepreneur Bryan Trogdon put it: "Big Data is the new oil," observing that, "…the companies, governments, and organisations that are able to mine this resource will have an enormous advantage over those that don’t. With speed, agility, and innovation determining the winners and losers, Big Data allows us to move from a mindset of ‘measure twice, cut once’ to one of ‘place small bets fast.’"