Trends on Tuesday: Big Data Equals Big Challenges

According to an article from Readwrite, the amount of money going to big data projects is steadily increasing despite widespread failure to achieve many results. For big data-related projects in global organizations, a total of $31 billion was spent in 2013 and that amount is expected to top $114 billion by 2018. The recognition that big data is important is present, but the results from big data projects have not illustrated this to the full extent. Here are a few challenges organizations are having when it comes to big data:

• Data is separate from workflow. One challenge is that many organizations have not fully integrated data into their work processes. Capgemini conducted a survey (PDF) and found that 79% of enterprises haven’t completely integrated data sources from across their organizations.
• Data management. Another challenge is the idea of data management. Within organizations, is your data being managed or is it managing you? The best analogy that comes to mind would be that of comparing a lake to a reservoir. Is your data easily portable or not portable? Is your data filtered or raw?
• Privacy. An additional challenge is the issue of privacy. The government could do more with big data, but has to be wary about the issues of sharing information and the challenges that come with it, especially, with Personally Identifiable Information (PII). Even with the collection of non-PII, there is still a concern of the federal government acting like “Big Brother.” The public may not be open to the idea of agencies sharing information despite the fact that it could lead to a more efficient, collaborative and open government.
• Multiple data sources. The use of big data usually elicits actions that require cross-collaboration across teams, functional areas and agencies. When data comes from multiple sources, the collection and analysis of the data becomes more difficult due to things such as differences in how data is used and collected.
  

Despite all the challenges above, there are some agencies that have successfully taken on big data projects. According to a study conducted on big data projects in the federal government, the agencies that are most successful are described as being active collaborators, having an automated and integrated data workflow, and using data to guide decision making.

These are just a few of the challenges and characteristics that are associated with big data. Do you know of any others? If so, please share below and comment.