{
    "version" : "https://jsonfeed.org/version/1",
    "content" : "news",
    "type" : "single",
    "title" : "The API Briefing: the Challenge of Government’s Dark Data |Digital.gov",
    "description": "The API Briefing: the Challenge of Government’s Dark Data",
    "home_page_url" : "/preview/gsa/digitalgov.gov/bc-archive-content-3/","feed_url" : "/preview/gsa/digitalgov.gov/bc-archive-content-3/2015/06/17/the-api-briefing-the-challenge-of-governments-dark-data/index.json","item" : [
    {"title" :"The API Briefing: the Challenge of Government’s Dark Data","summary" : "By now, you are familiar with “big data” or datasets that are so large that they cannot be analyzed by conventional analytical methods. You may have heard of “long data” which is data that has a temporal context. I work with long data when I analyze hiring patterns over time in workforce data. There is","date" : "2015-06-17T10:00:23-04:00","date_modified" : "2025-01-27T19:42:55-05:00","authors" : {"bbrantley" : "Bill Brantley"},"topics" : {
        
            "analytics" : "Analytics",
            "application-programming-interface" : "Application programming interface",
            "open-data" : "Open data"
            },"branch" : "bc-archive-content-3",
      "filename" :"2015-06-17-the-api-briefing-the-challenge-of-governments-dark-data.md",
      
      "filepath" :"news/2015/06/2015-06-17-the-api-briefing-the-challenge-of-governments-dark-data.md",
      "filepathURL" :"https://github.com/GSA/digitalgov.gov/blob/bc-archive-content-3/content/news/2015/06/2015-06-17-the-api-briefing-the-challenge-of-governments-dark-data.md",
      "editpathURL" :"https://github.com/GSA/digitalgov.gov/edit/bc-archive-content-3/content/news/2015/06/2015-06-17-the-api-briefing-the-challenge-of-governments-dark-data.md","slug" : "the-api-briefing-the-challenge-of-governments-dark-data","url" : "/preview/gsa/digitalgov.gov/bc-archive-content-3/2015/06/17/the-api-briefing-the-challenge-of-governments-dark-data/","content" :"\u003cdiv class=\"image\"\u003e\n  \u003cimg\n    src=\"https://s3.amazonaws.com/digitalgov/_legacy-img/2015/06/600-x-400-Server-room-Tomasz-Wyszolmirski-iStock-Thinkstock-178134397.jpg\"\n    alt=\"Server room\"/\u003e\u003c/div\u003e\n\n\n\u003cp\u003eBy now, you are familiar with “\u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/2015/01/20/trends-big-data-and-gov-in-2015/\"\u003ebig data\u003c/a\u003e” or datasets that are so large that they cannot be analyzed by conventional analytical methods. You may have heard of “long data” which is data that has a temporal context. I work with long data when I analyze hiring patterns over time in workforce data. There is also “small data.” Small data are datasets that describe a current condition. For example, if you have a smart home appliance such as a smart thermostat or a home security system, that appliance is constantly monitoring data such as temperature or if a door is open. The appliance does not store the data (or may store the data for a short time) but only reacts if there is a change in the data. As you can see, there are many types of data.\u003c/p\u003e\n\u003cp\u003eSo, what do you call data that is collected but isn’t analyzed or made readily available? “Dark data” is becoming a major issue for organizations as the ability to collect data is far outpacing the ability to analyze data. According to an article on CIO.com, \u003cem\u003e\u003ca href=\"http://www.cio.com/article/2926089/data-analytics/shedding-light-on-dark-data.html\" target=\"_blank\"\u003eShedding Light on Dark Data\u003c/a\u003e\u003c/em\u003e, dark data is created because the organization may not be aware that the data is being collected.\u003c/p\u003e\n\u003cp\u003eWhen different parts of the organization are not collaborating on data collection or data analysis, they could also create dark data. Also, the organization may not have the tools or the analytical skills to analyze the collected data. Finally, dark data may happen because the organization does not have good documentation on the organizational datasets. Dark data is like an attic or storage shed, where boxes of information are stored away with the promise that we will, one day, get around to working with the data.\u003c/p\u003e\n\u003cp\u003eThe federal government is a massive collector of information. \u003ca href=\"http://www.data.gov/\" target=\"_blank\"\u003eData.gov\u003c/a\u003e currently has 131,941 datasets and I am sure that is a small fraction of the total number of government datasets. An even smaller fraction of government datasets has been translated into APIs. As I have documented in earlier postings, apps built on \u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/2015/04/15/the-api-briefing-how-essential-is-government-data-to-the-american-economy/\"\u003egovernment APIs have become a vital part of the U.S. economy\u003c/a\u003e and provide valuable public services. The next big challenge for federal government data is to explore and utilize the dark data that exists in all of the government agencies. Much progress has been made, but more work is needed to unearth the hidden treasures in the federal government’s dark data.\u003cem\u003e*API – Application Programming Interface; how software programs and databases share data and functions with each other. Check out \u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/2013/04/30/apis-in-government/\"\u003eAPIs in Government\u003c/a\u003e for more information.\u003c/em\u003e\n\u003cem\u003eEach week, “\u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/topics/application-programming-interface/\"\u003eThe API Briefing\u003c/a\u003e” will showcase government APIs and the latest API news and trends. Visit this column every week to learn how government APIs are transforming government and improving government services for the American people. If you have ideas for a topic or have questions about APIs, please contact me via \u003ca href=\"mailto:%20bill.brantley@wdc.usda.gov\" target=\"_blank\"\u003eemail\u003c/a\u003e. All opinions are my own and do not reflect the opinions of the USDA and GSA.\u003c/em\u003e\u003c/p\u003e\n"}
  ]
}