{
    "version" : "https://jsonfeed.org/version/1",
    "content" : "news",
    "type" : "single",
    "title" : "Gather Your Agency’s Public Data with Let Me Get That Data for You |Digital.gov",
    "description": "Gather Your Agency’s Public Data with Let Me Get That Data for You",
    "home_page_url" : "/preview/gsa/digitalgov.gov/bc-archive-content-3/","feed_url" : "/preview/gsa/digitalgov.gov/bc-archive-content-3/2015/03/31/gather-your-agencys-public-data-with-let-me-get-that-data-for-you/index.json","item" : [
    {"title" :"Gather Your Agency\u0026#8217;s Public Data with Let Me Get That Data for You","summary" : "In case you missed it, U.S. Open Data recently launched a tool called: Let Me Get That Data For You (LMGTDY). The name is a play on the very funny Let Me Google That For You website. How LMGTDFY works Let Me Get That Data For You searches any website for","date" : "2015-03-31T10:38:06-04:00","date_modified" : "2025-01-27T19:42:55-05:00","authors" : {"rebecca-williams" : "Rebecca Williams"},"topics" : {
        
            "open-data" : "Open data",
            "software-engineering" : "Software engineering"
            },"branch" : "bc-archive-content-3",
      "filename" :"2015-03-31-gather-your-agencys-public-data-with-let-me-get-that-data-for-you.md",
      
      "filepath" :"news/2015/03/2015-03-31-gather-your-agencys-public-data-with-let-me-get-that-data-for-you.md",
      "filepathURL" :"https://github.com/GSA/digitalgov.gov/blob/bc-archive-content-3/content/news/2015/03/2015-03-31-gather-your-agencys-public-data-with-let-me-get-that-data-for-you.md",
      "editpathURL" :"https://github.com/GSA/digitalgov.gov/edit/bc-archive-content-3/content/news/2015/03/2015-03-31-gather-your-agencys-public-data-with-let-me-get-that-data-for-you.md","slug" : "gather-your-agencys-public-data-with-let-me-get-that-data-for-you","url" : "/preview/gsa/digitalgov.gov/bc-archive-content-3/2015/03/31/gather-your-agencys-public-data-with-let-me-get-that-data-for-you/","content" :"\u003cp\u003eIn case you missed it, U.S. Open Data \u003ca href=\"https://usopendata.org/2015/02/18/lmgtdfy/\"\u003erecently launched a tool\u003c/a\u003e called: \u003ca href=\"http://lmgtdfy.usopendata.org/\" target=\"_blank\"\u003eLet Me Get That Data For You (LMGTDY)\u003c/a\u003e. The name is a play on the very funny \u003ca href=\"http://lmgtfy.com/\" target=\"_blank\"\u003eLet Me Google That For You\u003c/a\u003e website.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://s3.amazonaws.com/digitalgov/_legacy-img/2015/03/Screen-Shot-2015-03-23-at-6.04.24-PM.png\"\u003e\u003cdiv class=\"image\"\u003e\n  \u003cimg\n    src=\"https://s3.amazonaws.com/digitalgov/_legacy-img/2015/03/Screen-Shot-2015-03-23-at-6.04.24-PM-669x400.png\"\n    alt=\"screenshotofNRELdataserarch\"/\u003e\u003c/div\u003e\n\n\u003c/a\u003e\u003c/p\u003e\n\u003ch2 id=\"how-lmgtdfy-works\"\u003eHow LMGTDFY works\u003c/h2\u003e\n\u003cp\u003eLet Me Get That Data For You searches any website for data in machine-readable formats and provides a list. Here is U.S. Open Data’s background reasoning for creating this tool:\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eWhen government agencies create an open data repository, they need to start by \u003ca href=\"http://how-to.usopendata.org/basics/inventorying-data.html\"\u003einventorying the data that the agency is already publishing on their website\u003c/a\u003e. This is a laborious process. It means searching their own site with a query like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode\u003esite:example.gov filetype:csv OR filetype:xls OR filetype:json\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen they have to read through all of the results, download all of the files, and create a spreadsheet that they can load into their repository. It’s a lot of work, and as a result it too often goes undone, resulting in a data repository that doesn’t actually contain all of that agency’s data.\u003c/p\u003e\n\u003cp\u003eRealizing that this was a common problem, we hired Silicon Valley Software Group to create a tool to automate the inventorying process. We worked with Dan Schultz and Ted Han, who created a system built on \u003ca href=\"https://www.djangoproject.com/\"\u003eDjango\u003c/a\u003e and \u003ca href=\"http://www.celeryproject.org/\"\u003eCelery\u003c/a\u003e, using Microsoft’s great \u003ca href=\"https://datamarket.azure.com/dataset/bing/search\"\u003eBing Search API\u003c/a\u003e as its data source. The result is a free, installable tool, which produces a CSV file that lists all CSV, XML, JSON, XLS, XLSX, XML, and Shapefiles found on a given domain name.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cdiv\u003e\n  \u003cp\u003e\n    The results were formerly limited to 300 files due to \u003ca href=\"https://github.com/opendata/lmgtdfy/issues/26\"\u003ethe cost of querying the Bing API\u003c/a\u003e, but due to a donation by Microsoft on March 23, 2015, the tool can currently return \u003ca href=\"https://twitter.com/opendata/status/580117534583713793\"\u003e2,000 files\u003c/a\u003e. Moreover, should an agency want to expand the query parameters: the code behind the site is \u003ca href=\"https://github.com/opendata/lmgtdfy\"\u003eall open source\u003c/a\u003e.\n  \u003c/p\u003e\n  \u003ch2\u003e\n    Use LMGTFY to vet your Public Data Listing\n  \u003c/h2\u003e\n  \u003cp\u003e\n    The \u003ca href=\"https://project-open-data.cio.gov/policy-memo/\"\u003eUS Federal Open Data Policy\u003c/a\u003e requires CFO-Act agencies to catalog all of their data in Enterprise Data Inventories. Tools like Let Me Get That Data For You and \u003ca href=\"https://usopendata.org/2014/05/23/municipal-data/\"\u003eGoogle\u0026#8217;s Advanced Search\u003c/a\u003e (e.g. search \u003ca href=\"https://www.google.com/search?as_q=\u0026as_epq=\u0026as_oq=\u0026as_eq=\u0026as_nlo=\u0026as_nhi=\u0026lr=\u0026cr=\u0026as_qdr=all\u0026as_sitesearch=gsa.gov\u0026as_occt=any\u0026safe=images\u0026tbs=\u0026as_filetype=xls\u0026as_rights=#as_qdr=all\u0026q=site:gsa.gov+filetype:xls\"\u003eby domain and by file type\u003c/a\u003e) and searching across shared drives by \u003ca href=\"http://windows.microsoft.com/en-us/windows7/advanced-tips-for-searching-in-windows\"\u003efile type\u003c/a\u003e can help agencies make sure that their required Enterprise Data Inventories are truly comprehensive.\n  \u003c/p\u003e\n\u003c/div\u003e"}
  ]
}