{
    "version" : "https://jsonfeed.org/version/1",
    "content" : "news",
    "type" : "single",
    "title" : "Developing a Data-Driven Roadmap |Digital.gov",
    "description": "Developing a Data-Driven Roadmap",
    "home_page_url" : "/preview/gsa/digitalgov.gov/cm-topics-button-component/","feed_url" : "/preview/gsa/digitalgov.gov/cm-topics-button-component/2021/06/14/developing-a-data-driven-roadmap/index.json","item" : [
    {"kicker" : "Data","title" :"Developing a Data-Driven Roadmap","deck" : "","summary" : "A Presidential Innovation Fellow working with the U.S. Department of Veterans Affairs shares how complex organizations can create data-driven roadmaps.","date" : "2021-06-14T17:00:00-05:00","date_modified" : "2024-04-02T09:45:13-04:00","authors" : {"babatunde-oguntade" : "Babatunde Oguntade"},"topics" : {
        
            "content-strategy" : "Content Strategy",
            "open-data" : "Open Data"
            },"primary_image" : { "uid" : "pif-card-alt", "alt" :
  "Presidential Innovation Fellows logo", "width" :
  "608", "height" :
  "340", "credit" :
  "", "caption" :
  "Presidential Innovation Fellows", "format" :
  "png" },"branch" : "cm-topics-button-component",
      "filename" :"2021-06-11-developing-a-data-driven-roadmap.md",
      
      "filepath" :"news/2021/06/2021-06-11-developing-a-data-driven-roadmap.md",
      "filepathURL" :"https://github.com/GSA/digitalgov.gov/blob/cm-topics-button-component/content/news/2021/06/2021-06-11-developing-a-data-driven-roadmap.md",
      "editpathURL" :"https://github.com/GSA/digitalgov.gov/edit/cm-topics-button-component/content/news/2021/06/2021-06-11-developing-a-data-driven-roadmap.md","slug" : "developing-a-data-driven-roadmap","url" : "/preview/gsa/digitalgov.gov/cm-topics-button-component/2021/06/14/developing-a-data-driven-roadmap/","weight" : "1","content" :"\u003cp\u003eHealthcare is complex — just like humans. The care needed to ensure the health and wellbeing of people is as unique as the individual receiving care. Today, smart, compassionate healthcare has never been more important. Across the federal government, we are combatting the COVID-19 pandemic and addressing health inequities.\u003c/p\u003e\n\u003cp\u003eGreat healthcare happens through everyday innovation. That’s why \u003ca href = \"https://presidentialinnovationfellows.gov/fellows/babatunde-oguntade\"\u003eI became a Presidential Innovation Fellow\u003c/a\u003e: to help relieve the burden of care through smart tools and compassionate people. My expertise is in process engineering and machine learning, and my passion is using mathematical models to improve performance, efficiency, and safety. This blog shares a deep dive into the small wins that help advance great healthcare. In this case, I explore how to develop a data-driven roadmap for health benefits claims processing.\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eI am working with VA Office of the Chief Technology Officer (CTO) to deliver Veterans’ benefits faster and use deep learning and health insights to optimize benefits dollars and care for Veterans more holistically. Like many federal agencies, VA is a data-rich organization. With massive data comes massive responsibility. Tech for tech’s sake won’t achieve our mission; we need to ensure we’re building the right products and services for the right reasons. To do so, we knew we needed to develop data-driven roadmaps. In this blog, I walk through a data-driven roadmap case study: claims processing with the CTO Benefits Team. I hope it serves as both a strategic and technical example that any data-rich organization can use.\u003c/p\u003e\n\u003cp\u003eVA offers tax-free monthly payments to Veterans who get injured during service or whose existing condition deteriorated as a result of service. Veterans must submit an application to receive these benefits, but the wait period to get a decision from VA is on average greater than 100 days. There are conflicting reasons for the long wait time, making it difficult to determine what areas of the process constitute the greatest bottlenecks and require immediate attention.\u003c/p\u003e\n\u003cp\u003eThis blog presents the results of a big data analysis on the trajectory of a claim from “inception to the completion”, using disability claims data from January 2020 through January 2021. The data from this period contained about 100 million unique records and over 30 unique fields. Our analysis includes:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eDeveloping a simple linear phenomenological model that explained over 92% of variance in the observed processing time,\u003cimg src=\"https://latex.codecogs.com/svg.image?t_{processing}\" title=\"t_{processing}\" /\u003e , thus enabling us to determine the contributions of the different process elements;\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eRanking the different process elements by importance to determine what would lead to the greatest decrease in processing time;\nand\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eProposing solutions to address the problems.\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"the-disability-claims-process-and-performance-measures\"\u003eThe Disability Claims Process and Performance Measures\u003c/h2\u003e\n\u003cp\u003eWhen VA receives a disability claim, an end product is created directly in the \u003cb\u003e\u003cem\u003eVeteran Benefits Management System (VBMS)\u003c/em\u003e\u003c/b\u003e for manual establishment, or through the \u003cb\u003e\u003cem\u003eCorporate Database\u003c/em\u003e\u003c/b\u003e and read by VBMS otherwise. \u003cb\u003e\u003cem\u003eThe end product (EP) system\u003c/em\u003e\u003c/b\u003e is the primary workload monitoring attribute for Veterans Service Centers. The end product creation involves three major steps, which we describe as the \u003cb\u003e\u003cem\u003eContention, Flash, and Special Issues\u003c/em\u003e\u003c/b\u003e processes:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eContention\u003c/strong\u003e: The contention process involves translating the conditions or issues raised by the Veteran into VA parlance. Example: A Veteran may have “ringing in the ear” as a contention, which must be translated to “tinnitus” during the contention process.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFlash\u003c/strong\u003e: The flash process is initially a tagging step that allows claimant-specific details to be added to a claim to ensure proper routing.  Subsequently, evidence-based details are added as flash as more information is gathered on the claim. Example: A common initial flash used in the routing of claims is “Homelessness.” Veterans who indicate they are homeless on their application are prioritized in claims processing.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eSpecial Issues\u003c/strong\u003e: The special issues process is similar to the flash process but the tag involved here is claim-specific. Example: Agent Orange-Vietnam is associated with claims where Veterans contend they were exposed to Agent Orange during their service in Vietnam.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eEnd product creation in VBMS is referred to as the “establishment” of a claim. The difference between when a claim is received and established in VBMS is depicted as \u003cimg src=\"https://latex.codecogs.com/svg.image?t_{establishment}\" title=\"t_{establishment}\" /\u003e and is a measure of the effectiveness of the claim’s intake method.\u003c/p\u003e\n\u003cp\u003eHistorically, it took weeks before claims were established, prompting a flurry of innovative solutions around the intake process. Specifically, VA Benefits Team, including Presidential Innovation Fellow Nelson A. Colon Vargas, PhD, created  \u003ca href = \"https://www.oit.va.gov/news/article/?read=va-launches-smart-tool-to-reduce-veteran-wait-times-for-disability-claims\"\u003ethe first artificial intelligence (AI) solution at VA\u003c/a\u003e to ensure that claims are promptly established. The AI solution known as the \u003cb\u003e\u003cem\u003eClaims Attributes Application Programming Interface version 1.0 (CAAPIv1.0)\u003c/em\u003e\u003c/b\u003e and the \u003cb\u003e\u003cem\u003eBenefits Claims Application Programming Interface (BCAPI)\u003c/em\u003e\u003c/b\u003e established preprocessed claims on receipt in VBMS by automatically translating contentions to classification codes. We make a distinction between claims established via the CAAPIv1.0 and BCAPI and those established through ‘manual’ techniques not discussed here in order to measure the performance of the AI solution in a production environment.\u003c/p\u003e\n\u003cp\u003eOnce claims are established, either through the manual or AI approach, they are routed to the appropriate Veteran Service Representative (VSR) to begin the evidence building or adjudication process. In the course of our discovery efforts, it became clear that the entry of data into VBMS is not a one-time event because the VSRs update and delete contentions, flashes and special issues multiple times before a claim is fully developed.\u003c/p\u003e\n\u003cp\u003eEven though we do not currently measure the revolving activities around VBMS, we believe it is central to claims’ long processing time. To measure these activities, we introduced two new measures for contentions, flashes and special issues. These measures capture the accuracy and agility of data flow into VBMS.\u003c/p\u003e\n\u003cp\u003eThe first measure known as the efficiency of a process is defined as:\u003c/p\u003e\n\u003cp align=\"right\"\u003e \n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\begin{align*}\\eta_i=\\frac{number\\\u0026space;of\\\u0026space;i\\\u0026space;entered\u0026space;-\u0026space;number\\\u0026space;of\\\u0026space;i\\\u0026space;removed}{number\\\u0026space;of\\\u0026space;i\\\u0026space;entered},\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(1)}\\end{align*}\" title=\"\\begin{align*}\\eta_i=\\frac{number\\ of i\\ entered - number of\\ i\\ removed}{number\\ of\\ i\\ entered}, \\qquad\\qquad\\qquad\\qquad {(1)}\\end{align*}\" /\u003e\n \u003c/p\u003e\n\u003cp\u003ewhere \u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;i\u0026space;\" title=\"\\inline i \" /\u003e is contentions, flashes or special issues and measures the accuracy of each process. The second measure \u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;t_{i}\" title=\"\\inline t_{i}\" /\u003e captures the agility of the data entry process. Again, \u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;i\u0026space;\" title=\"\\inline i \" /\u003e assumes value of contentions, flashes and special issues depending on the process being described. \u003cimg src=\"https://latex.codecogs.com/svg.image?t_{i}\" title=\"t_{i}\" /\u003e measures the time from initial establishment until the last element of a process is entered or removed from VBMS.\u003c/p\u003e\n\u003cp\u003eThe situation described above also leads to multiple VSRs in different locations working on a claim before it is concluded. To capture this, we introduced two other measures, the total number of unique VSRs or adjudicators that work on a claim  \u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;n_{adjudicators}\" title=\"\\inline n_{adjudicators}\" /\u003e and the total number of unique locations where a claim was processed - \u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;n_{locations}\" title=\"\\inline n_{locations}\" /\u003e. Finally, we introduce the \u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;n_{contentions}\" title=\"\\inline n_{contentions}\" /\u003e which measures the number of unique claimant contentions at submission.  Submissions with a lot of contentions would likely be more complicated than the ones with fewer contentions. Utilizing the huge dataset available to us, we proceeded to determine these metrics for claims established using manual methods and the AI based CAAPIv1.0/BCAPI.\u003c/p\u003e\n\u003ch2 id=\"disability-benefits-data\"\u003eDisability Benefits Data\u003c/h2\u003e\n\u003cp\u003eThe data used in our analysis was provided by the Office of Performance Analysis and Integrity. Specifically, they provided journal data detailing all activities on a claim. This allowed us to track the dynamics of all the claims we considered. We worked with 103 million unique rows and 38 fields, spanning January 2020 to January 2021. We focused only on claim submissions received, processed, and closed within the time period specified to ensure the comparison between claims that are AI established and those manually established is unbiased. We present some of our high-level findings in the next section.\u003c/p\u003e\n\u003ch2 id=\"disability-process-metrics\"\u003eDisability Process Metrics\u003c/h2\u003e\n\u003cp\u003eThe results of our analysis are presented here using a combination of the mean and standard deviation for each measure for both claim types under consideration: claims established through CAAPIv1.0/BCAPI protocol and those established using manual techniques.\u003c/p\u003e\n\u003ch3 id=\"claim-establishment-time\"\u003eClaim Establishment Time\u003c/h3\u003e\n\u003cp\u003eAs mentioned earlier, the claims establishment time \u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;t_{establishment}\u0026space;\" title=\"\\inline t_{establishment} \" /\u003e\nmeasures the time between when a claim is received and entered into VBMS. It is a proxy for the effectiveness of the claim’s intake process. The establishment times measured are:\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\begin{align*}\u0026space;\\overline{t}_{establishment,ai}\\\u0026space;=\u0026space;3.59\u0026space;\\pm\u0026space;13.40\\\u0026space;days\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(2)}\u0026space;\\end{align*}\" title=\"\\begin{align*} \\overline{t}_{establishment,ai}\\quad\\ = 3.59 \\pm 13.40\\ days\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(2)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\begin{align*}\u0026space;\\overline{t}_{establishment,manual}\u0026space;=\u0026space;3.77\u0026space;\\pm\u0026space;12.22\\\u0026space;days\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(3)}\u0026space;\\end{align*}\" title=\"\\begin{align*} \\overline{t}_{establishment,manual} = 3.77 \\pm 12.22\\ days\\qquad\\qquad\\qquad{(3)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003cp align=\"justify\"\u003e\nEquations 2 and 3 are the establishment times for the CAAPIv1.0/BCAPI establishment protocol and manual establishment methods, respectively. The two numbers are statistically but not operationally different with our AI protocol establishing claims five hours sooner than the manual establishment technique in use at VA.\n\u003c/p\u003e\n\u003ch3 id=\"claim-establishment-accuracy\"\u003eClaim Establishment Accuracy\u003c/h3\u003e\n\u003cp\u003eNext, we present the results of the contentions, flashes and special issues efficiencies as defined by Equation 1. The measures penalize the removal of previously entered data and have a range of 0 to 1. The results of the efficiency process are presented below:\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\begin{align*}\\quad\\overline{\\eta}_{contentions,ai}\u0026space;=\u0026space;0.83\u0026space;\\pm\u0026space;0.31\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(4)}\u0026space;\\end{align*}\" title=\"\\begin{align*}\\quad\\overline{\\eta}_{contentions,ai}\\quad\\ = \\quad\\0.84 \\pm 0.31\\quad\\quad {(4)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\begin{align*}\\overline{\\eta}_{contentions,manual}\u0026space;=\u0026space;0.81\u0026space;\\pm\u0026space;0.41\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(5)}\u0026space;\\end{align*}\" title=\"\\begin{align*}\\overline{\\eta}_{contentions,manual} = 0.84 \\pm 0.41 \\quad {(5)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003cp\u003eThe contentions efficiencies of both the AI process (Equation 4) and manual method (Equation 5) are similar, with the manual method efficiency slightly being more variable. The efficiencies of the flash process are also similar for the AI and manual establishment methods as shown below:\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\begin{align*}\u0026space;\\overline{\\eta}_{flashes,ai}\u0026space;=\u0026space;0.81\u0026space;\\pm\u0026space;0.20\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(6)}\u0026space;\\end{align*}\" title=\"\\begin{align*} \\overline{\\eta}_{flashes,ai}\\quad\\ = 0.84 \\pm 0.20\\qquad{(6)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\begin{align*}\u0026space;\\overline{\\eta}_{flashes,manual}\u0026space;=\u0026space;0.80\u0026space;\\pm\u0026space;0.20\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(7)}\u0026space;\\end{align*}\" title=\"\\begin{align*} \\overline{\\eta}_{flashes,manual} = 0.84 \\pm 0.20\\qquad{(7)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003cp\u003eThe special issues efficiency of the AI established claims is noticeably higher than that of manually established claims as shown below:\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\begin{align*}\u0026space;\\overline{\\eta}_{special\u0026space;\\\u0026space;issues,ai}\u0026space;=\u0026space;0.42\u0026space;\\pm\u0026space;0.20\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(8)}\u0026space;\\end{align*}\" title=\"\\begin{align*} \\overline{\\eta}_{special \\ issues,ai}\\quad\\ = 0.42 \\pm 0.20\\qquad\\ {(8)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\begin{align*}\u0026space;\\overline{\\eta}_{special\u0026space;\\\u0026space;issues,manual}\u0026space;=\u0026space;0.33\u0026space;\\pm\u0026space;0.23\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(9)}\u0026space;\\end{align*}\u0026space;\" title=\"\\begin{align*} \\overline{\\eta}_{special \\ issues,manual} = 0.33 \\pm 0.23 \\qquad {(9)} \\end{align*} \" /\u003e\n\u003c/p\u003e\n\u003ch3 id=\"claim-establishment-agility\"\u003eClaim Establishment Agility\u003c/h3\u003e\n\u003cp\u003eWhile the process efficiencies of the AI established claims are slightly better than those established through manual techniques, their processing times are noticeably slower as shown below:\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;\\begin{align*}\u0026space;\\overline{t}_{contentions,ai}\u0026space;=\u0026space;62\u0026space;\\pm\u0026space;68\\\u0026space;days\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(10)}\u0026space;\\end{align*}\" title=\"\\inline \\begin{align*} \\overline{t}_{contentions,ai}\\quad\\ = 62 \\pm 68\\ days \\qquad{(10)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;\\begin{align*}\\overline{t}_{contentions,manual}\u0026space;=\u0026space;55\u0026space;\\pm\u0026space;68\\\u0026space;days\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(11)}\\end{align*}\" title=\"\\inline \\begin{align*}\\overline{t}_{contentions,manual} = 55 \\pm 68\\ days\\qquad{(11)}\\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003chr\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;\\begin{align*}\\overline{t}_{flashes,ai}\u0026space;=\u0026space;82\u0026space;\\pm\u0026space;70\\\u0026space;days\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(12)}\u0026space;\\end{align*}\" title=\"\\inline \\begin{align*}\\overline{t}_{flashes,ai}\\quad\\ = 82 \\pm 70\\ days\\qquad\\qquad {(12)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;\\begin{align*}\\overline{t}_{flashes,manual}\u0026space;=\u0026space;77\u0026space;\\pm\u0026space;69\\\u0026space;days\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(13)}\\end{align*}\" title=\"\\inline \\begin{align*}\\overline{t}_{flashes,manual} = 77 \\pm 69\\ days\\qquad\\qquad {(13)}\\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003chr\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;\\begin{align*}\\overline{t}_{special\u0026space;\\\u0026space;issues,ai}\u0026space;=\u0026space;59\u0026space;\\pm\u0026space;67\\\u0026space;days\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(14)}\u0026space;\\end{align*}\" title=\"\\inline \\begin{align*}\\overline{t}_{special \\ issues,ai}\\quad\\ = 59 \\pm 67\\ days\\qquad\\qquad{(14)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;\\begin{align*}\\overline{t}_{special\u0026space;\\\u0026space;issues,manual}\u0026space;=\u0026space;53\u0026space;\\pm\u0026space;66\\\u0026space;days\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(15)}\u0026space;\\end{align*}\" title=\"\\inline \\begin{align*}\\overline{t}_{special \\ issues,manual} = 53 \\pm 66\\ days\\qquad\\qquad {(15)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003cp\u003eIt took 5 – 7 days longer for AI established claims to be completed in comparison to claims established through the manual method.\u003c/p\u003e\n\u003ch3 id=\"claim-establishment--other-measures\"\u003eClaim Establishment – Other Measures\u003c/h3\u003e\n\u003cp\u003eOur analysis revealed that an average of 3 VSRs work on every claim regardless of the establishment method as shown below:\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;\\begin{align*}\\overline{n}_{adjudicators,ai}\u0026space;=\u0026space;3\u0026space;\\pm\u0026space;1\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(16)}\u0026space;\\end{align*}\" title=\"\\inline \\begin{align*}\\overline{n}_{adjudicators,ai}\\quad\\ = 3 \\pm 1\\ \\qquad\\qquad{(16)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;\\begin{align*}\\overline{n}_{adjudicators,manual}\u0026space;=\u0026space;3\u0026space;\\pm\u0026space;2\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(17)}\u0026space;\\end{align*}\" title=\"\\inline \\begin{align*}\\overline{n}_{adjudicators,manual} = 3 \\pm 2\\ \\qquad\\qquad {(17)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003cp\u003eAI established claims were also processed at an average of 3 locations as opposed to 2 locations for claims established through the manual method:\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;\\begin{align*}\\overline{n}_{locations,ai}\u0026space;=\u0026space;3\u0026space;\\pm\u0026space;1\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(18)}\u0026space;\\end{align*}\" title=\"\\inline \\begin{align*}\\overline{n}_{locations,ai}\\quad\\ = 3 \\pm 1\\ \\qquad\\qquad{(18)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;\\begin{align*}\\overline{n}_{locations,manual}\u0026space;=\u0026space;2\u0026space;\\pm\u0026space;1\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(19)}\u0026space;\\end{align*}\" title=\"\\inline \\begin{align*}\\overline{n}_{locations,manual} = 2 \\pm 1\\ \\qquad\\qquad {(19)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003cp align=\"justify\"\u003e\nFinally, AI established claims contain fewer claimant contentions at submission:\n\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;\\begin{align*}\\overline{n}_{contentions,ai}\\\u0026space;=\u0026space;9\u0026space;\\pm\u0026space;8\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(20)}\u0026space;\\end{align*}\" title=\"\\inline \\begin{align*}\\overline{n}_{contentions,ai}\\ = 9 \\pm 8\\ \\qquad\\qquad {(20)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;\\begin{align*}\\overline{n}_{contentions,manual}\u0026space;=\u0026space;11\u0026space;\\pm\u0026space;13\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(21)}\u0026space;\\end{align*}\" title=\"\\inline \\begin{align*}\\overline{n}_{contentions,manual} = 11 \\pm 13\\ \\qquad\\qquad {(21)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003cp\u003eWe also estimated the time it took to close a claim once a ratings decision has been reached, \u003cimg src=\"https://latex.codecogs.com/svg.image?t_{ratings}\" title=\"t_{ratings}\" /\u003e. No noticeable difference was observed between AI established claims and claims established through the manual route as shown below:\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;\\begin{align*}\\overline{t}_{ratings,ai}\u0026space;=\u0026space;0.57\u0026space;\\pm\u0026space;3.02\\\u0026space;days\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(22)}\u0026space;\\end{align*}\" title=\"\\inline \\begin{align*}\\overline{t}_{ratings,ai}\\quad\\ = 0.57 \\pm 3.02\\ \\qquad\\qquad {(22)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003cp align=\"right\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?\\inline\u0026space;\\begin{align*}\\overline{t}_{ratings,manual}\u0026space;=\u0026space;0.57\u0026space;\\pm\u0026space;3.30\\\u0026space;days\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;\\qquad\u0026space;{(23)}\u0026space;\\end{align*}\" title=\"\\inline \\begin{align*}\\overline{t}_{ratings,manual} = 0.57 \\pm 3.30\\ \\qquad\\qquad{(23)} \\end{align*}\" /\u003e\n\u003c/p\u003e\n\u003ch2 id=\"a-claims-phenomenological-model\"\u003eA Claims Phenomenological Model\u003c/h2\u003e\n\u003cp\u003eHaving described these performance measures, phenomenological models relating them to processing times, \u003cimg src=\"https://latex.codecogs.com/svg.image?t_{processing}\" title=\"t_{processing}\" /\u003e, for both AI and manually established claims were developed. The processing time, \u003cimg src=\"https://latex.codecogs.com/svg.image?t_{processing}\" title=\"t_{processing}\" /\u003e, is defined strictly as the difference between when a claim was received at VA and closed after a disability rating decision was reached. The average processing time of AI established claims, \u003cimg src=\"https://latex.codecogs.com/svg.image?\\overline{t}_{processing,ai}\" title=\"\\overline{t}_{processing,ai}\" /\u003e was 88.2 days with a spread of 72.4 days while that of manually established claims, \u003cimg src=\"https://latex.codecogs.com/svg.image?\\overline{t}_{processing,manual}\" title=\"\\overline{t}_{processing,manual}\" /\u003e was 83 days with a spread of 72.4 days. The model presented here for the processing time of manually established claims, \u003cimg src=\"https://latex.codecogs.com/svg.image?{t}_{processing,manual}\" title=\"\\overline{t}_{processing,manual}\" /\u003e, did not take into account other details on the claims form including medical service records, demography information, and additional evidence relevant to determining validity or otherwise of a claim. Our model was able to describe over ninety two percent of variability in the processing time which is more than sufficient for its intended purpose of determining the contribution of each measure to the dependent variable.\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://latex.codecogs.com/svg.image?t_{processing,manual}=\u0026space;0.42*t_{flashes,manual}\u0026plus;0.24*t_{contentions,manual}\u0026plus;0.51*t_{special\u0026space;\\\u0026space;issues,manual}\u0026plus;0.70*t_{establisment,manual}\u0026plus;0.09*t_{ratings,manual}\u0026plus;5.85*n_{locations,manual}-1.22*n^{2}_{locations,manual}\u0026plus;\u0026space;2.10*n_{adjudicators,manual}\u0026plus;0.24*n_{contentions,manual}-\u0026space;24.0*\\eta_{special\\\u0026space;issues,manual}\u0026plus;44.0*\\eta^{2}_{special\u0026space;\\\u0026space;issues,manual}\u0026plus;20.2*\\eta^{2}_{flashes,manual}-5.62*\\eta_{contentions,manual}\" title=\"t_{processing \\ manual}= 0.42*t_{flashes}+0.24*t_{contentions}+0.51*t_{special \\ issues}+0.70*t_{establisment}+0.09*t_{ratings}+5.85*n_{locations}-1.22*n^{2}_{locations}+ 2.10*n_{adjudicators}+0.24*n_{contentions}- 24.0*\\eta_{special\\ issues}+44.0*\\eta^{2}_{special \\ issues}+20.2*\\eta^{2}_{flashes}-5.62*\\eta_{contentions}\" /\u003e\n\u003c/p\u003e\n\u003ch3 id=\"actionable-insights-from-the-claims-phenomenological-model\"\u003eActionable Insights from the Claims Phenomenological Model\u003c/h3\u003e\n\u003cp\u003eHere we discuss the deductions made using the average measures for the manual process and the phenomenological model.\u003c/p\u003e\n\u003ch4 id=\"_intake-process_\"\u003e\u003cem\u003eIntake Process\u003c/em\u003e\u003c/h4\u003e\n\u003cp\u003eDecreasing the number of days until a claim is established  \u003cimg src=\"https://latex.codecogs.com/svg.image?t_{establishment,manual}\" title=\"t_{establishment,manual}\" /\u003e, by 1 day and keeping all the other variables constant decreases the overall processing time by 0.70 days. It currently takes about 3.8 days to establish a claim according to Equation 3 which implies the total amount of time savings we can get from instantaneously establishing claims is 2.6 days.\u003c/p\u003e\n\u003ch4 id=\"_contentions_\"\u003e\u003cem\u003eContentions\u003c/em\u003e\u003c/h4\u003e\n\u003cp\u003eDecreasing the number of days until the last contention is added, \u003cimg src=\"https://latex.codecogs.com/svg.image?t_{contentions,manual}\" title=\"t_{contentions,manual}\" /\u003e, by 1 day and keeping all else constant, decreases the overall processing time by 0.24 day. Equation 11 suggests we can save 13 days if all the contentions are added on receipt since it currently takes an average of 55 days until the last contention is added.\u003c/p\u003e\n\u003cp\u003eIncreasing the efficiency of the contentions process, \u003cimg src=\"https://latex.codecogs.com/svg.image?\\eta_{contentions,manual}\" title=\"\\eta_{contentions,manual}\" /\u003e, by 1% decreases the overall processing by 0.05 day. Equation 5 shows that we can get an uplift of 0.8 days by improving the efficiency of the contentions process to unity.\u003c/p\u003e\n\u003ch4 id=\"_flash-process_\"\u003e\u003cem\u003eFlash Process\u003c/em\u003e\u003c/h4\u003e\n\u003cp\u003eDecreasing the current number of days until the last flash is added \u003cimg src=\"https://latex.codecogs.com/svg.image?t_{flashes,manual}\" title=\"t_{flashes,manual}\" /\u003e, by 1 day and keeping other variables constant decreases the overall processing time by 0.42 day. It currently takes an average of 77 days until the last flash is added according to Equation 13 implying the most savings in days, we can get by adding flashes on day zero is 32 days.\u003c/p\u003e\n\u003cp\u003eIncreasing the current efficiency of the flash process, \u003cimg src=\"https://latex.codecogs.com/svg.image?\\eta_{flashes,manual}\" title=\"\\eta_{flashes,manual}\" /\u003e, by 1% and keeping all else constant increases the overall processing time by 0.002 day. Equation 8 shows the most improvement we can get on the flash efficiency is 20% indicating we can potentially save 0 extra days by improving the accuracy of the flash entry process.\u003c/p\u003e\n\u003ch4 id=\"_special-issues_\"\u003e\u003cem\u003eSpecial Issues\u003c/em\u003e\u003c/h4\u003e\n\u003cp\u003eDecreasing the current number of days until the last special issue is added, \u003cimg src=\"https://latex.codecogs.com/svg.image?t_{special\\\u0026space;issues,manual}\" title=\"t_{special\\ issues,manual}\" /\u003e, by 1 day and keeping other variables constant decreases the overall processing time by 0.51 day. It currently takes an average of 53 days until the last special issue is added according to Equation 9, leading to a potential savings of 27 days if all special issues can be added at establishment.\u003c/p\u003e\n\u003cp\u003eIncreasing the efficiency of special issues, \u003cimg src=\"https://latex.codecogs.com/svg.image?\\eta_{special\\\u0026space;issues,manual}\" title=\"\\eta_{special\\ issues,manual}\" /\u003e, by 1% reduces the overall processing time by 0.24 day. Using Equation 15 as our basis, there is a potential of saving 16.1 days if  \u003cimg src=\"https://latex.codecogs.com/svg.image?\\eta_{special\\\u0026space;issues,manual}\" title=\"\\eta_{special\\ issues,manual}\" /\u003e can be increased by 67.3%.\u003c/p\u003e\n\u003ch4 id=\"_time-to-close-claim-after-ratings-decision_\"\u003e\u003cem\u003eTime to Close Claim after Ratings Decision\u003c/em\u003e\u003c/h4\u003e\n\u003cp\u003eDecreasing the number of days from ratings to closure \u003cimg src=\"https://latex.codecogs.com/svg.image?t_{ratings,manual}\" title=\"t_{ratings,manual}\" /\u003e, by 1 day reduces the overall processing time by 0.09 day while keeping all other variables constant. The total uplift possible according to Equation 23 is 0.05 days.\u003c/p\u003e\n\u003ch4 id=\"_number-of-locations_\"\u003e\u003cem\u003eNumber of Locations\u003c/em\u003e\u003c/h4\u003e\n\u003cp\u003eDecreasing the number of processing locations, \u003cimg src=\"https://latex.codecogs.com/svg.image?n_{locations,manual}\" title=\"n_{locations,manual}\" /\u003e, by 1 reduces the overall processing time by 4.63 days.\u003c/p\u003e\n\u003ch4 id=\"_number-of-adjudicators_\"\u003e\u003cem\u003eNumber of Adjudicators\u003c/em\u003e\u003c/h4\u003e\n\u003cp\u003eDecreasing the number of adjusters, \u003cimg src=\"https://latex.codecogs.com/svg.image?n_{adjudicators,manual}\" title=\"n_{adjudicators,manual}\" /\u003e, by 1 reduces the overall processing time by 2.10 days.\u003c/p\u003e\n\u003ch3 id=\"insights---summary\"\u003eInsights - Summary\u003c/h3\u003e\n\u003cp\u003eTable 1.0 below lists the insights gleaned from the analysis in order of decreasing potential savings in days. The numbers show that we can potentially save about 100 days if all claims are established on receipt, and the process swirl is eliminated.  Perfecting the intake process saves a maximum of 3 days which is not the most effective in decreasing total processing time. Shortening the time of data entry into VBMS and improving the accuracy of the special issues process are the most impactful in decreasing the processing time. These actions could potentially save 88 days in the overall lifecycle or processing time of a claim. Streamlining the processing locations could potentially save an additional 5 days per location removed.\u003c/p\u003e\n\u003cp\u003eThe results show that improving the current efficiencies of the contentions and flash processes would yield marginal-to-no improvements because the two efficiencies are currently around eighty percent. The time required to gather evidence, implicitly captured here as, \u003cimg src=\"https://latex.codecogs.com/svg.image?t_{flashes}\" title=\"t_{flashes}\" /\u003e, is a measure of the time it takes for Veterans to meet the administrative burden of compliance placed on them by VA. The naïve way to meet this burden at the time of claim establishment is to not ask for any evidence. In reality, this would be irresponsible and potentially illegal. We explore possible solutions to the problem in the next section.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eTable 1.0: Potential Savings in days.\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003ctable\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth style=\"text-align:left\"\u003e\u003cem\u003eProcess Action\u003c/em\u003e\u003c/th\u003e\n\u003cth style=\"text-align:center\"\u003e\u003cem\u003ePotential Decrease in Processing Time(days)\u003c/em\u003e\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003e\u003cimg src=\"https://latex.codecogs.com/svg.image?Decrease\u0026space;\\\u0026space;t_{flashes,manual}\" title=\"Decrease \\ t_{flashes,manual}\" /\u003e\u003c/td\u003e\n\u003ctd style=\"text-align:center\"\u003e\u003cem\u003e32\u003c/em\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003e\u003cimg src=\"https://latex.codecogs.com/svg.image?Decrease\u0026space;\\\u0026space;t_{special\\\u0026space;issues,manual}\" title=\"Decrease\\ t_{special\\ issues,manual}\" /\u003e\u003c/td\u003e\n\u003ctd style=\"text-align:center\"\u003e\u003cem\u003e27\u003c/em\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003e\u003cimg src=\"https://latex.codecogs.com/svg.image?Increase\\\u0026space;\\eta_{special\\\u0026space;issues,manual}\" title=\"Increase\\ \\eta_{special\\ issues,manual}\" /\u003e\u003c/td\u003e\n\u003ctd style=\"text-align:center\"\u003e\u003cem\u003e16\u003c/em\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003e\u003cimg src=\"https://latex.codecogs.com/svg.image?Decrease\u0026space;\\\u0026space;t_{contentions,manual}\u0026space;\" title=\"Decrease \\ t_{contention,manual} \" /\u003e\u003c/td\u003e\n\u003ctd style=\"text-align:center\"\u003e\u003cem\u003e13\u003c/em\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003e\u003cimg src=\"https://latex.codecogs.com/svg.image?Decrease\u0026space;\\\u0026space;n_{locations,manual}\u0026space;\" title=\"Decrease \\ n_{locations,manual} \" /\u003e\u003c/td\u003e\n\u003ctd style=\"text-align:center\"\u003e\u003cem\u003e5\u003c/em\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003e\u003cimg src=\"https://latex.codecogs.com/svg.image?Decrease\u0026space;\\\u0026space;t_{establishment,manual}\" title=\"t_{establishment,manual}\" /\u003e\u003c/td\u003e\n\u003ctd style=\"text-align:center\"\u003e\u003cem\u003e3\u003c/em\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003e\u003cimg src=\"https://latex.codecogs.com/svg.image?Decrease\u0026space;\\\u0026space;n_{adjudicators,manual}\u0026space;\" title=\"Decrease \\ n_{adjudicators,manual} \" /\u003e\u003c/td\u003e\n\u003ctd style=\"text-align:center\"\u003e\u003cem\u003e2\u003c/em\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003e\u003cimg src=\"https://latex.codecogs.com/svg.image?Increase\u0026space;\\\u0026space;\\eta_{contentions,manual}\" title=\"Increase \\ \\eta_{contentions,manual}\" /\u003e\u003c/td\u003e\n\u003ctd style=\"text-align:center\"\u003e\u003cem\u003e1\u003c/em\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003e\u003cimg src=\"https://latex.codecogs.com/svg.image?Increase\\\u0026space;\\eta_{flashes,manual}\" title=\"Increase\\ \\eta_{flashes,manual}\" /\u003e\u003c/td\u003e\n\u003ctd style=\"text-align:center\"\u003e\u003cem\u003e0\u003c/em\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\u003ch2 id=\"conclusions-and-recommendations\"\u003eConclusions and Recommendations\u003c/h2\u003e\n\u003cp\u003eOver 50 percent of the disability claims processed at VA are \u003cem\u003esubsequent\u003c/em\u003e claims – EP 020, based on 2020 data. This means that prior EP details already exist in VBMS, suggesting these claims could be processed on receipt without asking Veterans for information we already have.  VA has limited ability to make Veterans submit documents on time, thus making the option of extracting data from VBMS and/or other accessible databases attractive.\u003c/p\u003e\n\u003cp\u003eSpecial Issues efficiency, \u003cimg src=\"https://latex.codecogs.com/svg.image?\\eta_{special\\\u0026space;issues,manual}\" title=\"\\eta_{special\\ issues,manual}\" /\u003e, is very low at the moment and tighter definitions are needed for it.\u003c/p\u003e\n\u003cp\u003eMy analysis revealed that piecewise AI solutions could have \u003cb\u003e\u003cem\u003eunintended\u003c/em\u003e\u003c/b\u003e overall consequences. Although the CAAPIv1.0/BCAPI helps us to establish claims quicker than the manual establishment process, these claims end up having longer processing times, 5.2 days to be precise because of possible operational inefficiencies. The new iteration of the API, CAAPIv2.0/BCAPI will add initial flashes and special issues at the point of establishment and may speed up the processing times of claims established through it.\u003c/p\u003e\n\u003cp\u003eAnother intriguing and compelling solution is to make final ratings decisions directly from claimant’s initial submission thereby bypassing the current framework that relies on classification codes, flashes and special issues through the use of an \u003cb\u003e\u003cem\u003eEnd-To-End AI Solution\u003c/em\u003e\u003c/b\u003e. This approach will be a compromise solution between the naïve approach mentioned earlier and the manual method currently in use at VA. The data required to develop this solution has been provided by the Office of Performance Analysis and Integrity and is being worked on now. Once the solution is completed, we would be able to probe the abstraction to determine the minimum amount of administrative burden that VA must place on Veterans to meet its compliance requirements and perhaps change policy.\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;m grateful to work with VA CTO colleagues to center Veterans in all we do. AI is merely a tool for better service to our nation’s Veterans. When we remove administrative burdens, we’re not making any colleague’s work obsolete; we’re enabling VA to focus on what matters: efficient, safe, and meaningful healthcare.\u003c/p\u003e\n\u003chr\u003e\n\u003cp\u003e\u003cem\u003eAre you passionate about technology as a tool to improve lives? Interested in using your expertise for public good? That’s what Presidential Innovation Fellows (PIF) is all about. Learn more about PIF at \u003ca href=\"https://www.presidentialinnovationfellows.gov/\"\u003eour website\u003c/a\u003e, and follow us on \u003ca href=\"https://www.twitter.com/PIFgov\"\u003eTwitter\u003c/a\u003e and \u003ca href=\"https://www.linkedin.com/company/white-house-presidential-innovation-fellows/\"\u003eLinkedIn\u003c/a\u003e to find the latest work from our Fellows\u003c/em\u003e.\u003c/p\u003e\n"}
  ]
}