7 experts and industry representatives t identify an exhaustive pl f relevant perfrmance indicatrs that were either in the public dmain r were being prepared fr near-term disseminatin. All relevant measures were retrieved and the team reviewed the methdlgy used in their design t assess their quality. We abstracted each perfrmance indicatr, nting its data surce, the disrder t which it applied the strength f the evidence fr the prcess measured by the indicatr, and IOM dmain. 2. Identify recmmendatins with empirical supprt that are nt cvered by the existing measures, and create new perfrmance indicatrs t address these gaps. We reviewed VA and APA Clinical Practice Guidelines fr the 5 disrders included in the prgram evaluatin (the VA CPG fr psychses includes recmmendatins fr bth schizphrenia and biplar disrder), and listed all individual recmmendatin statements. Multi-part recmmendatins were separated int individual parts and duplicative recmmendatins were deleted. We defined key terms, examined the recmmendatins fr incnsistency r ambiguity, and prduced a list f explicit, unambiguus measures that had empirical supprt fr the prcess-utcme link. Where discrepancies existed between the APA and VA guidelines the team cnsulted utside experts and discussed the discrepancy until cnsensus was reached. 3. Select measures fr further technical specificatin. Because f the large number f candidate measures, we engaged in a systematic selectin prcess. First, we identified whether the data needed t ppulate the indicatrs existed in the necessary frm in either the administrative r in the medical recrd, and recmmendatins that culd nt be defined peratinally because f lack f data were eliminated. Next, the research team reviewed the measures fr meaningfulness and feasibility, and described the measures predictive validity thrugh an evaluatin f the strength f the prcess-utcme link. A subset f measures was selected t be reviewed by external clinical experts wh further pruned them n the basis f clinical significance. All measures were reviewed with a VA clinical advisry grup, whse members were selected fr their clinical expertise and familiarity with the subject matter. The advisry grup evaluated recmmendatins fr validity and feasibility, and usefulness fr VA s peratinal management and strategic leadership. Lastly, VA and VHA leadership rated the indicatrs n their imprtance t the VHA and cntributin t presenting a cmprehensive quality prfile. As a result f this prcess, we identified a cre set f measures that were valid, feasible, and a VA pririty. Mst f them described prcesses that were identified with acute treatment. 4. Generate a new set f measures pertaining t the psychscial aspects f care. Because the prcess used abve required measures t have an empirical basis f supprt, the dmains f patient-centeredness and recvery were neglected. Althugh nt evidence-based r guideline-supprted, bth dmains are endrsed by the Institute f Medicine and the VA Mental Health Strategic Plan as critical t quality. We therefre used a cllabrative prcess between the research team and the VA clinical advisry grup t identify key cnstructs pertaining t patient-centeredness and recvery. Amng the many pssible cnstructs, we chse t fcus n the psychscial aspects f care such as attentin t scial supprts, husing and emplyment. Indicatr develpment invlved recruiting experts and engaging them in the prcess f identifying a cre set f crss-cutting psychscial indicatrs. Because f the difficulty evaluating the predictive validity f the psychscial aspects f care, they will be used descriptively. 5

8 5. Develp technical specificatins fr finalized indicatrs and categrize their strength f evidence We generated detailed technical specificatins fr all finalized perfrmance indicatrs with respect t VHA administrative data and electrnic medical recrds, and identified data surces that efficiently prvided the infrmatin necessary t ppulate the indicatrs. Each indicatr cntained an indicatr statement and executive summary describing the surce(s) f the specificatins and clinical ratinale fr the selected indicatr. We als included the indicatr grade, which reflected the strength f the prcess-utcme link, and whether the indicatr wuld be used as a benchmark r descriptively. We created numeratrs and denminatrs fr each indicatr based n the data that wuld be available, and defined the ppulatin t which the indicatr applied. Fr example, if the indicatr applied nly t peple in a new treatment episde, we defined the term new treatment episde. All clinical and measurement terms were defined peratinally, and we summarized anticipated data cllectin prblems and ther feasibility issues. These included any prblems that we culd fresee prir t starting abstractin, such as data elements that might be time-cnsuming t cllect r which required a judgement t be made by the abstractr. Fr cmplex prcesses f care with multiple cmpnents f varying clinical r utcme relevance (e.g., delivery f CBT/SST r assessment f mental status), we sught expert input t select and peratinalize critical cmpnents. Technical specificatins were reviewed by bth external clinical experts and the VA Advisry grup in rder t ensure that specificatins were bth feasible given the data available, and meaningful t this particular ppulatin. We categrized indicatrs accrding t the strength f the prcess-utcme link using the grading system develped by the AHRQ s US Preventive Services Task Frce. vi Grade I measures are thse where the link between prcess and utcme has been established thrugh randmized clinical trials, grade II measures are supprted by well-designed, nnrandmized trials, and grade III measures are supprted by expert pinin. A caveat t drawing cnclusins frm this grading system is that smetimes the utcmes literature may nt be specific enugh abut the ingredients f the interventin that are critical t its efficacy/effectiveness. Fr example, althugh randmized cntrlled trials have established the value f psychtherapy in the treatment f several disrders, nt enugh evidence exists t ascertain the minimum dse (r number f sessins) and duratin required fr the utcme advantage t emerge. We als nte that the grading des nt reflect translatinal validity, r the certainty that the technical specificatins accurately reflect the prcess f care they are trying t capture. 6. Determine data abstractin elements and sequence f abstractin Starting with the technical specificatins develped abve, we described the data abstractin elements and abstractin sequence fr each indicatr. Since many indicatrs required verlapping infrmatin, we remved redundancy and gruped questins fr efficiency. Fr example, all questins abut medicatins were placed tgether,, since the medicatins prescribed t a veteran are fund in a single sectin f the recrd. We created abstractin frms fr each diagnsis. 7. Pilt test indicatrs fr translatinal validity and perfrmance Clinical nurse abstractrs pilted each indicatr fr timing and perfrmance using pencil and paper and mdificatins were made in rder t keep data cllectin time t a minimum. We 6

9 fund that sme data elements were nt fund in the part f the medical recrd t which we had access, and, after review with the clinical advisry grup, deleted these indicatrs. After the initial paper and pencil pilt test, an electrnic abstractin frm was created and a secnd pilt test was perfrmed t make sure that the questins flwed crrectly and that there were n prgramming errrs. Discussin In this reprt we present a cmprehensive set f indicatrs fr evaluating the perfrmance f mental health care systems with tw different data surces, administrative and medical recrds. One f the greatest difficulties in evaluating mental health care is btaining meaningful data t measure the key elements f the system. In rder t evaluate the structure f care, we develped indicatrs that used a cmbinatin f bth data surces available, while recgnizing that bth surces f data, either singly r in cmbinatin, have inherent strengths and weaknesses. The main strength f using administrative data is their availability and cmprehensive enumeratin f the study ppulatin. vii Mrever, the databases were relatively large, enabling the study team t analyze ppulatin subgrups and specific gegraphic areas separately, which was particularly useful, since mst prblems related t access and availability are nt unifrm acrss ppulatins r within areas. In many cases, hwever, items were missing r the accuracy f the infrmatin prvided culd nt be guaranteed. This is nt uncmmn when data are cllected and used fr different purpses. Other studies als supprt the use f administrative data cmbined with chart review t assess perfrmance. viii While the structure-prcess-utcmes evaluatin mdel presented herein hlds prmise fr advancing the science f mental health care quality imprvement bth within and utside the VHA, a few final caveats are in rder. First, in any health care system, the prgressin frm evidence-based practice guidelines t perfrmance indicatrs t imprved patient utcmes is fraught with cmplexity. Great care must be taken t measure precisely what is intended t be measured thrugh effective and efficient dcumentatin s that the burden f measurement des nt utpace quality care prvisin. In additin, cntinued awareness f the cmplicated linkages between evidence-based practice and individual patient preferences and utcmes is essential. As recent studies amply demnstrate, even the mst basic f evidence-based practice imprvements can result in different utcmes fr different patients and fr different reasns. Attentin must als be paid t ensuring that quality imprvement becmes a part f the fabric f care at bth the rganizatinal and individual levels, thrugh resurce investment, staff training, etc. Secnd, nt all mental health care systems lk r perate like the VHA mental health care system. Public and private sectr mental health care functins largely as a cttage industry, with the majrity f psychiatrists practicing in sl r tw-physician practices; infrmatin technlgy is less well develped; there are few centralized administrative databases; and there is n single entity r rganizatin respnsible fr implementing and mnitring quality imprvement strategies. While these differences must be recgnized and addressed in the cntext f nging quality imprvement, the same high quality standards shuld nevertheless apply. Third, t what extent this mdel can be adapted fr use in ther systems and in ther cntexts is nt clear. It is pssible that certain cmpnents f the mdel will be mre suitable fr mental health quality imprvement effrts at the natinal r state levels r in large systems (e.g., managed care netwrks), while thers will wrk well in mre lcalized cntexts (e.g., cmmunity mental health centers). 7

10 VA has undertaken the mst extensive, systematic, and rigrus evaluatin f the mental health care delivery ever cnducted. Althugh this quality imprvement effrt is still in its early stages, and much remains t be learned, the framewrk, methdlgy, and preliminary results ffer a fertile grund upn which ther stakehlders in the mental health field can cntinue t build and expand bth in the near- and lnger-term. 8

20 CN709, Antipsychtics, Other CN750, Lithium Salts Inpatient admissin 3 where any psychiatric diagnsis is the primary diagnsis (ICD- 9 cdes: 290.xx-319.xx) and, if the primary diagnsis is nt a diagnsis in Table 1B, at least ne secndary diagnsis cmes frm Table 1B. If it is impssible t determine which diagnsis fr an utpatient encunter is the primary diagnsis, then a diagnsis in Table 1B must be listed as ne f the diagnses fr the encunter. 4 Instructins: The start f the new treatment episde fr biplar disrder will be defined by: 1) The admissin date r transfer date fr any inpatient hspitalizatin as defined abve. OR 2) An utpatient encunter where biplar disrder (Table 1B) is the primary diagnsis fllwing a clean perid f five r mre mnths (based n a 90-day prescriptin) fr which there is: NO prescriptin filled fr selected medicatins, AND (in the same time perid f five r mre mnths) NO utpatient encunter in any clinic where biplar disrder is the primary r secndary diagnsis. The first visit after the clean perid in which biplar disrder is the primary diagnsis will indicate the start date fr the new treatment episde. IIb. Schizphrenia The new treatment episde fr schizphrenia is defined as: OR A recent, diagnsis-related admissin r transfer t an inpatient/residential mental health bed, An utpatient encunter where schizphrenia (Table 1B) is the primary diagnsis fllwing a break in care. 3 Defining the NTE based n inpatient discharges was mdified such that the primary diagnsis must be any psychiatric diagnsis (210.xx-319.xx) and, if the primary diagnsis was nt ne f thse in Table 1B, an added requirement is that a diagnsis frm Table 1B must be listed as a secndary diagnsis. 4 Definitin fr hw an utpatient encunter triggers a NTE was mdified t be made cnsistent with the practicalities f the data being extracted frm medical recrds by WVMI. It is nt always pssible t determine which f the diagnses listed fr an utpatient encunter is the primary diagnsis based n the clinical ntes. In these cases, a diagnsis frm Table 1B must be listed as ne f the diagnses fr the encunter. 18

The University f Kent Crprate Standards fr data quality and the cllatin f data fr external presentatin This paper intrduces a set f standards with the aim f safeguarding the University s psitin in published

Enrllee Health Assessment Prgram Implementatin Guide and Best Practices March 2015 033129 (03-2015) This guide will help yu answer these questins: What is the Enrllee Health Assessment (EHA) prgram and

Appendix A Page 1 f 5 The items in the list f database technical requirements belw was develped thrugh several meetings between First 5 LA Research and Evaluatin, Infrmatin Technlgy, and Prgram Develpment

Oakland Unified Schl District Impact Assessment Perfrmance Management in Actin The perfrmance management system that has been built in this district prvides the systems that supprt ur cmmitment t scial

1.0 Purpse The purpse f this plicy and prcedure utline the methdlgy t develp training and assessment strategies, implementing a cmprehensive training and assessment strategy, and t ensure that all training

THE CITY UNIVERSITY OF NEW YORK IDENTITY THEFT PREVENTION PROGRAM 1. Prgram Adptin The City University f New Yrk (the "University") develped this Identity Theft Preventin Prgram (the "Prgram") pursuant

Ending Addictin Changes Everything RECOMMENDATIONS FOR STATES RECOMMENDED STATE ACTIONS TO IMPROVE HEALTH OUTCOMES AND REDUCE COSTS OF RISKY SUBSTANCE USE AND ADDICTION TO GOVERNMENT OCTOBER 2013 There

Client TERM COUNSELING DEFINITIONS DEFINITION The client is the business, if it exists. In the case f a prspective business, the client is the individual. In-Business: Cmpleted required registratin(s),

SECTION 5: EVALUATION METHODOLOGY The State f Oklahma will cnduct a cmprehensive, fair, and impartial evaluatin f bids received in respnse t this ITB. Technical and Cst Bids will be evaluated and scred

POLISH STANDARDS ON HEALTH AND SAFETY AS A TOOL FOR IMPLEMENTING REQUIREMENTS OF THE EUROPEAN DIRECTIVES INTO THE PRACTICE OF ENTERPRISES M. PĘCIŁŁO Central Institute fr Labur Prtectin ul. Czerniakwska

INTRODUCTION Dimensins f psychactive substance use and dependence The use f alchl and ther psychactive substances alters bth the functin and, ultimately, the structure f the brain by the altered stimulatin

Privacy Breach and Cmplaint Prtcl Effective: December 31, 2012 Apprved by: Le McKenna, CFO 1.0 General Privacy breaches and privacy cmplaints will be handled in accrdance with this prtcl. This prtcl is

Department f Health and Human Services OFFICE OF INSPECTOR GENERAL PENETRATION TEST OF THE INDIAN HEALTH SERVICE S COMPUTER NETWORK Inquiries abut this reprt may be addressed t the Office f Public Affairs