Research notebook: substance use quality measures

Instead of dribbling out many posts on specific papers in this area, I’ve put a bunch together in one, monster post. It’s super long. But it’s just a “research notebook” type post meant mostly for my own, future use. If it interests you, so much the better. If not, aren’t you glad it’s just one post?

Improving Health Care for Veterans by Improving Health Care Quality Measures, by Alex Harris (VA HSR&D Forum, 2012)

• “Implementing quality measures without sufficient validation may promote poor or incomplete care, divert effort and attention from more important activities, and create skepticism and ill will toward the entire quality management enterprise. Unfortunately, quality measures are often formulated and implemented without careful empirical validation or adequate appreciation for possible unintended consequences.”
• Beware the ecological fallacy in considering facility- or area-level relationships between process quality and outcomes.
• “The ability to use available administrative data to accurately identify patients with particular characteristics and the occurrence of specific health care events is central to the validity of many quality measures. Specification validity refers to the sensitivity and specificity of the coding strategies used to identify and define the relevant patients and processes.”
• “Quality measures vary in their vulnerability to improving measured performance without improving actual performance. […] Facilities can improve measured performance by increasing the numerator, which is the intention, or by restricting the number of patients who qualify for the denominator. The validity of many quality measures relies on the invulnerability of the denominator to manipulation.” See also “Gaming the denominator.”

Longer Length of Stay Is Not Associated with Better Outcomes in VHA’s Substance Abuse Residential Rehabilitation Treatment Programs, by Alex Harris, Daniel Kivlahan, Paul Barnett, and John Finney (The Journal of Behavioral Health Services & Research, 2012)

• “Are longer stays in Veterans Health Administration (VHA) substance abuse residential rehabilitation treatment programs (SARRTPs) associated with better substance-related outcomes? To investigate, up to 50 new patients were randomly selected from each of 28 randomly selected programs (1,307 patients). The goal was to examine if patient and program average length of stay (ALOS) were associated with improvement on Addiction Severity Index (ASI) Alcohol and Drug composite scores in covariate-adjusted, multi-level regression models. Patients in programs with ALOS greater than 90 days tended to have more mental health treatment prior to the index episode and less severe substance-related symptoms, but more homelessness. At follow-up, programs longer than 90 days had the least improvement in the ASI Alcohol composite and significantly less improvement than programs with ALOSs of 15 to 30 and 31 to 45 days (both p<0.05). Therefore, in VHA SARRTPs, ALOS greater than 90 days cannot be justified by the substance use disorder severity of the patients served or the magnitude of the clinical improvement observed.”
• “Again, patients in the longest programs (990 days) had the least severe markers of alcohol use, as well as the highest rate of abstinence from all substances in the 30 days prior to baseline (41%). However, the longest programs also had significantly more patients reporting homelessness or inadequate housing (61%).”
• “Clinicians, and clinical managers must attempt to avoid stays that are either too short, thereby reducing effectiveness and increasing the probability of relapse, or too long, thereby reducing access for waiting patients and unnecessarily increasing costs.”
• “Reviews by Miller and Hester and by Mattick and Jarvis of randomized trials found no outcome difference in any study in which alcohol patients were assigned to shorter versus longer residential stays.” [Hyperlinks added.]
• “In more recent randomized trials, longer stays have been associated with either no benefits or only beneficial effects on some of several outcomes. Witbrodt et al. found that longer length of residential and day hospital addiction treatment up to 2 months, but not beyond, was associated with greater likelihood of abstinence at follow-up.” [Hyperlink added.]
• Observational studies find that longer stays are associated with better outcomes.
• “Programs with the longest average LOS tended to have more heavily “pre-treated” and less symptomatic patients (except for more homelessness) and had significantly less improvement in SUD outcomes than programs with an ALOS less than 45 days. Especially given the significantly higher rates of homelessness in the longest programs, it could be that these SARRTPs are being used primarily as housing rather than for their explicit purpose—the specialized treatment of SUDs.”

The Case For Measuring Quality In Mental Health And Substance Abuse Care, by Harold Pincus, Brigitta Spaeth-Rublee, and Katherine Watkins (Health Affairs, 2011)

• “The [Institute of Medicine (IOM)] committee cited distinctive characteristics and barriers that impede quality improvement in these areas: fewer objective, standardized methods for diagnosing mental health and substance use disorders than for general health conditions; weakness of the evidence base supporting quality measures; inadequate leadership and resources for developing such measures; and lack of coherent strategies for adopting and implementing quality measures.”
• “Response to [IOM] recommendations has been tepid.” […] Fewer than 5 percent of the National Quality Forum’s list of more than 650 vetted indicators specifically relate to care for people with mental health and substance use conditions.”
• “The total economic burden of mental illness— excluding homelessness, incarceration, other health conditions, and early mortality—was estimated to be $317.6 billion in 2002, the equivalent of more than $1,000 for every man, woman, and child in the United States.” [Hyperlink added.]
• “Mark Chassin and colleagues point out that measures should have tight, evidence-based links between process performance and patient outcomes, making them useful for accountability purposes such as accreditation, public reporting, and pay-for-performance. However, few fully validated and reliable performance measures now exist for mental health and substance use disorders.” [Hyperlink added.]

Developing and Validating Process Measures of Health Care Quality: An Application to Alcohol Use Disorder Treatment, by Alex Harris, Daniel Kivlahan, Thomas Bowe, John Finney, and Keith Humphreys (Medical Care, 2009)

• “To optimally serve their intended purposes, process quality measures should: (1) predict facility-level outcomes, such that facilities with higher rates of satisfying the measures also produce better outcomes, (2) predict patient-level outcomes, such that patients who get the care described in the measure have better outcomes than those who do not, and (3) specify care that is supported by the scientific evidence and is acceptable to patients and clinical staff.”
• “Many widely implemented process quality measures fail to meet these rudimentary standards.”
• “The RAND Method is a prominent strategy for translating clinical evidence and practice guidelines into quality measures.21 However, like less systematic strategies, the RAND method does not include verifying directly that the resultant measures are associated with patient outcomes.” [No, I don’t know what the RAND method is, nor where to find a good, brief summary of it.]
• “Recent research found that the VA’s current substance use disorder (SUD) treatment quality measures, focused on at least 90 days of low-intensity treatment retention, is not positively associated with facility-level outcomes and modestly related to patient-level outcomes only in some patient subgroups.” [Hyperlink added.]
• “The candidate measures with the strongest association with improvement were focused on patients receiving 3 to 6 visits in the first month of care.”

Adapting Washington Circle Performance Measures for Public Sector Substance Abuse Treatment Systems, by Deborah Garnick, Margaret Lee, Constance Horgan, Andrea Acevedo, and the Washington Circle Public Sector Workgroup  (Journal of Substance Abuse Treatment, 2009)

• “The Veterans Health Administration currently uses [Washington Circle] WC-like measures to monitor delivery of care. The three performance measures are: [1] Identification—percentage of adult enrollees with a substance abuse claim, defined as containing a diagnosis of substance abuse or dependence or a specific substance abuse-related service, on an annual basis. [2] Treatment Initiation—percentage of adults with an inpatient (IP) substance abuse admission or with an outpatient (OP) claim for substance abuse or dependence and any additional substance abuse services within 14 days following identification. [3] Treatment Engagement—percentage of adults diagnosed with substance abuse disorders who receive two additional substance abuse services within 30 days of the initiation of care.” [Bold added. These terms are used below.]
• “The WC measures focus only on a segment of treatment for substance use conditions […] ‘early engagement‘.” [Hyperlink added.]
• “Several studies already have been focused on the association of the WC process measures with outcomes of substance abuse treatment. Using data from Oklahoma, researchers found that clients who initiated a new episode of OP treatment and who engaged in treatment were significantly less likely to be arrested or incarcerated in the following year (Garnick et al., 2007). These results were replicated using data from Washington State (Campbell, [] 2008). In addition, in a study focused on adolescents in residential treatment, those who achieved continuity of care after their treatment had a significantly increased likelihood of being abstinent at the 3-month follow-up interview (Garner, Godley, Funk, Lee, & Garnick, [2010]).” [Hyperlinks added.]

Measuring the quality of substance use disorder treatment: Evaluating the validity of the Department of Veterans Affairs continuity of care performance measure, by Alex Harris, Keith Humphreys, Thomas Bowe, Daniel Kivlahan, and John Finney (Journal of Substance Abuse Treatment, 2009)

This study examined the patient- and facility-level associations between the continuity of care performance measure adopted by the Department of Veterans Affairs (VA) and improvements in self-administered Addiction Severity Index (ASI) composites and other indicators of problematic substance use. Up to 50 patients from each of a nationally representative sample of 109 VA substance use disorder (SUD) treatment programs at 73 VA facilities were assessed at intake and posttreatment. The continuity of care performance measure specifies that patients should receive at least two SUD outpatient visits in each of the three consecutive 30-day periods after they qualify as new SUD patients. In analyses adjusting for baseline characteristics, meeting the continuity of care performance measure was not associated with patient-level improvements in the ASI alcohol or drug composites, days of alcohol intoxication, or days of substance-related problems. Facility-level rates of continuity of care were negatively associated with improvements in ASI alcohol and drug composites. The continuity of care performance measure derived from prior patient-level evidence did not discriminate facility-level performance as predicted. Translating research into process-of-care quality measures requires postconstruction validation.

Veterans Affairs facility performance on Washington Circle indicators and casemix-adjusted effectiveness, by Alex Harris, Keith Humphreys, and John Finney (Journal of Substance Abuse Treatment, 2007)

• “Excellence in one area (e.g., prevention, detection, treatment, or maintenance) does not necessarily ensure excellence in others []. For example, a highly effective system of screening and detection will not produce good outcomes unless it is complemented with guideline concordant follow-ups to positive screens.”
• “The National Committee for Quality Assurance, the Department of Veterans Affairs’ (VA) Program Evaluation and Resource Center (PERC), and many public and private managed care organizations have adopted three SUD performance indicators (identification, initiation, and engagement) developed by the Washington Circle (WC), an organization supported by the federal Center for Substance Abuse Treatment.” [Hyperlink added.]
• “A substantial literature already shows that individuals with SUDs who are diagnosed and treated have better long-term outcomes than untreated individuals (e.g., Timko, Moos, Finney, & Moos, 1994; Weisner, Matzger, & Kaskutas, 2003). One study (Garnick, Horgan, Lee, et al., 2006 [unpublished presentation]) found that adult SUD outpatients engaging in treatment, as defined by the WC indicator, but not those initiating treatment, were less likely to be arrested and incarcerated subsequently. In contrast, the present study views WC indicators as characteristics of the facility rather than as patient-level factors. We were interested if the WC indicators can be used as proxy measures for average clinical improvement for patients who are already identified, initiated, and, for the most part, engaged in treatment. Put differently, should administrators, clinical managers, and consumers assume that facilities having higher scores on WC indicators also produce better facility-level patient outcomes?” [Hyperlinks added.]
• “”Higher initiation rates were not associated with facility-level improvement in ASI [Addiction Severity Index, self-administered version] alcohol composite scores but were modestly associated with greater improvements in ASI drug composite scores. […] These results do not imply that being identified is unimportant for individual patients. Rather, they indicate that, in the VA system, there is no outcome advantage for identified patients who are treated at facilities with high identification rates.”
• “A 0.10 increase in initiation rate, from 0.25 to 0.35, for example, predicts a 0.008 point average improvement in ASI drug composite scores, which would be roughly associated with a 3% increase in abstinence from drugs at 7-month follow-up. Again, we are not evaluating the effect of initiation for individual patients. In a sample of VA patients who all had initiated treatment, being treated in a facility with higher overall initiation rates predicted a better outcome of treatment as measured by improvement on the ASI drug composite.”
• “Engagement rates were not associated with average changes in ASI alcohol composite scores or ASI drug composite scores.”
• “Our retrospective evaluation of a large VA data set suggested that these WC performance measures were not potent as predictors of posttreatment outcomes 7 months after treatment intake.”
• “Our results also raise questions about the mechanisms underlying the associations between WC indicators and patient casemix variables. Do identification, initiation, and/ or engagement processes affect casemix, do casemix factors causally influence performance on the WC indicators, or are relationships between casemix factors and WC indicators due to the effects of third variables, such as urban versus other location of a health care facility? To begin to address these issues, research on the correlates of change in WC indicators is needed. The results should be interpreted in the context of the time between measurement points. Average length of follow-up was about 7 months, whereas the WC measures all focused on the first month of treatment. Many variables, including many not associated with treatment, can intervene between these two time points to affect patient outcomes.”
• “We conclude that the three WC indicators focused on the early stage of treatment may be best thought of as tapping necessary but insufficient processes for patients with SUD to achieve better longer term outcomes. A facility may have excellent performance in identification, initiation, and engagement, but its patients may not have greater improvement than patients at facilities with lower rates on these process indicators. Conversely, facilities may have lower rates of identification, initiation, and engagement, yet may have a high rate of improvement among patients with SUD. Ideally, the WC indicators would be supplemented with other measures of treatment effectiveness.”

The Quality of Care for Adults with Mental and Addictive Disorders: Issues in Performance Measurement, by Constance Horgan and Deborah Garnick (for the IOM Committee on Crossing the Quality Chasm, 2005)

Studies of treatment retention indicate that patients who remain in treatment for a longer duration of time have better outcomes (Grella et al., 1999; Hubbard et al., 2003). This positive association between retention and treatment success, along with evidence about treatment intensity (Fiorentine & Anglin, 1996; McLellan et al., 1993; Simpson et al., 1995), provided the rationale for the substance abuse initiation and engagement measures developed by the Washington Circle and adopted by the NCQA. In brief, becoming engaged in treatment with an initial intensity of services was considered to be an important intermediate process measure that is closely related to outcomes (McCorry et al., 2000). [Hyperlinks added.]

Predictors of Initiation and Engagement in VA Substance Use Disorder (SUD) Treatment, by Alex Harris and Tom Bowe

We examined if patient or setting characteristics predict subsequent addiction treatment involvement in a sample of 270,877 Veteran Administration (VA) patients identified with substance use disorders (SUD). Patient characteristics (e.g., gender, age, marital status) and treatment specialty of the unit where the SUD was identified (SUD, Psychiatric, or Other) were used to predict two aspects of treatment involvement, specifically the HEDIS Initiation and Engagement quality indicators (National Committee for Quality Assurance, 2006 [link to the 2014 version]). Overall, patients who were female, not married, younger, and had their SUDs identified in SUD or Psychiatric treatment units initiated and engaged in treatment at significantly higher rates. For example, a younger, single patient who was identified with an SUD in an SUD specialty unit had a predicted probability of meeting the Initiation criteria of 0.54 compared to 0.14 for an older married patient identified in a general medical setting. This research facilitates the identification of patients with lower likelihoods of initiation and engagement in treatment and may inform intervention efforts to improve rates of initiation and engagement in targeted groups and settings.

HEDIS Initiation and Engagement Quality Measures of Substance Use Disorder Care: Impact of Setting and Health Care Specialty, by Alex Harris, Thomas Bowe, John Finney, and Keith Humphreys (Population Health Management, 2009)

Many health care systems track the HEDIS measures of initiation and engagement in substance use disorder (SUD) care. However, the impact of setting of care (inpatient vs. outpatient) and health care specialty (SUD, psychiatric, other) on the likelihood of patients meeting the initiation and engagement criteria are unknown. If the vast majority of initiation and engagement occurs within SUD specialty clinics, then these quality measures could be used to discriminate among and incentivize SUD clinic managers. However, if these criteria are satisfied in different settings and specialties, then they should be considered characteristics of the entire facility, rather than just specialty SUD units.

Using a Markov model, the probabilities of advancing to treatment initiation and engagement given initial setting and specialty of care were estimated for 320,238 SUD-diagnosed Veterans Health Affairs (VA) patients. Patients in SUD specialty units progressed more often (diagnosis to initiation, initiation to engagement) than patients in other specialties. Progression through the criteria differed for inpatients vs. outpatients. Approximately 25% of initiation and over 40% of engagement occurred outside of SUD specialty care.

VA patients who have contact with SUD specialty treatment have higher rates of advancing to initiation, and from initiation to engagement, compared to SUD-diagnosed patients in psychiatric or other medical locations. Even so, a substantial portion of initiation and engagement occurs outside of SUD specialty units. Therefore, these quality measures should be considered measures of facility performance rather than measures of the quality of SUD specialty care. The usual combining of inpatient and outpatient performance on these measures into overall facility scores clouds measurement and interpretation. […]

The initiation and engagement measures were originally developed by the Washington Circle, an organization supported by the federal Center for Substance Abuse Treatment, and then adapted for use as HEDIS measures. Initiation refers to the percentage of patients diagnosed with a substance use disorder (SUD) with (a) at least a 60-day SUD service-free period, and (b) either an inpatient= residential admission with a SUD diagnosis, or both an initial SUD-related outpatient visit and an additional SUD-related visit within 14 days. Engagement refers to the percentage of outpatients with diagnosed SUDs that (a) meet the initiation criteria, and (b) receive 2 additional SUD-related visits within 30 days following initiation. Whether the patient initiates or engages after a qualifying visit, the patient is not eligible to qualify again during the same year. For those who initiate in inpatient=residential settings, the 2 additional visits for engagement must occur within 30 days of discharge.

A conceptual framework for drug treatment process and outcomes, by D. Dwayne Simpson (Journal of Substance Abuse Treatment, 2004)

Evidence from specialized treatment evaluations and large-scale natural studies of treatment effectiveness is organized conceptually into a “treatment model” for summarizing how drug treatment works. Sequential relationships between patient and treatment program attributes, early patient engagement, recovery stages, retention, and favorable outcomes are discussed–along with behavioral, cognitive, and skills training interventions that have been shown to be effective for enhancing specific stages of the patient recovery process. Applications of the treatment model for incorporating science-based innovations into clinical practice for improving early engagement and retention, performance measurements of patient progress, program monitoring and management using aggregated patient records, and organizational functioning and systems change also are addressed.

Veterans Health Administration Mental Health Program Evaluation Capstone Report, by Katherine Watkins and Harold Pincus (Altarum Institute and RAND, 2011)

In our study, fewer than 15 percent of veterans in the SUD cohort who were in an NTE [new treatment event] initiated treatment within the first two weeks of the NTE, and fewer than 14 percent became engaged with treatment (defined as two or more outpatient visits) within a month of the NTE. One-quarter (25 percent) received follow-up care within 90 days of the NTE. The NCQA [National Committee for Quality Assurance] reports results for treatment initiation and engagement annually across the commercial, Medicare, and Medicaid markets []. The relevant results from 2007 are shown in Table 5.6 [below]. As discussed in Chapter 8, the specification of this indicator in the present study aligns closely but not perfectly with the specification reported by the NCQA. In particular, the denominator populations differ in a number of potentially important ways. First, the NCQA specification includes members between the ages of 13 and 18 who are not eligible for entry into the armed services or our study cohort. Second, the NCQA specification requires a four-month break in care before a patient can qualify for a ―new episode of care, while our study requires a five-month break in care before a patient can qualify for an NTE. Third, the NCQA specification considers only members with alcohol or other drug dependence (AOD), while our study considers all veterans in the SUD cohort, including those with alcohol or drug abuse disorders. Finally, since the VA has mandatory alcohol screening, identified patients in the VA cohort are likely to be, on average, less sick than patients in the private sector, on Medicare, or on Medicaid, as these patients are identified in claims data only after having received a diagnosis. The likely direction of bias from these differences is unclear.

@afrakt