Not a subscriber?

You may access and print any article from the Journal of Clinical Sleep Medicine for your personal scholarly, research,
and educational use. Please note, access to the article is from the computer on which the article was purchased only.
Purchase of the article does not permit distribution, electronic or otherwise, of the article without the written
permission of AASM. Further, purchase does not permit the posting of the article text on an online forum or website.

ABSTRACT

Study Objectives:

To identify and validate a surrogate measure of long-term adherence to positive airway pressure (PAP) therapy in patients with obstructive sleep apnea (OSA).

Design:

Retrospective cohort study.

Setting:

Academic center.

Participants:

220 consecutive patients with OSA.

Interventions:

N/A.

Measurements:

In patients with OSA who were receiving PAP therapy (for > 1 year), PAP adherence measured by device-download and defined by Medicare criteria was compared to refill rates for mask and other PAP therapy accessories. First, receiver operating characteristic (ROC) curves were constructed to identify a threshold value of refills per year that discriminated best between PAP adherent and non-adherent patients (derivation set; n = 100). Then the predictive accuracy of the threshold value of refills per year was tested in an additional 120 consecutive patients (validation set).

Results:

From the derivation set, ROC curve with good discriminant characteristics (ROC 0.83; 95% confidence intervals [CI], 0.75, 0.91, p < 0.0001) was used to identify a threshold value of refills (0.7 refills/year) for distinguishing PAP adherent and non-adherent patients. Subsequently, when the threshold was applied to the validation set, the likelihood ratio for a positive test (weighted for prevalence) predicting adherence to PAP therapy was 7.3 (95%CI, 3.8, 14), and likelihood ratio for a negative test was 0.6 (95%CI; 0.4, 0.8).

Conclusion:

Refill rate of PAP accessories exhibited good test characteristics for predicting long-term PAP adherence. Such a surrogate measure based upon insurance claims data can be a powerful epidemiological tool in bioinformatics-aided comparative-effectiveness research and to monitor clinical performance of health systems.

Obstructive sleep apnea (OSA) is a prevalent condition that is most frequently treated with positive airway pressure (PAP) therapy.1,2 Non-adherence to PAP therapy, however, afflicts a high proportion of adults with sleep apnea (46% to 83%),3 and such poor adherence is associated with increased risk for fatal and non-fatal cardiovascular events.4 Consequently, assessment of adherence to PAP therapy has been advocated as a physician performance assessment tool in the management of patients with OSA.5 However, obstacles to the universal and consistent measurement of PAP adherence remain.5 Therefore, a reliable surrogate for measuring long-term PAP adherence—both within and across healthcare systems—is direly needed for more consistent assessment of the performance of healthcare systems and providers.

Adherence to medications can be measured both directly (by measuring metabolites of drug in urine) or indirectly by measuring claims data pertaining to medication refills.6 While the former methodology may be more accurate, the latter offers a more pragmatic yet reliable tool for assessing medication adherence and is universally accepted as an adequate surrogate.6 Superior medication refill rates have been associated with more favorable patient outcomes, such as reduced exacerbations in chronic lung disease, reduced hospitalizations in patients with cardiovascular disease and schizophrenia, and reduced relapse rates in patients with rheumatoid arthritis and multiple sclerosis.7–12 Similarly, although PAP device download affords the best measure of adherence to PAP therapy,13–18 information pertaining to refill of PAP accessories (masks, hoses, filters, humidifiers, and even PAP machines) due to wear and tear may potentially serve as a surrogate measure of adherence. To our knowledge, a systematic assessment of the relationship between refills of such PAP accessories and objectively measured device-downloaded adherence data has not been performed. Such an association between refills of PAP accessories and objective device-downloaded adherence, if validated, could better facilitate adherence monitoring in health systems and support quality improvement initiatives that target management of OSA.

BRIEF SUMMARY

Current Knowledge/Study Rationale: Adherence to medications assessed by medication refill rate is a pragmatic real-world bioinformatics aided tool that has been associated with patient outcomes in many disease conditions. To our knowledge, a systematic assessment of whether refills of PAP accessories can parallel objectively measured device-downloaded adherence data has not been performed.

Study Impact: Refill rates of PAP device accessories exhibited good test characteristics for predicting long-term adherence to PAP therapy. Such a surrogate measure of long-term PAP adherence - based upon insurance claims data - can be a powerful epidemiological tool to assess quality of healthcare delivery in the management of patients with sleep-disordered breathing.

In line with such rationale, we set out to identify and validate a surrogate measure of long-term adherence to positive airway pressure (PAP) therapy in patients with obstructive sleep apnea (OSA). We hypothesized that refill rates for PAP device accessories can predict long-term PAP adherence in patients with OSA.

METHODS

Study Population

This is an analysis of an historical cohort contained in an electronic disease-registry that was set-up for a quality initiative in 2003 at the Tucson VA Sleep Program. Two hundred twenty consecutive patients who had received PAP therapy > 1 year were included in the analysis as follows: (a) An initial derivation set of 100 consecutive patients with OSA receiving PAP therapy (> 1 year) in whom PAP adherence was measured by device-download. (b) A validation data set of the next 120 consecutive patients with OSA receiving PAP therapy (> 1 year) in whom PAP adherence was measured by device download.

OSA was diagnosed by polysomnography that revealed an apnea-hypopnea index > 5/h, with hypopneas defined as > 50% reduction in flow or effort and ≥ 3% oxygen desaturation.

Fourteen percent of the consecutive patients were excluded due to unavailability of the download that resulted in the 220 patients reported in this study. The study was approved by the University of Arizona Institutional Review Board and the VA Research and Development Committee.

Gold Standard

Adherence to PAP therapy was measured by device download using manufacturer software and mailed in memory cards (ENCORE Pro, Philips-Respironics, Murrysville, PA). The usage was expressed in minutes of use per day, and adherence was defined by Medicare criteria requiring ≥ 4 h of PAP use on 70% of nights (5 of 7, which translated into 21 of 30 days) during 30 consecutive days within the prior 90-day period. Unlike Medicare criteria that require such data from the first 90 days of therapy, the adherence reported here was collected from the prior 90 days of therapy because our objective was to measure a correlate of long-term PAP adherence. Additionally, adherence was also defined by a more strict criterion of device usage > 4 h/day on all days.19,20 For the latter, the total number of hours of usage during the prior 90-day period available in the device memory was divided by the total number of days (90 days), which yielded average use of PAP therapy. Patients with an average device use ≥ 4 h were considered to be adherent.

Surrogate Measure

Refills of masks, hoses, filters, humidifiers, chin straps, and even PAP devices were derived from the dispensation logs that were maintained for each patient in the prosthetics department of the integrated healthcare system at Tucson VA. The number of total refills during the entire follow-up period (ranging from 1 to 7 years) was divided by follow-up period expressed in years to yield the refills per year. Moreover, similar refill rates were calculated for each of the above-mentioned accessories. All of the patients were receiving care exclusively from the VA in Tucson. Patients at the VA in Tucson are “means tested” and seldom have third party insurance. Patients who died or disenrolled from the Tucson VA were included if they had been on PAP therapy for at ≥ 1 year of PAP therapy, as this was our a priori definition for long-term PAP adherence and served as an inclusion criterion. For those patients who expired, or disenrolled, after being on 1 year of PAP therapy, the duration of time that they were alive since the time they received PAP therapy was censored at the day of death (or disenrollment) and expressed in years.

Statistical Methodology

Adherence to PAP therapy was treated as a binary dependent variable (defined by Medicare criteria). In the derivation set of 100 patients, the sensitivity and specificity with which the total number of refills per year could discriminate PAP adherent and non-adherent status was assessed with receiver operating characteristic (ROC) curves.21 The area under the ROC curve, which summarizes the performance of the refills per year in predicting PAP adherence, was calculated. Cross-plots between sensitivity and specificity values were performed to objectively identify the threshold value with the best sensitivity and specificity to discriminate PAP adherent and non-adherent patients.

The threshold value of refills per year derived from the initial derivation data-set was then used a test to predict long-term PAP adherence or non-adherence in a validation dataset of 120 patients. True positive, true negative, false positive, false negative, sensitivity, specificity, positive predictive value, and negative predictive value were derived as per standard convention.22

Likelihood ratio (LR) for refills per year as a diagnostic test to predict long-term PAP adherence was performed using conventional and prevalence weighted techniques for positive test (LR+) and negative test (LR−) results.23 The LR+ is calculated as LR+ = (sensitivity)/(1-specificity) and LR− = (1-sensitivity)/(specificity). In our case, LR+ greater than 1 indicates the test result (i.e., refills per year) was associated with the condition of interest (i.e., long-term PAP adherence), whereas LR− less than 1 indicates that refills per year was associated with non-adherence. LR+ greater than 1 would argue in favor of long-term PAP adherence; the bigger the number, the more convincingly the finding suggests that long-term PAP adherence exists. Findings whose LR+ lie between 0 and 1 argue against PAP adherence; the closer the LR is to 0, the less likely the patient is PAP adherent. Findings whose LRs equal 1, or whose 95% confidence interval (95% CI) overlaps the value “1” lack predictive value. Because LR+ can change based upon the prevalence of the condition (in this case the prevalence of long-term PAP adherence), LR+ was weighted for prevalence as follows LR+[P] = (prevalence)(sensitivity)/(1-prevalence)(1-specificity).24 Similarly, LR− was weighted for prevalence as follows LR−[P] = (prevalence)(1-sensitivity)/(1-prevalence)(specificity).

Sensitivity analysis with adherence defined by a stricter definition of 4 h/night was also performed. Also, test characteristics for mask refills alone was performed. Sensitivity analysis with arbitrarily defined threshold levels of refills (total and mask alone) were also performed. We performed univariate linear regression with long-term PAP adherence (expressed as average min used per day) and determinants such as age, gender, BMI, days on PAP, Epworth score, AHI, total refills per year, mask refills per year, AHI at therapeutic pressure (residual AHI), and PAP pressure. Significant determinants (p < 0.05) were entered into a multivariate regression model (forced entry), and care was taken to prevent collinearity between determinants by performing collinearity diagnostics. Group comparisons of continuous variables were made by unpaired t-tests or nonparametric equivalents. p values < 0.05 were considered significant. All data are shown as mean and standard deviation (SD), or median and interquartile range (IQR). SPSS v12.01 (SPSS Inc., Chicago IL) was used for statistical analysis.

RESULTS

There were 220 patients (age 57 ± 11 years; 94% men) with a body mass index 37.4 ± 5.8 kg/m2 (Table 1). Average therapeutic CPAP pressure was 10.7 ± 3.1 cm H2O. Seven percent of patients received bilevel PAP therapy, while 93% received CPAP therapy. The mode of therapy did not influence adherence. In the derivation set, the duration of follow-up was 4.8 ± 1.2 years (range 1 to 7 years). In the validation set, the duration of follow-up was 4.7 ± 1.2 years (range 1 to 6.25 years). For both definitions of adherence, the duration of follow-up was not different between the adherent and non-adherent groups (p > 0.8). In the validation set, the proportion of patients who were adherent by Medicare criteria was 67%, and by the strict 4-h criterion was 54%. Of all the various types of refills, besides total refills per year, we selected mask refills alone rather than refill rates of hose, filter, humidifier, or even device by themselves, because the mask refills constituted the bulk of refills. Specifically, masks constituted 54.7% of total refills, whereas filters (30.4%), hose (5.8%), device (5.3%), and humidifier (3.7%) constituted a minority of the total refills requested.

Table 1

Derivation Set

In the derivation set, the refill rate of all PAP-related accessories was a median of 0.75/year (interquartile range [IQR], 0.17, 1.8) and for masks alone was a median of 0.4/year (IQR; 0, 0.8). Filters, hoses, humidifier, and PAP device constituted the remainder of the refills, with a median of 0.2/year (IQR; 0, 1.0).

All adherence information is represented as minutes of use per day on all days of the week. Refill rate of PAP device accessories was closely related to adherence to PAP therapy (R = 0.42; p < 0.0001).

Figure 1

Long-term adherence to positive airway pressure (PAP) therapy downloaded from PAP device is plotted against refill rate of PAP device accessories (masks, hoses, filters, humidifier, chin strap, and even PAP devices)All adherence information is represented as minutes of use per day on all days of the week. Refill rate of...

ROC Curves from Derivation Dataset

For patients in the derivation dataset (n = 100), ROC curves were constructed for various threshold levels of total refills and mask-only refills and are shown in Figure 2. Different ROC curves were plotted for the various definitions of long-term PAP adherence (Figure 2). The corresponding area under the curves and the threshold value of refill rate associated with the best sensitivity and specificity are shown in Table 2.

{ label (or @symbol) needed for fn } Area, Area under the curve plotted by receiver operating characteristics curve that depicted the pattern of sensitivities and specificities observed when the performance of the test is evaluated at several different diagnostic thresholds; SE, standard error; 95%CI, 95% confidence intervals (upper and lower); Threshold, Threshold value with the best sensitivity and specificity identified by performing a cross-plot of sensitivity and specificity at varying thresholds.

* Based upon device-download of usage with adherence defined as usage > 4 h/day on 5 days a week.

** Based upon device-download of usage with adherence defined as usage > 4 h/day on all days.

Table 2

Validation Set

Test characteristics of threshold values of refill rates used to predict long-term PAP adherence in the validation dataset (n = 120) are shown in Table 3. The refill rate of all PAP related accessories had the best LR+[P] for predicting long-term PAP adherence by Medicare criteria. In general, the test performance characteristics of refill rate of all PAP related accessories were good for both definitions of long-term PAP adherence (Table 3). We also performed sensitivity analysis with regard to the threshold level of mask refills by arbitrarily selecting a threshold of 3, 2, and 1 mask per year as well, as for 3, 2, and 1 total refills (any PAP accessory including mask) per year. Such thresholds were associated with excellent LR+ (0.7 to 1.0) and PPV (2.5 to ∞), but the LR− (0.3 to 0.84) and NPV (0.37 to 0.7) suffered.

Test characteristics of the threshold values of refill rates used to predict long-term PAP adherence

Table 3

Multivariate Regression

We performed linear regression with long-term PAP adherence (expressed as average min used per day) and age, gender, BMI, days on PAP, Epworth score, AHI, total refills per year, mask refills per year, AHI at therapeutic pressure (residual AHI), and PAP pressure. In univariate regressions, greater BMI, higher AHI, greater total refills, or greater mask refills per year were found to be significantly associated with adherence to PAP therapy (p < 0.0001). Multivariate regressions yielded greater AHI and greater total refills per year as being independently associated with greater adherence to PAP therapy (model R2 = 0.20; p < 0.0001). We did not load mask refills per year together with total refills per year, as they were collinear. When masks per year was loaded into the model (model R2 = 0.27; p < 0.0001), greater masks per year was independently associated with greater odds of long-term PAP adherence.

DISCUSSION

Refill rate of PAP device accessories exhibited good test characteristics for predicting long-term adherence to PAP therapy (defined as > 1 year of therapy). Specifically, refill rate of all PAP accessories (mask, hoses, filters, humidifier, and even PAP devices combined) had the best predictive value for discriminating long-term PAP adherence and was superior to that of mask refill rate alone. Notably, refill rates were much greater in PAP adherent patients than in non-adherent patients, regardless of how adherence was defined—Medicare criteria or 4 hours per day on all days. To our knowledge this is the first study to systematically develop and validate a surrogate measure of long-term PAP adherence based upon refills of PAP accessories in a manner similar to that of accepted techniques of measuring medication adherence based upon refill rates.6

Non-adherence to PAP therapy afflicts a high proportion of adults with sleep apnea,3 and such poor adherence is associated with increased risk for fatal and non-fatal cardiovascular events.4 There are numerous interventions and factors that either promote or are associated with better adherence to PAP therapy.3,14,20,25–28 However, in order for such reversible factors to be targeted, or interventions to be implemented, physicians and healthcare systems need to universally monitor PAP adherence. But there are many impediments to implementing such practices, including greater cost, lack of care coordination with durable medical equipment companies, differing software platforms across various manufacturers, and lack of patient cooperation (such as failure to bring or mail memory cards or even transmit data via phone or internet).

While objective measurement of short-term PAP adherence (90 days) is required in Medicare beneficiaries, there is no requirement for measuring long-term PAP adherence.29 Although data suggests that long-term adherence patterns are established early,30 this may not be true for all patients.15 Long-term adherence over one year may differ from adherence measured in the first month of treatment initiation.15 In a time series analysis by Aloia and colleagues, slow decliners, slow improvers, and variable users constituted as much as 44% of a cohort of PAP users followed for one year.15 Our validated surrogate measure of PAP adherenc—based upon PAP accessory refills—was measured beyond 1 year and up to 7 years of PAP therapy and is therefore indicative of long-term PAP adherence.

Long-term PAP adherence has been shown to confer cardiovascular disease modification and survival benefits in observational cohort studies over a 10-year follow-up period.4 Such benefits cannot possibly be achieved within the recommended 90 days of commencing PAP therapy.29 Therefore, it follows that we should be cognizant of measuring long-term (> 1 year) adherence while keeping costs of such adherence monitoring low. Measurement of refill rate for PAP accessories can be a powerful tool in integrated healthcare systems or in Medicare beneficiaries to monitor long-term adherence to PAP therapy in a cost-effective manner. Such surrogate monitoring of PAP adherence can assist health services researchers to account for the effects of PAP therapy when measuring patient outcomes, healthcare utilization, and performance of healthcare systems.5 Also, it is unclear as to whether the health systems that struggle to accomplish PAP adherence measurements over the first 90-day period can indeed effectively “monitor” adherence over the years that it takes to accomplish benefits. We believe that this implementation gap can be effectively filled by the surrogate measure of adherence that we have proposed and validated.

The test characteristics of all refills of PAP accessories reveal that this surrogate measure has good PPV and LR+[P] values for predicting long-term PAP adherence (Table 2). However, the NPV and LR−[P], although acceptable, are not as good. This discrepancy may be explained by the fact that some patients may have continued to use their old masks and PAP accessories over the long term. Therefore, unlike medications, resilient PAP accessories may have lower refill rates that may conceivably underestimate long-term PAP adherence (Figure 1).

Limitations

There are limitations to our study. First, the proposed surrogate measure for long-term PAP adherence is dependent on refill benefits of a given healthcare system. While the PAP accessory refill allowances of our integrated healthcare system is mirrored after the Medicare benefits, other systems may have more lenient or stricter refill benefits. Therefore, the threshold rate that we propose in this paper may not be generalizable to other integrated healthcare systems or healthcare systems that are not integrated. However, this problem can be surmounted by calibrating or validating the surrogate measure of long-term PAP adherence within other healthcare systems prior to implementation. Also, outcomes studies or physician performance assessments within a healthcare system should not be influenced by such sources of error. Moreover, our study was made feasible because we are an integrated healthcare system with PAP refill information made easily available from the electronic medical records-derived disease registry, which may not be as easily accomplished in systems that do not have such an integrated model. All of the patients were receiving care exclusively from the VA in Tucson, but we realize that they may sought PAP refills through other sources. However such refills from outside the system are unlikely for the following reasons: (a) Patients at the VA in Tucson are “means tested” and seldom have third party insurance. (b) Even when such private insurance exists, the veterans are unlikely to seek refills from such sources because they would need to pay full price (online) or co-pays (private insurance-based) for such refills, whereas the refills from the Tucson VA's prosthetics department are available at no cost.

Second, considering the time period of 1 to 7 years in our validation dataset, this surrogate measure is not recommended for the assessment of PAP adherence in the short term (61-90 days) or support continued coverage from Medicare. Third, our data were primarily derived from middle-aged to older men; this limits generalizability to women. Systems with greater representation of women may need to reproduce our results Fourth, while the threshold levels of the surrogate measure predicted long-term PAP adherence in a validation dataset, care should be taken in not using refills of PAP accessories as a prospective predictor of adherence on the basis of this study. Lastly, masks or total refill was associated with long-term PAP adherence, and therefore a causal effect cannot be implied.

CONCLUSION

Refill rate of PAP accessories exhibited good test characteristics for predicting long-term adherence to PAP therapy. Such a surrogate measure based upon insurance claims data can be a powerful epidemiological tool in comparative-effectiveness research utilizing bioinformatics of claims data from integrated healthcare systems or Medicare beneficiaries, and for monitoring performance of health systems and supporting quality improvement initiatives that promote PAP adherence.

DISCLOSURE STATEMENT

This was not an industry supported study. Dr. Parthasarathy has received a Philips-Respironics, Inc. research grant to Biomedical Research Education Foundation of Southern Arizona. The other authors have indicated no financial conflicts of interest.

ACKNOWLEDGMENTS

Work conducted at Southern Arizona VA Health Care System, Tucson, AZ. Support from the National Institutes of Health (Grant No. HL095748 awarded to Dr. Parthasarathy) and Department of Veterans Affairs Health Service Research & Development (Grant No. PPO 10-066 awarded to Dr. Parthasarathy).