Article Text
PDF
Abstract
Objective Primary healthcare (PHC) quality improvement (QI) initiatives are designed to improve patient care and health outcomes. We evaluated the Quality Improvement and Innovation Partnership (QIIP), an Ontario-wide PHC QI program on access to care, diabetes management and colorectal cancer screening. This manuscript highlights the impact of QIIP on diabetes outcomes and associated vascular risk factors.
Research design and methods A cluster matched-control, retrospective prechart and postchart audit was conducted. One physician per QIIP-PHC team (N=34) and control (N=34) were recruited for the audit. Eligible charts were reviewed for prespecified type 2 diabetes mellitus clinical process and outcome data at baseline, during (intervention range: 15–17.5 months) and post. Primary outcome measures were the A1c of patients above study target and proportion of patients with an annual foot exam. Secondary outcome measures included glycemic, hypertension and lipid outcomes and management, screening for diabetes-related complications, healthcare utilization, and diabetes counseling, education and self-management goal setting.
Results More patients in the QIIP group achieved statistically improved lipid testing, eye examinations, peripheral neuropathy exams, and documented body mass index. No statistical differences in A1c, low-density lipoprotein or systolic/diastolic blood pressure values were noted, with no significant differences in medication prescription, specialist referrals, or chart-reported diabetes counseling, education or self-management goals. Patients of QIIP physicians had significantly more PHC visits.
Conclusion The QIIP-learning collaborative program evaluation using stratified random selection of participants and the inclusion of a control group makes this one of the most rigorous and promising efforts to date evaluating the impact of a QI program in PHC. The chart audit component of this evaluation highlighted that while QIIP improved some secondary diabetes measures, no improvements in clinical outcomes were noted. This study highlights the importance of formalized evaluation of QI initiatives to provide an evidence base to inform future program planning and scale-up.
- diabetes
- evidence-based medicine
- primary healthcare
- health policy
- quality improvement
- program evaluation
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Significance of this study
What is already known about this subject?
Primary healthcare quality improvement (QI) initiatives targeting type 2 diabetes mellitus (T2DM) are designed to improve diabetes care and associated health outcomes.
What are the new findings?
The Quality Improvement and Innovation Partnership, a province-wide QI initiative in Ontario, Canada, improved some secondary T2DM measures (lipid testing, eye examinations, peripheral neuropathy exams, and documented body mass index); however, no improvements in T2DM clinical outcomes (A1c, low-density lipoprotein or systolic/diastolic blood pressure values) were noted.
How might these results change the focus of research or clinical practice?
With resources in primary healthcare already strained by the rising economic and public health burden of T2DM, this research highlights the importance of rigorous and formalized evaluation of primary healthcare QI initiatives to provide an evidence base to inform future program planning and scale-up.
Introduction
The prevalence of diabetes is rising at an alarming rate.1 2 Given that 75% of patients in Ontario with diabetes receive care exclusively from their family physician (FP),3 the primary healthcare (PHC) system is being placed under immense pressure to perform well and meet clinical targets and patient needs. As more Canadians are diagnosed and receiving diabetes care, the cost of diabetes is also proving to be a significant burden on the economy.1 2 These combined stressors are placing strain on the PHC system, all of which are only expected to increase, hence, the need to develop innovative strategies for facilitating diabetes prevention and management in PHC.
In an effort to reform and improve PHC delivery, the Ontario Ministry of Health and Long Term Care (MOHLTC) established Family Health Teams (FHTs) in 2005.4–6 These new models were envisioned as a multidisciplinary, team-based, patient-centered, and proactive healthcare delivery system focused on health promotion, disease prevention and chronic disease management. To help FHTs navigate this transition, the provincial Quality Improvement and Innovation Partnership (QIIP), now amalgamated with Health Quality Ontario (HQO), was established in 2005 by the MOHLTC. The overall goal of QIIP was to enable PHC teams to enhance the care experience and outcomes of the populations they serve through continuous quality improvement (QI) in effective, efficient, accessible, comprehensive, patient-centered and team-based PHC.
QIIP program description
From 2008 to 2010, QIIP used the Institute for Healthcare Improvement’s Breakthrough Series (IHI-BTS) Model7 to implement a QI learning collaborative (LC) program with teams of FPs, allied health professionals, and administrative staff from FHTs and community health centers (CHCs) across Ontario. IHI-BTS uses an adult learning model that typically brings together teams to learn from each other and experts in topic areas targeted for improvements.7 The overall purpose of QIIP was to educate, train and enable PHC teams to improve patient care and outcomes in three areas: (1) chronic disease management—program target: type 2 diabetes mellitus (T2DM) management; (2) disease prevention—program target: colorectal cancer (CRC) screening; and (3) office access and efficiency—program targets: ‘advanced access’ to healthcare and team functioning.
The QIIP-LC program was delivered over three waves; each wave was standardized in content and consisted of prework, three 2-day learning sessions (separated by action periods) and one summative congress. This ensured all participants received the same program. In total, each wave of the program lasted between 15.5 and 17 months. Learning sessions provided opportunities for teams to learn QI methodology such as Plan-Do-Study-Act cycles8 and Ontario’s Chronic Disease Prevention and Management framework.9 Learning sessions also provided a rich opportunity to network and learn from each other. The QIIP-LC program included support from QI coaches (QIC) whose role was to facilitate and mentor participants throughout the program and support electronic medical record optimization.10 A series of communication strategies or channels included a virtual office, list serve and monthly teleconferences between PHC teams and their QICs. Participation in QIIP was voluntary and open to PHC organizations across Ontario. More details about the QIIP-LC program characteristics are available in the program logic model (online supplementary file S1) and published in Harris et al.11
Research design and methods
An external evaluation of the QIIP-LC program was conducted using a rigorous mixed-method, multimeasure design. The evaluation included: (1) development of a logic model (see online supplementary file S1) and assessment oriented process evaluation; (2) a retrospective, cluster, matched-control, prechart and postchart audit on the management of T2DM and rate of CRC screening; (3) a controlled post-only survey of practices participating in the chart audit on advanced access to healthcare; (4) semistructured, post-only, in-depth telephone interviews; (5) post-only web-based participant survey; and (6) health administrative data analysis with the Institute for Clinical Evaluative Sciences.11
Primary outcome results11 and the impact of QIIP on team functioning12 are published elsewhere. This paper presents the results of the chart audit related to the impact of the QIIP-LC program on T2DM management, including: (1) glycemic outcomes and management; (2) vascular protection (hypertension and dyslipidemia) and (3) screening for DM-related complications. This paper also presents the chart audit findings related to healthcare utilization (healthcare professional (HCP) visits, specialist referral for diabetes care) and diabetes counseling, education and self-management goal setting.
Evaluation of the QIIP-LC was approved by the Research Ethics Board at Western University and Queen’s University. A waiver of patient consent for the chart audit was granted under the Ontario Personal Health Information Protection Act from each Ethics Review Board.
Design and participants
A retrospective, cluster, matched-control prechart and postchart audit was conducted for this component of the evaluation. PHC teams were randomly selected from a sampling frame based on the proportional distribution of participating teams: (1) model of care (academic, FHT and CHC); (2) wave of LC; (3) geographical region (Local Health Integration Network (LHIN) boundaries); and (4) practice setting (rural/urban). LHINS are geographical areas responsible for the regional administration of public healthcare services in the province of Ontario, Canada. Created 1 April 2007, they are mandated with planning, integrating, and distributing provincial funding as well as engaging with their local communities.13
One randomly selected physician per team (N=34) and matched control (N=34) were recruited. Control physicians were identified based on the sampling framework of their matched QIIP-LC physician using a pragmatic priority approach selecting controls. When identifying controls matched to a QIIP-LC physician within an FHT, ideally controls were ranked according to distance (prioritizing control physicians geographically further away to reduce contamination). For CHCs, controls were identified from alternate CHC practices, as most CHCs in Ontario operate in one practice location. In the event that an appropriate control physician did not exist, physicians from alternate FHT practices or community practices were recruited.
Physicians were included in the evaluation if they were in active clinical practice at least 1 year prior to the program start and had a minimum of 20 patients with T2DM in their practice. Physicians meeting eligibility criteria generated a list of patients with T2DM using ICD-9 250 billing code (International Classification of Diseases, Ninth Revision). Deceased patients and those residing in a nursing homes were excluded.
External, trained auditors were assigned to a physician. Auditors were not informed whether physicians were part of the QIIP-LC program or control group. Standard data abstraction forms were used. Auditors randomly selected patients from the list and screened for eligibility. Eligible patients were: (1) ≥18 years of age; (2) diagnosed with T2DM; and (3) diagnosed prior to the start of the audit timeframe. Eligible charts were reviewed for prespecified T2DM clinical process and outcome data 12 months prior to the LC (baseline), during the LC (intervention range: 15–17.5 months), and 12 months after the LC (post). Auditors were instructed to review charts until 12 eligible patients per physician were reached. Not all physicians had 12 eligible patients to contribute; therefore, the first dataset totaling 809 patients was used for analysis on all diabetes outcome measures. For analysis of glycemic outcomes, a second dataset was constructed of patients above study target A1c (≥7.3%). Not all physicians had five patients (from the original 12) above study target A1c; therefore, auditing continued until a minimum of 5 at baseline was met. The second dataset had a total of 310 patients. Data collection dates for control physicians were based on the time frame of their matched QIIP physician.
Primary outcome measures
Two primary outcome measures were calculated to evaluate the effectiveness of the QIIP-LC program on T2DM management. The primary clinical outcome was the A1c of patients above study target, while the primary process outcome was the proportion of patients with an annual foot exam. Investigators defined A1c to be above target for this study as 7.3% rather than the Canadian Diabetes Association Clinical Practice Guideline (CPG) of 7%14 in order to identify patients in whom glycemic treatment intensification was more likely.
Secondary outcome measures
Secondary clinical measures included glycemic outcomes and management (A1c value, A1c at CPG target, oral hypoglycemic agent (OHA)/insulin medication prescription), hypertension outcomes and management (systolic/diastolic blood pressure (BP) values, BP at CPG target and antihypertensive medication prescription) and lipid outcomes and management (low-density lipoprotein (LDL) value, LDL at CPG target and statin medication prescription). Intensification of glycemic, hypertension and lipid management were also included as secondary clinical outcome measures. Intensified glycemic management was characterized by adding an OHA, increasing the total daily dose of an OHA, adding insulin and/or increasing the total daily dose of an insulin. Intensified hypertension management was characterized by adding an antihypertensive and/or increasing the total daily dose of an antihypertensive. Intensified lipid management was characterized by adding a statin, increasing the total daily dose of a statin and/or switching statin medication to atorvastatin or rosuvastatin (higher potency statins).
Secondary process measures were the completion rate of screening for other diabetes-related complications (BP, lipid profile, albumin:creatinine ratio, glomerular filtration rate, serum creatinine, foot exam, eye exam, peripheral neuropathy exam, ECG exam, waist circumference documented and body mass index (BMI) documented). Visits to the PHC team were documented by date and provider type to determine the total number of visits to all HCPs. Documentation of diabetes counseling and education (exercise, weight, diet, smoking cessation, hypoglycemic events and adjustment of treatment plans) and self-management goals were collected by date. Specialist referrals were documented by date and type.
Sample size
Sample size calculations were conducted taking into account the sample size requirements for each primary outcome and adjusting for clustering and loss to follow-up. The final number of physicians required per group was 33.11
Analysis
Analysis was performed on all outcomes using SAS V.9.2.15 Three time periods were constructed for each outcome variable: (1) 12 months prior to the LC (baseline); (2) during the LC (during); and (3) 12 months after the LC (post). The generalized linear model (Proc Genmod in SAS) was used to compare change in outcome measures over time (baseline–during–post) between the QIIP-LC physicians (hereafter reported as QIIP group) and the control group, accounting for clustering within the physician’s practice and controlling for baseline measures.
Process/dichotomous outcome variables (eg, foot exam) were considered complete if documented in patient charts at least once during each time period. For continuous outcome variables (eg, A1c value), the most recent documented value during each time period was used. Missing values were populated with data carried forward from the previous time period. Other continuous outcome variables such as the number of visits and number of HCPs seen were calculated from the data set.
For each time period, the most recently prescribed medications and the total daily dose for each medication was used to construct treatment intensification variables by medication category (oral antihyperglycemic agents, insulin, antihypertensives and statins). Treatment intensification was determined by comparing baseline medication to the medication regimen during the LC or 12 months post-LC.
Results
Baseline and demographic characteristics of physicians and their patients are provided in table 1. Patient demographics were comparatively equal between groups, with the exception of mean age at diagnosis for the subset of patients with A1c ≥7.3% at baseline (mean age QIIP=54.7; control=50.4 years, p=0.01).
Demographics of physicians and their patients with type 2 diabetes mellitus
Comorbidities and diabetes-related complications of patients are presented in table 2. Most patients were diagnosed with at least one comorbidity (QIIP=95.8%; control=95.8%). More patients in the QIIP group were diagnosed with dyslipidemia (p=0.03), with no other significant differences found. QIIP group patients above study target A1c were more likely to be diagnosed with dyslipidemia (p=0.03) and depression (p=0.004). Statistically significant baseline and demographic characteristics were included as covariates during the analyses, with no significant effect on outcomes.
Comorbidities and complications of patients with type 2 diabetes mellitus
Primary outcome results
Mean A1c was significantly lower in the QIIP group during the 12-month program (p=0.01); however, these improvements were not sustained post-LC (p=0.10, table 3). No difference between groups was observed for foot exams (p=0.15, table 4).
Glycemic, hypertension and lipid outcomes and management
Screening for diabetes-related complications (QIIP N=406; control N=403)
Secondary outcome results
No significant changes in other glycemic outcomes and management were observed between groups (in the subset of patients above study target (A1c ≥7.3%)) (table 3). Similarly, no changes in vascular protection from baseline were observed for hypertension outcomes and management (BP management and medication, table 3), while lipid outcomes and management highlighted that more patients in the QIIP group achieving CPG target LDL cholesterol (≤2.0) over time compared with the control group (p=0.03). No other significant differences in lipid management between groups were found. Table 4 presents the screening for diabetes-related complications. Lipid profile testing (p=0.02), eye examinations (p=0.03), peripheral neuropathy exams (p=0.01), and documented BMI (p=0.04) were significantly higher for the QIIP group. Patients in the QIIP group had significantly more visits to HCPs from baseline compared with the control group (p=0.001, table 5). There were no differences between groups for the number of HCP seen (p=0.39) or the number of patients who saw more than one HCP per visit on average (p=0.89) (an indicator of team-based patient care). QIIP group referrals to a specialist decreased from 31.5% (pre) to 28.8% (post), while control group referrals increased slightly from 28.5% (pre) to 30.0% (post); however, these differences were not statistically significant, p=0.08. The proportion of patients who had documented counseling/education for diabetes (weight, diet, exercise, hypoglycemic events, treatment plan, smoking cessation, or self-management goals) over the study period was examined, with no significant differences between groups (p=0.43).
Patient visits to the primary healthcare team, all patients (QIIP N=406; control N=403)
Discussion
The rigorous design and methodology of the QIIP-LC program evaluation using a stratified random selection of participants and a control group is the most significant and promising effort to date demonstrating the impact of a QI program in PHC. The chart audit component of this evaluation established the success of the QIIP-LC program at significantly increasing a number of important diabetes process measures in the QIIP group, including lipid profile testing, eye examinations, peripheral neuropathy exams and BMI measurement. Furthermore, more patients in the QIIP group achieved CPG target LDL cholesterol (≤2.0) over time compared with the control group. No statistical differences in key diabetes outcome measures were noted, including A1c, LDL or BP values. It also appears there were no significant differences in how medications for these conditions were prescribed, specialist referrals, or chart reported diabetes counseling, education or self-management goals, even though patients of QIIP-LC participants had significantly more HCP visits.
Our findings are consistent with previous QI evaluation literature demonstrating improvements in diabetes clinical process measures, while the impact on important diabetes outcomes remains to be seen. Some recent studies showed no improvement in A1c, LDL, and BP, while other results vary depending on the length of the postperiod.16–23 Similarly, improvement in the intensification of diabetes medications has been demonstrated elsewhere; however, no improvements were observed in our evaluation.16 18–20 24
It is important to note that patients from both QIIP and control physicians were well-controlled for BP and lipid management in comparison with recent findings from the national Diabetes Mellitus Status in Canada survey.25 This survey reported significant treatment gaps in global vascular protection, with only 36% of patients with BP at CPG target (≤130/80/80 mm Hg) and 57% of patients with LDL cholesterol at CPG target (≤2.0 mmol/L). QIIP and control physicians had more patients at BP target post-LC (52.8% and 46.6%, respectively), with more QIIP patients at LDL target post-LC (59.2%). The relatively well-controlled patient population for both QIIP and control physicians potentially explains the lack of effect of the QIIP-LC on BP and lipid management simply for the reason that physicians experienced a ceiling effect, with the majority of their patients already in good control. These findings are supported by recent findings in Ontario26 and the USA27 demonstrating a significant reduction in cardiovascular disease in patients with diabetes. Similarly, trends in decreasing mortality rates for patients with diabetes have also been noted in Ontario, Canada, and the UK.28 Reducing vascular risk for patients with diabetes is critical, especially when the aim is to reduce the overall risk of either a primary or secondary cardiovascular event. In this regard, the findings of this evaluation are a positive step in the right direction for patients with diabetes.
An interesting finding from this evaluation was the significantly higher number of HCP visits in the QIIP group. On one hand, this may suggest that efforts to use a team-based approach to care may have had some success, a finding consistent with the perceptions of some QICs and PHC participants; on the other hand, one could also speculate that an increase in healthcare utilization is resource intensive without demonstrating significant clinical improvements. Future research is necessary to gain a more nuanced understanding of healthcare utilization, including PHC team formation, approach and care delivery. Numerous authors have posited about the effectiveness of FHTs and interprofessional team-based care in Ontario for patients with diabetes and chronic disease,29–32 suggesting that more time may be required for teams to form and learn to work together, and thus see significant differences in care such as diabetes clinical outcome measures.29–31 A longitudinal research approach, with more time between team formation, QI initiatives and evaluation, may be beneficial for understanding the true impact of team-based care and QI programs like QIIP.
Strengths and limitations
The QIIP-LC program evaluation was designed to address the growing consensus in the literature for more rigorous study designs of QI interventions.33–38 This evaluation expands the literature on QI evaluation methodology with its use of a control group, a rare study design in research and evaluation of QI programs. To maximize the effectiveness of the control and minimize the risk of contamination, control physicians were recruited using a pragmatic priority approach selecting controls ranked according to distance. Furthermore, collection of clinical chart audit data by independent audit rather than self-report heightened the strength of the findings, and the stratified random selection of participants, pre–post, multimeasure design added strength to this rigorous methodology. QIIP PHC teams were challenged to focus on more than one QI area at a time in a busy clinic setting,11 and it is possible that diabetes was not the focus of teams assessed by chart audit for T2DM outcomes. QIIP had a number of program targets (T2DM, CRC screening, advanced access and team functioning), and it would be beneficial for future evaluations to have the capacity to assess the degree to which participants focused on each target area of the intervention.
Complex QI initiatives are challenging to evaluate, and the QIIP-LC program was no exception. The retrospective nature of the evaluation may have limited the controlled design, and use of the last observation carried forward technique to impute missing values in the dataset may have resulted in a biased estimate of the intervention, or underestimated the variability of the results. Future analyses would benefit from statistical techniques designed to understand the magnitude of this bias and complement existing methods for dealing with missing data. Furthermore, participation bias existed on two levels. Clinical teams volunteered to participate in the program suggesting their interest in improving their practice. This bias was amplified by virtue of informed consent to participate in the evaluation. Participants who refused may have done so based on their lack of interest in QI interventions or their perceived lack of change or improvements. The impact of the QIIP-LC program may have been diluted by a number of simultaneous Ontario MOHLTC provincial initiatives and mandates targeting improvements in diabetes management in primary care.39–42 Design and implementation of QI programs would benefit from complimentary, rather than overlapping initiatives to optimize the impact of these programs. Finally, physicians and their respective teams were encouraged through the QIIP-LC to develop sustainability plans to integrate successful concepts into the practice and to share lessons learnt in the program to colleagues both within and external to the collaborative. Sustainability and spread of lessons learnt were key program components of the QIIP-LC program, and improvements in both QIIP and control groups could be an indication that changes initiated by QIIP-LC participants were integrated into the whole practice organization and thus, control practices within the same FHT may have adopted these new learnings.
Conclusion
The QIIP-LC program evaluation, including stratified random selection of participants and inclusion of control groups, is one of the most rigorous and promising efforts to date evaluating the impact of a QI program in PHC. While QIIP improved some diabetes process measures, no improvements in clinical outcome measures were noted. With resources in PHC already strained by the rising economic and public health burden of diabetes, this study highlights the importance of formalized evaluation of QI initiatives to provide an evidence base to inform future program planning and scale-up.
Acknowledgments
We would like to take this opportunity to acknowledge all the primary care teams for their voluntary participation in this detailed evaluation study and especially the support received from Susan Wheeler, Director, Knowledge Transfer and Exchange, Health Quality Ontario. Additionally, we acknowledge the cooperation of the Quality Improvement & Innovation Partnership Learning Collaborative leadership team, including Dr Brian Hutchinson, Brenda Fraser, and Dr Nick Kates.
References
Footnotes
Contributors SBH, MG, RB, JBB, GR, SW-B, JP-W, AT, SR, JK and MS contributed to the conception and design of the study. SW-B, MF, SBH, RB, JK, HH, GR, AT, MS and MG researched the diabetes chart audit data. SBH, SW-B, MF, SM, MG, and AT analysed the data. SBH, RB, MG, SW-B, MF, SM, SR and JWT interpreted the data. SR and JWT drafted the manuscript. All authors contributed to intellectual discussion and reviewed and edited the manuscript. SBH and SR are the guarantors of this study.
Funding Funding for this evaluation was provided through a contract agreement between Health Quality Ontario and Western/Queen’s University.
Competing interests None declared.
Patient consent A waiver of patient consent for the chart audit was granted under the Ontario Personal Health Information Protection Act (PHIPA) from the Ethics Review Board at Western University London Ontario Canada and Queen's University, Kingston Ontario, Canada.
Ethics approval Ethics approval for the evaluation was obtained from Western University and Queen’s University. A waiver of patient consent for the chart audit was granted under the Ontario Personal Health Information Protection Act (PHIPA) from each Ethics Review Board.
Provenance and peer review Not commissioned; externally peer reviewed.