Article Text
PDF
Abstract
Introduction Despite the widespread availability of clinical guidelines, considerable gaps remain between the care that is recommended (appropriate care) and the care provided. This protocol describes a research methodology to develop clinical indicators for appropriate care for common paediatric conditions.
Methods and analysis We will identify conditions amenable to population-level appropriateness of care research and develop clinical indicators for each condition. Candidate conditions have been identified from published research; burden of disease, prevalence and frequency of presentation data; and quality of care priority lists. Clinical indicators will be developed through searches of national and international guidelines, and formatted with explicit criteria for inclusion, exclusion, time frame and setting. Experts will review the indicators using a wiki-based approach and modified Delphi process. A formative evaluation of the wiki process will be undertaken.
Ethics and dissemination Human Research Ethics Committee approvals have been received from Sydney Children's Hospital Network, Children's Health Queensland Hospital and Health Service, and the Women's and Children's Health Network (South Australia). Applications are under review with Macquarie University and the Royal Australian College of General Practitioners. We will submit the results of the study to relevant journals and offer national and international presentations.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
Using and evaluating a novel method for ratifying indicators of ‘appropriate care’ for 20 paediatric conditions.
Achieving consensus on clinical indicators of ‘appropriate care’ that may be used for point-of-care decision-making and benchmarking purposes.
The recruitment of experts for the review process may introduce selection biases.
Introduction
Australian paediatricians commonly see children with a diverse range of sometimes complex health conditions.1 Clinical practice guidelines (CPGs) are available to help healthcare providers deliver appropriate care (care in line with evidence-based or consensus-based guidelines).2–4 However, it is not always easy for healthcare providers to navigate their way through CPGs due to factors such as: lack of timely access, multiple CPG sources and hence a lack of consensus, and lengthy recommendations that may not be specific or practical for point-of-care decision-making.5–10 Definitions of objective or measurable compliance with processes and outcomes are often lacking.5–10
Research was undertaken in the USA between 1998 and 2000 to develop recommendations for a range of paediatric conditions, and to benchmark the quality of ambulatory care against these recommendations.11 However, no such study has been conducted in Australia or elsewhere. The overall objective of CareTrack Kids (CTK) is to determine the appropriateness and safety of healthcare for common conditions delivered to children in Australia. In order to achieve this, a set of measurable clinical indicators is required.5
The CTK project involves a suite of three separate but related studies: part 1 (this study)—developing a set of clinical ‘appropriateness’ indicators for common paediatric conditions; part 2—measuring the appropriateness of paediatric care in Australia against these clinical indicators (using an onsite retrospective review of medical records during 2012 and 2013);12 and part 3—collecting information regarding the prevalence and characteristics of adverse events in paediatric healthcare encounters during 2012 and 2013.13
This protocol describes the clinical indicator development process for part 1 of CTK. It extends the methods of two studies from the USA11 ,14 and the CareTrack Australia (CTA) study15 ,16 using a collaborative online approach (a ‘wiki’). The outcomes are consensus on a set of clinical indicators for appropriate care for a range of common paediatric conditions in Australia during 2012 and 2013, and an evaluation of the clinical indicator development process.
Methods and analysis
There are three key components for this study protocol: to identify candidate paediatric conditions, to develop clinical indicators for these conditions and to evaluate the wiki methodology. The methods for the other CTK studies will be presented in separate protocol papers.12 ,13
Component 1: identify candidate paediatric conditions
A scoping exercise was undertaken to identify common candidate conditions from published research,1 ,17 ,18 burden of disease (BOD),17 prevalence (Children's and General Hospital 2011–12 New South Wales. SNOMED-CT Emergency departments, personal correspondence; Bettering the Evaluation and Care of Health. Top 100 most frequently managed problems 2011–12 weighted data: Age 0–17 years, personal correspondence, 2012),19 frequency of presentation (Children's and General Hospital 2011–12 New South Wales, personal correspondence)19 and quality of care priority lists (table 1).20 Using these data, the CTK research team identified an initial list of 28 conditions common across a range of healthcare practice facilities, including primary care provided by general practitioners, secondary care provided by outpatient paediatricians and tertiary care in hospitals.
Prevalent paediatric conditions managed in Australian health facilities 2011–2012
The CTK research team further refined the list of candidate conditions by evaluating them against three criteria to determine their eligibility for inclusion: high prevalence (Bettering the Evaluation and Care of Health, personal correspondence),1 high clinical impact20 and BOD.17 Two additional conditions were recognised by the research team as having significant clinical importance and were therefore recommended for inclusion: obesity (high prevalence, impact and BOD)20–22 and urinary tract infection (high impact).23 ,24 Using these criteria, a list of 20 paediatric conditions (table 2) has been selected for inclusion.
List of paediatric conditions and the key factors for their inclusion
Component 2: develop clinical indicators
Clinical indicators for this study will be developed using a three-stage process: (1) search and source relevant CPGs; (2) select, draft and format proposed clinical indicators; and (3) subject the indicators to several rounds of internal and external review using a modified Delphi approach.
Stage 1: search and source relevant CPGs
All clinical indicators will be derived from published CPGs relevant for the years 2012 and 2013. A systematic search will be undertaken sequentially in order of priority for national-level CPGs from Australia (eg, for the National Health and Medical Research Council (NHMRC)),2 and internationally (eg, for the National Institute for Health and Care Excellence (NICE) in England,3 Scottish Intercollegiate Guidelines Network (SIGN)4 and the Agency for Healthcare Research and Quality's (AHRQ's) National Guideline Clearinghouse in the USA).25 In the absence of Australian national and international CPGs, the CPGs of relevant professional medical colleges and associations will also be searched, as well as state or professional level (eg, the New South Wales (NSW) Clinical Practice Guidelines for Paediatrics).26 Three research team members (LKM, TDH and PDH) will be primarily responsible for conducting the CPG searches and developing the proposed clinical indicators. Full details of the search strategy are provided in online supplementary appendix A.27 ,28
Stage 2: select, draft and format proposed indicators
Recommendations from each CPG will be collated and used to inform the content and format of the proposed clinical indicators. For the purposes of this study, a clinical indicator is defined as a “measurable component of a standard or guideline, with explicit criteria for inclusion, exclusion, time frame and setting”.5 Not all recommendations published in CPGs will become indicators for the CTK study. Recommendations will be excluded based on the following criteria (table 3):
Strength of the wording of the recommendation (ie, ‘may’ and ‘could’ statements would be excluded; ‘should’ and ‘must’ statements would be included);
Low likelihood of information being documented in the medical record;
Guiding statements without recommended actions.
Example recommendations mapped against indicator eligibility criteria
All clinical indicators will be written in a structured and standardised format (ie, starting with the inclusion criteria followed by the action). For example, the inclusion criteria will define the age (infant, child, adolescent), condition and the phase of care (at diagnosis/presentation or ‘with’, indicating the diagnosis is existing). Indicators will be arranged chronologically according to phases of care (ie, screening, diagnosis, treatment, ongoing management). Table 4 provides examples.
Examples of possible condition indicators
Stage 3: subject the indicators to several rounds of internal and external review
There are two stages involved in the indicator review. The proposed clinical indicators will initially undergo an internal review (stage 3a), followed by an external wiki-style review using a modified Delphi process (stage 3b).32 This approach has been chosen to enhance methodological rigour and optimise the content and face validity of the final set of clinical indicators.
Stage 3a: internal review processes
Internal reviews will be conducted by paediatricians and general practitioners sourced from within the CTK research team and their professional networks. It is envisaged that these reviewers will be employed as the head or director of a relevant paediatric department in a large hospital, hold at least an adjunct academic appointment or be directly involved in clinical care (eg, clinical psychologist). The clinical indicators for each condition will be reviewed by a panel of at least three different reviewers who, depending on their self-reported scope of practice and expertise, are able to participate in review panels for more than one condition. The total number of invitations will therefore depend on the skill-mix of invited reviewers and overall recruitment rate. This selection strategy is well supported within the Delphi process literature.33 ,34
The internal review will consist of three rounds. In the first round, drafts of proposed clinical indicators for each condition, and the recommendations on which they are based, will be sent via email to a panel of at least three internal reviewers. The review criteria to be used are based on the methods from the previous US and Australian studies.11 ,14 ,15 Internal reviewers will be asked to: score each indicator against three key criteria: acceptability, feasibility and impact (online supplementary appendix B); to recommend the indicators for inclusion or exclusion; and provide any additional comments. In this round, reviewers will complete their assignments independently from one another to minimise bias from ‘group-think’.35 ,36 Three team members (LKM, TDH and PDH) will collate the feedback and revise the content, structure and format of each indicator. The refined set of indicators (including the original indicator and de-identified feedback) will be sent to the same internal reviewers for a second round of scoring. The same approach will be used in this round, with a request for more information and identification of indicators to be included or excluded. In the final round, the proposed set of indicators will be sent to one of the three existing reviewers to review and approve the indicators for the external wiki-based review process.
Stage 3b: external wiki-based review
External reviews will be conducted by invited paediatricians and general practitioners. Relevant medical colleges, professional associations and networks, such as the Royal Australasian College of Physicians, Australian Paediatric Research Network (APRN), Children's Healthcare Australasia (CHA), Australian Research Alliance for Children and Youth (ARACY), Bettering the Evaluation and Care of Health (BEACH) programme, New South Wales (NSW) Kids and Families, Sydney Children's Hospitals Network, Children's Health Queensland (QLD), Women's and Children's Health Network (South Australia), will be contacted, requesting assistance with the recruitment of clinical experts to register as external reviewers. Invitations will comprise email notifications to members, media releases and articles within newsletters. Clinical experts will self-nominate as reviewers for one or more of the CTK conditions based on their interest, scope of practice and clinical experience.33 ,34
The external review will involve a wiki-based process whereby indicators for each condition (from round 3 of the internal review) will be posted to an online wiki site. The wiki ‘live’ time will depend on the recruitment rate of experts and the progress of their reviews, but is anticipated to be no longer than 3 months. The aim is for each condition to be independently reviewed by a minimum of nine experts. In addition to the scoring criteria used in the internal review process, indicators will be scored on a nine-point Likert scale as representative of appropriate care delivered to children during 2012 and 2013.11 ,15 ,37 A clinical champion for each condition will follow-up and manage external reviewers’ responses, and to make final recommendations regarding the inclusion, content, structure and format of indicators. For most conditions, this role will be undertaken by one of the stage 3a internal reviewers as previously identified by the CTK research team. The final sets of clinical indicators for care during 2012 and 2013 will be considered representative of appropriate care for Australian children for the candidate conditions and will form the suite of indicators for the second (part 2) CTK study.12 These indicators will also be useful for other purposes, such as providing healthcare practitioners with succinct and measurable compliance points to assist with point-of-care decision-making.
Component 3: evaluate the ‘wiki’ methodology
A multimethods evaluation of the wiki's methodology will be undertaken, with the aim of assessing utilisation, accessibility and ease of use.38 ,39 Three data sources will be used to inform the evaluation: (A) utilisation statistics sourced from the wiki logs—these will include demographics of users and rates and times of use; (B) the nature and content of reviewers’ and clinical champion's comments (eg, the format and rationale of proposed changes to indicators, level of agreement between reviewers, and resulting changes to the indicators); and (C) user perspectives. Following completion of their clinical indicator reviews, external reviewers will be invited to participate in an online survey of their experiences and perspectives. Both quantitative and qualitative analyses of these three data sources will be undertaken including: descriptive statistics for the characteristics of wiki users, their patterns of use and scores for clinical indicators; and frequency counts and content analyses of ratings and free-text responses from the user perspectives survey. Using these data, recommendations will be developed regarding the overall feasibility of the wiki process for future indicator developments and potential changes for improvement.
Ethics and dissemination
Ethical approval
Applications are under review with the Royal Australian College of General Practitioners.
To minimise the risk of bias, all reviewers will be required to complete a Conflict of Interest (COI) declaration. The COI declarations will be recorded for each reviewer and managed according to the NHMRC protocol for conflict.40
Dissemination
We will submit the results of the study to relevant national and international journals with the intention of publishing the results widely. As well, we will make national and international oral presentations to stakeholder groups including those involving patients, researchers, clinicians, managers and policymakers.
Discussion
We recognise several potential limitations to our study. The approaches used to identify candidate paediatric conditions and develop clinical indicators are specific to the Australian healthcare setting for the years 2012 and 2013. Findings can therefore not be generalised beyond these contexts. Experts will be invited to participate and self-nominate to review conditions that are within their scope of practice and for which they have current clinical experience. This may introduce a selection bias, as reviews will be representative of the participating sample rather than the general population of Australian paediatricians and general practitioners. There is potential for the final sets of clinical indicators to be a function of the scope of practice, clinical experience, level of research participation and degree of literacy with information technology of a subset of healthcare practitioners.
The CTK part 1 study will use and evaluate a novel method for ratifying indicators of ‘appropriate care’ for 20 paediatric conditions. These indicators will form the criteria against which the CTK part 2 study can, for the first time in Australia, measure appropriateness of paediatric care in 2012 and 2013. This study will help to establish the feasibility of using wiki-based technology for the purposes of developing clinical indicators. This information can be used to inform the design of future studies using Delphi processes to establish consensus on recommended healthcare, and will be relevant for national and international researchers, policymakers, healthcare practitioners and patients.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online appendix A
- Data supplement 2 - Online appendix B
Footnotes
Contributors JB and PDH initiated the project and led the National Health and Medical Research Council (NHMRC) grant proposal. JB, AJ, LKW, CTC and MFH are chief investigators on the project, led the design of the grant and shared in the development of the protocol and the initial drafting of the grant application and protocol. TDH, PDH, NM and LKW are project team members and did the first drafting of the protocol manuscript. WBR, SG, ARH, JGW, EM, AL, GW, HMW and CH are associate investigators or industry partners and helped write the grant proposal, protocol and manuscript.
Funding This work is supported by NHMRC Partnership Grant APP1065898. It is led by the Australian Institute of Health Innovation, Macquarie University. The partners in the research are BUPA Health Foundation, Sydney Children's Hospitals Network, NSW Kids and Families, Children's Health Queensland, the South Australian Department of Health, the University of South Australia (UNISA) and the NSW Clinical Excellence Commission.
Competing interests None.
Ethics approval Human Research Ethics Committee approvals have been received from Macquarie University, the Sydney Children's Hospital Network, Children's Health Queensland Hospital and Health Service and Women's and Children's Health Network (South Australia).
Provenance and peer review Not commissioned; externally peer reviewed.
Linked Articles
- Health services research
- Evidence based practice