Intended for healthcare professionals

  1. Research
  2. When and how to use...
  3. When and how to use data from randomised trials to develop or validate prognostic models
Research Methods & Reporting

When and how to use data from randomised trials to develop or validate prognostic models

BMJ 2019; 365 doi: https://doi.org/10.1136/bmj.l2154 (Published 29 May 2019) Cite this as: BMJ 2019;365:l2154
  1. Romin Pajouheshnia, postdoctoral research fellow1 2,
  2. Rolf H H Groenwold, professor3,
  3. Linda M Peelen, associate professor1,
  4. Johannes B Reitsma, associate professor1 4,
  5. Karel G M Moons, professor1 4
  1. 1Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, 3508 GA Utrecht, Netherlands
  2. 2Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Utrecht University, Utrecht, Netherlands
  3. 3Department of Clinical Epidemiology, Leiden University Medical Centre, Leiden, Netherlands
  4. 4Cochrane Netherlands, Utrecht, Netherlands
  1. Correspondence to: Romin Pajouheshnia R.Pajouheshnia{at}uu.nl
  • Accepted 26 March 2019

Prediction models have become an integral part of clinical practice, providing information for patients and clinicians and providing support for their shared decision making. The development and validation of prognostic prediction models requires substantial volumes of high quality information on relevant predictors and patient health outcomes. Primary data collection dedicated to prognostic model (development or validation) research could come with substantial time and costs and can be seen as a waste of resources if suitable data are already available. Randomised clinical trials are a source of high quality clinical data with a largely untapped potential for use in further research. This article addresses when and how data from a randomised clinical trial can be used additionally for prognostic model research, and provides guidance for researchers with access to trial data to evaluate the suitability of their data for the development and validation of prognostic prediction models.

Prognostic prediction models—or prognostic models—are used to provide probabilistic predictions of an individual’s prognosis, which can be used to support patient counselling and evidence based decision making in clinical practice, as well as research.1 The development and validation of these models requires substantial amounts of high quality patient and clinical data (information on the development and validation of prognostic models can be found elsewhere23). Although prospective data collection designed specifically to develop or validate a prognostic model is typically advocated,1 this is often not feasible or desirable due to the costs involved.

Randomised clinical trials (RCTs) provide a tempting alternative data source for the development and validation of prognostic models: in the year 2018, nearly 25 000 RCTs of treatment interventions were published, generating a large quantity of data (see supplement for the search query). Yet the valuable information gathered in RCTs remains largely untapped by the research community, and could be seen as a source of research waste. At the same time, despite the widespread belief that RCTs are the so-called gold standard for data generation, their suitability for answering questions of a descriptive (that is, predictive) nature has been questioned.145 This article starts from the perspective that we would like to develop or externally validate a prognostic model and we have access to individual participant data (referred to as “data” in this article) from a relevant phase III RCT. We present the opportunities that RCT data can offer, describe potential limitations that must be considered, and navigate the do’s and don’ts of developing or externally validating a prognostic model with RCT data.

Summary points

  • To minimise research waste, data from randomised clinical trials (RCTs) might be considered for the development or validation of a prognostic prediction model

  • Advantages of RCT data can include completeness, quality, detailed protocols, and broad informed consent

  • Randomised treatment allocation facilitates the development and validation of prognostic models that predict risk in the presence or absence of a particular treatment

  • RCT data might be less suitable because of selective patient or centre inclusion, extraneous trial effects, or overly specialised predictor measurement, which all could limit the generalisability and thus applicability of prognostic models to real life practice

  • Other limitations might be surrogate outcomes that are too short term or clinically irrelevant, or an insufficient sample size for prognostic model development or validation

  • This paper provides guidance to appraise the suitability of RCT data for prognostic model research by examining both potential benefits and limitations

Opportunities arising from RCT data use

So far, several prognostic models have been effectively developed and validated using RCT data (table 1). Data generated by an RCT can confer specific benefits over data from alternative sources, such as from predesigned observational studies, electronic health records, disease specific registers, or administrative medical databases. We outline the key opportunities that RCT data might provide when developing or externally validating a prognostic model.

Table 1

Examples of studies in which prognostic models have been developed or validated using data from randomised clinical trials (RCTs)

View this table:

Data quality: completeness

Missing data are a serious and almost ubiquitous issue for studies that develop or externally validate prognostic models.1819 To develop a prognostic model, one ideally needs complete information on all candidate predictors and outcomes, measured in all individuals in the study. External validation requires complete information on all predictors and outcomes of the model that is under evaluation. Information on predictor variables that are not routinely measured in practice could have limited functional value, as discussed later. But for variables that can be readily measured in practice, complete recording of measurements can reduce the risk of bias (due to selective “missingness” of data) and improve the certainty around model coefficients in a development study or measures of model performance in an external validation study. Although many methods can handle missing data, the best solution is undoubtedly its prevention.

The completeness of data from RCTs can be an important asset for prognostic model studies. Throughout the design and conduct of an RCT, several strategies can be used to help collect a complete set of information on predictors and outcomes in all trial participants.20 These methods might include the training of research staff before starting data collection, and incentives for data collectors to collect complete information. While these strategies might be a challenge in multicentre trials, a common shared protocol can be established to help maintain consistent data collection across trial centres. A unique feature of RCTs is detailed study monitoring, usually by several separate committees.21 Trial oversight committees, such as data monitoring committees, monitor the presence of missing data in a trial. These efforts work synergistically with central and onsite monitoring to keep track of missing data, which can help to identify and prevent additional missing data.

In addition, RCT data can include detailed information on important post-baseline events, which could affect the prognosis of participants. Such details are often not available for all patients in observational databases. Post-baseline events (such as changes in treatment, the use of rescue drug treatments, or competing outcome events) might need to be accounted for when developing or externally validating a prognostic model, and should be reported alongside the results.2223

Data quality: accuracy and consistency

Accurate and consistent predictor and outcome information is a requisite for accurate prognostic models. The accuracy of predictor measurements in a prognostic model study should reflect the accuracy of those measurements in clinical practice, as discussed later. However, concerns have been raised over the quality of the recording of information that is collected from patients in routine practice. RCTs are commonly regarded as a source of high quality health data. As with data completeness, considerable amounts of time and money are spent to ensure that data are correctly measured and recorded.

Firstly, adherence to the trial protocol and standard operating procedures facilitates the accurate and consistent measurement of predictors and outcomes, in particular for specific variables of interest in the trial (although this might not reflect actual variation in practice, as discussed later). Secondly, case report forms require the recording of detailed patient and clinical information and can help to prevent the recording of impossible values, forming a part of the quality assurance process in an RCT.24 Thirdly, as with data completeness, study monitoring in RCTs helps to maintain accuracy and consistency in the recorded data. For example, central monitoring includes the checking of data for unusual patterns or implausible values.21 In addition, source data verification and electronic data capture methods form an additional layer of data validation.25 Finally, a centralised system for the adjudication of outcome events can be especially important when outcome measurements are subjective. Altogether, these systems and processes can yield data that satisfy the quality requirements of prognostic model studies.

Protocol and records

A trial protocol provides information on the modality and timing of predictor and outcome measurements. Firstly, the protocol promotes the standardisation of measurements and the recording of any protocol deviations. As discussed earlier, this standardisation can improve the accuracy and consistency in the recording of measurements, but could lead to issues with the generalisability of a model if deviations in predictor measurement are flagged and corrected (as discussed later). In addition, the details recorded in a protocol might provide insight into the suitability of a predictor for inclusion in a prognostic model. For example, if the protocol states that a certain predictor should be measured at a time point that is not relevant to routine clinical practice, one might not select the predictor.

Secondly, knowledge of the operationalisation of predictor or outcome measurements provides insight into how well a model can perform in practice, and can inform the assessment of the risk of bias when reviewing a prognostic model study.26 In addition, information on how and when variables were collected and recorded might have predictive value. For example, the timing of measurements (eg, whether taken during the day or night) can be highly predictive of clinical outcomes.27

Treatments

Often in practice, healthcare providers are interested in asking: “What will happen to the patient if I do not treat them?” With this question, prognostic models can be used to support clinical decisions as well as provide information to healthcare providers and patients for counselling.1 For this purpose, prognostic models must predict risks for patients in the absence of a certain treatment—which can prove challenging in non-RCT data, because of the non-random use of treatments by patients,28 and because advanced statistical methods might be needed to correctly account for this.2930 In the case of RCT data, the effect of treatment use can be solved by simply developing or validating the prognostic model in the control trial arm (control treatment, untreated, or placebo treated) or by including the randomised treatment as a predictor in the model, along with terms for any other treatment-predictor interactions (model development only).24 However, the placebo arm of a placebo controlled trial might not represent truly untreated patients in usual practice (as discussed later), whereas the control arm of a randomised pragmatic or comparative effectiveness trial better reflects daily practice.

Threats to the viability of RCT data use

Available data from an RCT can have several limitations that might reduce the viability of its use for prognostic model development and validation. Below, we present key challenges when considering RCT data use for prognostic model research. Where necessary, the issues are discussed separately for model development and model validation.

Consent

Consent to reuse RCT data for prognostic modelling might not have been given by the trial participants. By contrast with data repositories established specifically for scientific research purposes (eg, UK Biobank31), which have very broad consent for data reuse,32 trials might not always have asked for a sufficiently broad consent. However, compared with routinely collected data (which have even greater consent challenges in light of the 2016 EU General Data Protection Regulation33), RCT data might prove more accessible, especially if trials begin to adopt broad consent for data reuse, as recommended.34 It is likely that researchers will need to consult their institutional review board before using RCT data for secondary analysis, but whether this satisfies ethical and legal requirements needs further examination.

Selective inclusion of centres

The centres that participate in RCTs might not be representative of medical practice in general.35 Specifically, generalisability of a prognostic model or the findings of a validation study might be limited if only specialist trial centres (eg, academic medical centres) or experienced clinicians with high performance ratings were invited to participate.36 In such cases, the associations between predictors and the outcome, and the incidence of outcomes could be different in the trial setting compared with routine clinical practice, of which both could affect the performance of a prognostic model.37

Selective eligibility and enrolment

RCTs commonly have narrow participant eligibility criteria, for example, often excluding patients who are frail, who have multimorbidity, or who are vulnerable.3839404142 At the same time, some of the most challenging clinical decisions are for these groups of patients. Thus, RCTs might not provide sufficient information for prognostic research in these clinically relevant patient subgroups. When developing or validating a prognostic model using a selective subset of patients, the predictor effects and functional forms of their associations with an outcome are assumed to be the same across the patient subsets included and excluded from the RCT. In addition, the participants invited to enter an RCT and those who actually enrol and remain in the trial until completion can substantially differ.43 For example, the requirement of informed consent from participants has been shown to result in differences between the patients enrolled and not enrolled in a trial.4445 As with selective eligibility, this challenge can limit the value of RCT data for prognostic model development; it might not be as problematic for external validation, but could limit the generalisability of results to broader patient populations.

Predictor measurement

As discussed earlier, the strengths of protocol driven data collection by trained research staff highlight that improved availability, accuracy, and consistency of clinical measurements can improve the viability of a prognostic model.46 With this opportunity, however, come challenging threats. For the purpose of prognostic prediction, the measurement of predictors should closely reflect how they are measured in regular clinical practice. Thus, the use of unrealistically accurate measurements—which could occur if specialist personal or equipment are used in an RCT—when developing a prognostic model could reduce the generalisability of the model to clinical practice,47 and the findings of a validation study might not represent how the model will truly perform in practice.48 In addition, it is essential that any predictors considered when developing a prognostic model are (or potentially will be) routinely measured in practice. Supplemental variables collected in an RCT should not be incorporated in a prognostic model if they will not be available in practice.

Extraneous trial effects

The effects of trial enrolment on participant behaviour have been documented extensively, which can vary greatly between trials.49 Knowledge of enrolment in a trial can lead to participants behaving differently, even reporting more optimistic outcomes,5051 an effect commonly termed the “Hawthorne effect.” The enrolment of a centre in an RCT might also affect the behaviour of healthcare professionals and as a result the prognosis of a patient enrolled in a trial might be better than if the patient had received routine care.52 In placebo controlled trials, patients on placebo do not reflect usual or current care, and might also show a placebo or nocebo effect, which could positively or negatively affect their outcomes.53 If data from a control arm with a strong placebo effect are used to develop a prognostic model to predict a subjective outcome (such as pain experience), the model might underestimate the outcome when applied in practice.

The protocol effect or care effect can arise when the adherence of centres to a strict trial protocol might improve patient outcomes (eg, through additional monitoring) compared with patients not enrolled in the trial.5455 The presence of these effects could hamper the generalisability of models developed or validated using RCT data to clinical practice, possibly due to close monitoring or specialised care being specified in the trial protocol, or because of a subconscious effect that trial participation has on care givers. In both cases, if participation in a trial results in better patient outcomes, a prognostic model developed using these data might underestimate risks when the model is applied in practice. Thus, a trial with strong extraneous effects might not provide suitable data for prognosis research.

Short term and surrogate outcomes

Long term, patient relevant outcomes are often of interest when making prognostic predictions in daily practice. For example, models to predict cardiovascular disease risk are commonly designed to predict outcomes within 10 years.56 Development and validation of such models require very long follow-up, which is rarely available in RCTs. However, unlike the validation of a model for predicting long term prognosis with short term outcome data (which is not advisable), there could still be medical use of a prognostic model developed with short term outcomes. In addition, RCTs often opt for surrogate endpoints to replace more costly long term outcomes.57 If a prognostic model is to be used to inform patients and healthcare professionals, surrogate endpoints could have insufficient clinical relevance if the surrogate is imperfectly correlated with the clinical outcome, whether used to develop or validate a prognostic model.

Sample size

Research on the development of prognostic models often requires substantial samples. While no consensus currently exists on the sample sizes required for prognostic prediction, the required sample size depends on several factors, including the number of predictors, total sample size, and number or proportion of events.5859 Thus, large sample sizes could be needed to reliably develop a prognostic model, especially when tens or hundreds of candidate predictor variables are considered. Similarly, reliable prognostic model validation requires data samples with a minimum of several hundreds of outcome events.60 Obviously, RCTs are not designed and powered with prognosis research in mind. Thus, the number of participants in a single phase III RCT might not be sufficient, and the problem worsens if smaller phase IIb RCTs are considered. Although approaches such as penalised regression can help to prevent the overfitting of prognostic models in small datasets,61 large samples may yet be needed for modern modelling techniques.62 Data from large, multicentre trials can, however, provide a solution to this issue. In addition, as seen in table 1, the combination of individual participant data from more than one RCT can greatly increase the amount of available data.

How and when to use trial data for prognostic prediction research

When data from an RCT are available, researchers must weigh the advantages (data quality, completeness, treated and untreated arms, and protocol) against any limitations, both described earlier. We suggest that the decision process be separated as follows:

  • Criteria that must be met: there must be acceptable patient consent (or under certain conditions a waiver by an institutional review board 63) for reuse of the RCT data for prognostic prediction research.

  • Criteria that could seriously limit the usefulness of the data: insufficient sample size or follow-up, or no availability of important predictors or outcomes will seriously limit the suitability of RCT data.

  • Criteria that could limit the usefulness of the data: selection of patients or centres, experimental effects, and predictor measurements highly driven by the protocol could all limit how representative the trial data are of the target population for the prognostic model.

To aid in this process, table 2 presents a series of questions to ask when assessing whether data from an RCT are suitable for developing or validating a prognostic model. For each situation, general advice is provided to help researchers reach a decision. The decision to use a given set of data from an RCT will depend on the specific research question and remains largely subjective. In addition, to help gain an overall picture of the suitability of an RCT as a whole, researchers can benefit from constructing a diagram, such as in figure 1. In this fictional example, consent for secondary use of the data was available and the data received a high “score” for this criterion, after which the remaining criteria were assessed. From this, we see that the dataset might be a good candidate for a validation study, but centre and participant selection could limit the generalisability of prognostic models developed using the data. With such a picture, researchers can decide whether the benefits of using the RCT data outweigh the limitations. Finally, if a decision is made to use the RCT data, this information can be used when reporting any study limitations.

Table 2

How to assess whether data from a randomised clinical trial is suitable for developing or validating a prognostic model

View this table:
Fig 1
Fig 1

Graphical example of how suitability of data from a randomised clinical trial can be assessed after consent for use has been confirmed. Assuming consent for use of the data are obtained, researchers might attribute a level of confidence that the RCT does have additional threats (as described earlier). In this example, the dark purple boundaries converge towards the ends of four axes, but limited sample size and representativeness of the participants in the RCT might force researchers to question its suitability for prognostic modelling

Use of data from more than one study

As seen in table 1, multiple RCTs can be combined when developing or validating a prognostic model, which has the clear advantage of increasing the number of patients and outcomes in the analysis, and can provide an opportunity to assess the impact of differences in definitions and measurements on model performance. The IMPACT model,6 for example, combined data from both RCTs and observational studies. Datasets ranged in size from 139 to 1574 patients (8509 in total), providing much more information than any single study. A cross validation procedure was performed to assess performance of the model across the studies, giving more insight into the robustness of the findings. How data from RCTs and observational studies should be best combined for prognostic model studies requires further research. For now, we suggest that researchers use figure 1 to assess and compare the suitability of multiple RCTs, and that readers should refer to existing guidance on individual patient data meta-analysis for prognostic modelling studies.64

Conclusion

When data from an RCT are available for the secondary purpose of developing or validating a prognostic model, the opportunities and limitations of these data require careful consideration. Available data from RCTs can, if used appropriately, be a viable substitute for costly and labour intensive data collection for prognostic prediction research. By recognising the opportunities that RCT data offer and carefully appraising available data, we can maximise the chance of using data that allow high quality prognostic model research, while avoiding unnecessary, costly primary data collection.

Inevitably, fundamental challenges remain that are universal to the secondary use of data for research, such as the systematic absence of data on certain key predictors. In these circumstances, researchers might consider designing a dedicated study to collect data to develop or externally validate a prognostic model. Alternatively, they could consider greater integration of prognosis research questions during the design of clinical trials, which could help to overcome barriers such as consent. We hope that researchers will cautiously seize the opportunities that data generated by RCTs provide, to improve both the quality and efficiency of future prognostic prediction research.

Acknowledgments

We thank Rieke van der Graaf for providing insight into ethics and consent for trial data reuse.

Footnotes

  • Contributors: KGMM conceived the paper objectives. The manuscript was first drafted by RP, which was subsequently reviewed several times by all authors. All authors contributed to the design of the study, manuscript content, and the writing of the manuscript. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

  • Funding: There was no specific funding for this paper.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for the submitted work; RG receives funding from the Netherlands Organisation for Scientific Research (project 917.16.430); JBR is supported by a TOP grant from the Netherlands Organisation for Health Research and Development (ZonMw) entitled “Promoting tailored healthcare: improving methods to investigate subgroup effects in treatment response when having multiple individual participant datasets” (grant 91215058); KGMM receives funding from the Netherlands Organisation for Scientific Research (project 9120.8004 and 918.10.615); no other relationships or activities that could appear to have influenced the submitted work.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

References

  1. ↵
    1. Moons KG,
    2. Royston P,
    3. Vergouwe Y,
    4. Grobbee DE,
    5. Altman DG
    . Prognosis and prognostic research: what, why, and how?BMJ2009;338:b375. doi:10.1136/bmj.b375 pmid:19237405
    OpenUrlFREE Full Text
  2. ↵
    1. Steyerberg EW,
    2. Moons KG,
    3. van der Windt DA,
    4. et al.,
    5. PROGRESS Group
    . Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med2013;10:e1001381. doi:10.1371/journal.pmed.1001381 pmid:23393430
    OpenUrlCrossRefPubMed
  3. ↵
    1. Altman DG,
    2. Vergouwe Y,
    3. Royston P,
    4. Moons KG
    . Prognosis and prognostic research: validating a prognostic model. BMJ2009;338:b605. doi:10.1136/bmj.b605 pmid:19477892
    OpenUrlFREE Full Text
  4. ↵
    1. Vandenbroucke JP
    . Observational research, randomised trials, and two views of medical science. PLoS Med2008;5:e67. doi:10.1371/journal.pmed.0050067 pmid:18336067
    OpenUrlCrossRefPubMed
  5. ↵
    1. Iorio A,
    2. Spencer FA,
    3. Falavigna M,
    4. et al
    . Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients. BMJ2015;350:h870. doi:10.1136/bmj.h870 pmid:25775931
    OpenUrlFREE Full Text
  6. ↵
    1. Steyerberg EW,
    2. Mushkudiani N,
    3. Perel P,
    4. et al
    . Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Med2008;5:e165, discussion e165. doi:10.1371/journal.pmed.0050165 pmid:18684008
    OpenUrlCrossRefPubMed
    1. Antman EM,
    2. Cohen M,
    3. Bernink PJ,
    4. et al
    . The TIMI risk score for unstable angina/non-ST elevation MI: A method for prognostication and therapeutic decision making. JAMA2000;284:835-42. doi:10.1001/jama.284.7.835 pmid:10938172
    OpenUrlCrossRefPubMedWeb of Science
    1. Sylvester RJ,
    2. van der Meijden AP,
    3. Oosterlinck W,
    4. et al
    . Predicting recurrence and progression in individual patients with stage Ta T1 bladder cancer using EORTC risk tables: a combined analysis of 2596 patients from seven EORTC trials. Eur Urol2006;49:466-5, discussion 475-7. doi:10.1016/j.eururo.2005.12.031 pmid:16442208
    OpenUrlCrossRefPubMedWeb of Science
    1. Kengne AP,
    2. Patel A,
    3. Marre M,
    4. et al.,
    5. ADVANCE Collaborative Group
    . Contemporary model for cardiovascular risk prediction in people with type 2 diabetes. Eur J Cardiovasc Prev Rehabil2011;18:393-8. doi:10.1177/1741826710394270 pmid:21450612
    OpenUrlCrossRefPubMedWeb of Science
    1. Hilkens NA,
    2. Algra A,
    3. Diener HC,
    4. et al.,
    5. Cerebrovascular Antiplatelet Trialists’ Collaborative Group
    . Predicting major bleeding in patients with noncardioembolic stroke on antiplatelets: S2TOP-BLEED. Neurology2017;89:936-43. doi:10.1212/WNL.0000000000004289 pmid:28768848
    OpenUrlCrossRefPubMed
    1. Schuit E,
    2. Amer-Wahlin I,
    3. Groenwold RH,
    4. Mol BW,
    5. Moons KG,
    6. Kwee A
    . Prediction of neonatal metabolic acidosis in women with a singleton term pregnancy in cephalic presentation: an external validation study. Am J Perinatol2012;29:681-6. doi:10.1055/s-0032-1314888 pmid:22639352
    OpenUrlCrossRefPubMed
    1. Hoster E,
    2. Klapper W,
    3. Hermine O,
    4. et al
    . Confirmation of the mantle-cell lymphoma International Prognostic Index in randomized trials of the European Mantle-Cell Lymphoma Network. J Clin Oncol2014;32:1338-46. doi:10.1200/JCO.2013.52.2466 pmid:24687837
    OpenUrlAbstract/FREE Full Text
    1. Wessler BS,
    2. Ruthazer R,
    3. Udelson JE,
    4. et al
    . Regional validation and recalibration of clinical predictive models for patients with acute heart failure. J Am Heart Assoc2017;6:e006121. doi:10.1161/JAHA.117.006121 pmid:29151026
    OpenUrlAbstract/FREE Full Text
    1. Sotomi Y,
    2. Cavalcante R,
    3. van Klaveren D,
    4. et al
    . Individual long-term mortality prediction following either coronary stenting or bypass surgery in patients with multivessel and/or unprotected left main disease: an external validation of the SYNTAX score II model in the 1,480 patients of the BEST and PRECOMBAT randomized controlled trials. JACC Cardiovasc Interv2016;9:1564-72. doi:10.1016/j.jcin.2016.04.023 pmid:27491605
    OpenUrlCrossRefPubMed
    1. Takwoingi Y,
    2. Botello AP,
    3. Burr JM,
    4. et al.,
    5. Surveillance for Ocular Hypertension Study Group
    . External validation of the OHTS-EGPS model for predicting the 5-year risk of open-angle glaucoma in ocular hypertensives. Br J Ophthalmol2014;98:309-14. doi:10.1136/bjophthalmol-2013-303622 pmid:24357494
    OpenUrlAbstract/FREE Full Text
    1. Royston P,
    2. Moons KG,
    3. Altman DG,
    4. Vergouwe Y
    . Prognosis and prognostic research: Developing a prognostic model. BMJ2009;338:b604. doi:10.1136/bmj.b604 pmid:19336487
    OpenUrlFREE Full Text
    1. Collins GS,
    2. Reitsma JB,
    3. Altman DG,
    4. Moons KG
    . Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Med2015;13:1. doi:10.1186/s12916-014-0241-z pmid:25563062
    OpenUrlCrossRefPubMed
  18. ↵
    1. Collins GS,
    2. Mallett S,
    3. Omar O,
    4. Yu L-M
    . Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting. BMC Med2011;9:103. doi:10.1186/1741-7015-9-103 pmid:21902820
    OpenUrlCrossRefPubMed
  19. ↵
    1. Collins GS,
    2. de Groot JA,
    3. Dutton S,
    4. et al
    . External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol2014;14:40. doi:10.1186/1471-2288-14-40 pmid:24645774
    OpenUrlCrossRefPubMed
  20. ↵
    1. Little RJ,
    2. Cohen ML,
    3. Dickersin K,
    4. et al
    . The design and conduct of clinical trials to limit missing data. Stat Med2012;31:3433-43. doi:10.1002/sim.5519 pmid:22829439
    OpenUrlCrossRefPubMed
  21. ↵
    1. Baigent C,
    2. Harrell FE,
    3. Buyse M,
    4. Emberson JR,
    5. Altman DG
    . Ensuring trial validity by data quality assurance and diversification of monitoring methods. Clin Trials2008;5:49-55. doi:10.1177/1740774507087554 pmid:18283080
    OpenUrlCrossRefPubMedWeb of Science
  22. ↵
    1. Pajouheshnia R,
    2. Damen JA,
    3. Groenwold RH,
    4. Moons KG,
    5. Peelen LM
    . Treatment use in prognostic model research: a systematic review of cardiovascular prognostic studies. Diagnostic and Prognostic Research2017;1:15doi:10.1186/s41512-017-0015-0.
    OpenUrlCrossRef
  23. ↵
    1. Moons KG,
    2. Altman DG,
    3. Reitsma JB,
    4. et al
    . Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med2015;162:W1-73. doi:10.7326/M14-0698 pmid:25560730
    OpenUrlCrossRefPubMed
  24. ↵
    1. Bellary S,
    2. Krishnankutty B,
    3. Latha MS
    . Basics of case report form designing in clinical research. Perspect Clin Res2014;5:159-66. doi:10.4103/2229-3485.140555 pmid:25276625
    OpenUrlCrossRefPubMed
  25. ↵
    1. Nahm ML,
    2. Pieper CF,
    3. Cunningham MM
    . Quantifying data quality for clinical trials using electronic data capture. PLoS One2008;3:e3049. doi:10.1371/journal.pone.0003049 pmid:18725958
    OpenUrlCrossRefPubMed
  26. ↵
    1. Debray TPA,
    2. Damen JAAG,
    3. Snell KIE,
    4. et al
    . A guide to systematic review and meta-analysis of prediction model performance. BMJ2017;356:i6460. doi:10.1136/bmj.i6460 pmid:28057641
    OpenUrlFREE Full Text
  27. ↵
    1. Agniel D,
    2. Kohane IS,
    3. Weber GM
    . Biases in electronic health record data due to processes within the healthcare system: retrospective observational study [correction in: BMJ 2018;363:k441]. BMJ2018;361:k1479. doi:10.1136/bmj.k1479 pmid:29712648
    OpenUrlAbstract/FREE Full Text
  28. ↵
    1. Groenwold RH,
    2. Moons KG,
    3. Pajouheshnia R,
    4. et al
    . Explicit inclusion of treatment in prognostic modeling was recommended in observational and randomized settings. J Clin Epidemiol2016;78:90-100. doi:10.1016/j.jclinepi.2016.03.017 pmid:27045189
    OpenUrlCrossRefPubMed
  29. ↵
    1. Pajouheshnia R,
    2. Peelen LM,
    3. Moons KGM,
    4. Reitsma JB,
    5. Groenwold RHH
    . Accounting for treatment use when validating a prognostic model: a simulation study. BMC Med Res Methodol2017;17:103. doi:10.1186/s12874-017-0375-8 pmid:28709404
    OpenUrlCrossRefPubMed
  30. ↵
    1. Sperrin M,
    2. Martin GP,
    3. Pate A,
    4. Van Staa T,
    5. Peek N,
    6. Buchan I
    . Using marginal structural models to adjust for treatment drop-in when developing clinical prediction models. Stat Med2018;37:4142-54. doi:10.1002/sim.7913 pmid:30073700
    OpenUrlCrossRefPubMed
  31. ↵
    1. Sudlow C,
    2. Gallacher J,
    3. Allen N,
    4. et al
    . UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS Med2015;12:e1001779. doi:10.1371/journal.pmed.1001779 pmid:25826379
    OpenUrlCrossRefPubMed
  32. ↵
    1. Grady C,
    2. Eckstein L,
    3. Berkman B,
    4. et al
    . Broad consent for research with biological samples: workshop conclusions. Am J Bioeth2015;15:34-42. doi:10.1080/15265161.2015.1062162 pmid:26305750
    OpenUrlCrossRefPubMed
  33. ↵
    EUR-LEX. General data protection regulation. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=OJ:L:2016:119:TOC (accessed 16 May 2017).
  34. ↵
    1. Ohmann C,
    2. Banzi R,
    3. Canham S,
    4. et al
    . Sharing and reuse of individual participant data from clinical trials: principles and recommendations. BMJ Open2017;7:e018647. doi:10.1136/bmjopen-2017-018647 pmid:29247106
    OpenUrlCrossRefPubMed
  35. ↵
    1. Gheorghe A,
    2. Roberts TE,
    3. Ives JC,
    4. Fletcher BR,
    5. Calvert M
    . Centre selection for clinical trials and the generalisability of results: a mixed methods study. PLoS One2013;8:e56560. doi:10.1371/journal.pone.0056560 pmid:23451055
    OpenUrlCrossRefPubMed
  36. ↵
    1. Rothwell PM
    . External validity of randomised controlled trials: “to whom do the results of this trial apply?”. Lancet2005;365:82-93. doi:10.1016/S0140-6736(04)17670-8 pmid:15639683
    OpenUrlCrossRefPubMedWeb of Science
  37. ↵
    1. Vergouwe Y,
    2. Moons KGM,
    3. Steyerberg EW
    . External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol2010;172:971-80. doi:10.1093/aje/kwq223 pmid:20807737
    OpenUrlCrossRefPubMedWeb of Science
  38. ↵
    1. Zulman DM,
    2. Sussman JB,
    3. Chen X,
    4. Cigolle CT,
    5. Blaum CS,
    6. Hayward RA
    . Examining the evidence: a systematic review of the inclusion and analysis of older adults in randomized controlled trials. J Gen Intern Med2011;26:783-90. doi:10.1007/s11606-010-1629-x pmid:21286840
    OpenUrlCrossRefPubMed
  39. ↵
    1. Shields KE,
    2. Lyerly AD
    . Exclusion of pregnant women from industry-sponsored clinical trials. Obstet Gynecol2013;122:1077-81. doi:10.1097/AOG.0b013e3182a9ca67 pmid:24104789
    OpenUrlCrossRefPubMedWeb of Science
  40. ↵
    1. Hutchins LF,
    2. Unger JM,
    3. Crowley JJ,
    4. Coltman CA Jr.,
    5. Albain KS
    . Underrepresentation of patients 65 years of age or older in cancer-treatment trials. N Engl J Med1999;341:2061-7. doi:10.1056/NEJM199912303412706 pmid:10615079
    OpenUrlCrossRefPubMedWeb of Science
  41. ↵
    1. Lee PY,
    2. Alexander KP,
    3. Hammill BG,
    4. Pasquali SK,
    5. Peterson ED
    . Representation of elderly persons and women in published randomized trials of acute coronary syndromes. JAMA2001;286:708-13. doi:10.1001/jama.286.6.708 pmid:11495621
    OpenUrlCrossRefPubMedWeb of Science
  42. ↵
    1. Schmidt AF,
    2. Groenwold RHH,
    3. van Delden JJM,
    4. et al
    . Justification of exclusion criteria was underreported in a review of cardiovascular trials. J Clin Epidemiol2014;67:635-44. doi:10.1016/j.jclinepi.2013.12.005 pmid:24613498
    OpenUrlCrossRefPubMed
  43. ↵
    1. van Staa TP,
    2. Dyson L,
    3. McCann G,
    4. et al
    . The opportunities and challenges of pragmatic point-of-care randomised trials using routinely collected electronic records: evaluations of two exemplar trials. Health Technol Assess2014;18:1-146. doi:10.3310/hta18430 pmid:25011568
    OpenUrlCrossRefPubMed
  44. ↵
    1. Britton A,
    2. McKee M,
    3. Black N,
    4. McPherson K,
    5. Sanderson C,
    6. Bain C
    . Threats to applicability of randomised trials: exclusions and selective participation. J Health Serv Res Policy1999;4:112-21. doi:10.1177/135581969900400210 pmid:10387403
    OpenUrlCrossRefPubMed
  45. ↵
    1. Junghans C,
    2. Feder G,
    3. Hemingway H,
    4. Timmis A,
    5. Jones M
    . Recruiting patients to medical research: double blind randomised trial of “opt-in” versus “opt-out” strategies. BMJ2005;331:940. doi:10.1136/bmj.38583.625613.AE pmid:16157604
    OpenUrlAbstract/FREE Full Text
  46. ↵
    1. Khudyakov P,
    2. Gorfine M,
    3. Zucker D,
    4. Spiegelman D
    . The impact of covariate measurement error on risk prediction. Stat Med2015;34:2353-67. doi:10.1002/sim.6498 pmid:25865315
    OpenUrlCrossRefPubMed
  47. ↵
    Luijken K, Groenwold RH, van Calster B, Steyerberg EW, van Smeden M. Impact of predictor measurement heterogeneity across settings on performance of prediction models: a measurement error perspective. arXiv:1806.10495 [Preprint]. 2018. https://arxiv.org/abs/1806.10495
  48. ↵
    1. Pajouheshnia R,
    2. van Smeden M,
    3. Peelen LM,
    4. Groenwold RHH
    . How variation in predictor measurement affects the discriminative ability and transportability of a prediction model. J Clin Epidemiol2019;105:136-41. doi:10.1016/j.jclinepi.2018.09.001 pmid:30223065
    OpenUrlCrossRefPubMed
  49. ↵
    1. McCarney R,
    2. Warner J,
    3. Iliffe S,
    4. van Haselen R,
    5. Griffin M,
    6. Fisher P
    . The Hawthorne Effect: a randomised, controlled trial. BMC Med Res Methodol2007;7:30. doi:10.1186/1471-2288-7-30 pmid:17608932
    OpenUrlCrossRefPubMed
  50. ↵
    1. McCambridge J,
    2. Witton J,
    3. Elbourne DR
    . Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects. J Clin Epidemiol2014;67:267-77. doi:10.1016/j.jclinepi.2013.08.015 pmid:24275499
    OpenUrlCrossRefPubMed
  51. ↵
    1. Wolfe F,
    2. Michaud K
    . The Hawthorne effect, sponsored trials, and the overestimation of treatment effectiveness. J Rheumatol2010;37:2216-20. doi:10.3899/jrheum.100497 pmid:20843902
    OpenUrlAbstract/FREE Full Text
  52. ↵
    1. Sedgwick P,
    2. Greenwood N
    . Understanding the Hawthorne effect. BMJ2015;351:h4672. doi:10.1136/bmj.h4672 pmid:26341898
    OpenUrlFREE Full Text
  53. ↵
    1. Finniss DG,
    2. Kaptchuk TJ,
    3. Miller F,
    4. Benedetti F
    . Biological, clinical, and ethical advances of placebo effects. Lancet2010;375:686-95. doi:10.1016/S0140-6736(09)61706-2 pmid:20171404
    OpenUrlCrossRefPubMedWeb of Science
  54. ↵
    1. Grimshaw JM,
    2. Russell IT
    . Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet1993;342:1317-22. doi:10.1016/0140-6736(93)92244-N pmid:7901634
    OpenUrlCrossRefPubMedWeb of Science
  55. ↵
    1. Braunholtz DA,
    2. Edwards SJL,
    3. Lilford RJ
    . Are randomized clinical trials good for us (in the short term)? Evidence for a “trial effect”. J Clin Epidemiol2001;54:217-24. doi:10.1016/S0895-4356(00)00305-X pmid:11223318
    OpenUrlCrossRefPubMedWeb of Science
  56. ↵
    1. Damen JAAG,
    2. Hooft L,
    3. Schuit E,
    4. et al
    . Prediction models for cardiovascular disease risk in the general population: systematic review. BMJ2016;353:i2416. doi:10.1136/bmj.i2416 pmid:27184143
    OpenUrlAbstract/FREE Full Text
  57. ↵
    1. Fleming TR,
    2. DeMets DL
    . Surrogate end points in clinical trials: are we being misled?Ann Intern Med1996;125:605-13. doi:10.7326/0003-4819-125-7-199610010-00011 pmid:8815760
    OpenUrlCrossRefPubMedWeb of Science
  58. ↵
    1. van Smeden M,
    2. Moons KG,
    3. de Groot JA,
    4. et al
    . Sample size for binary logistic prediction models: Beyond events per variable criteria. Stat Methods Med Res2018;962280218784726.pmid:29966490
  59. ↵
    1. Riley RD,
    2. Snell KI,
    3. Ensor J,
    4. et al
    . Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes. Stat Med2019;38:1276-96. doi:10.1002/sim.7992 pmid:30357870
    OpenUrlCrossRefPubMed
  60. ↵
    1. Collins GS,
    2. Ogundimu EO,
    3. Altman DG
    . Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med2016;35:214-26. doi:10.1002/sim.6787 pmid:26553135
    OpenUrlCrossRefPubMed
  61. ↵
    1. Pavlou M,
    2. Ambler G,
    3. Seaman SR,
    4. et al
    . How to develop a more accurate risk prediction model when there are few events [correction in: BMJ 2016;353:i3235]. BMJ2015;351:h3868. doi:10.1136/bmj.h3868 pmid:26264962
    OpenUrlFREE Full Text
  62. ↵
    1. van der Ploeg T,
    2. Austin PC,
    3. Steyerberg EW
    . Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints. BMC Med Res Methodol2014;14:137. doi:10.1186/1471-2288-14-137 pmid:25532820
    OpenUrlCrossRefPubMed
  63. ↵
    1. Organization WH
    . Sciences CfIOoM. International ethical guidelines for health-related research involving humans.Council for International Organizations of Medical Sciences, 2016.
  64. ↵
    1. Debray TP,
    2. Riley RD,
    3. Rovers MM,
    4. Reitsma JB,
    5. Moons KG,
    6. Cochrane IPD Meta-analysis Methods group
    . Individual participant data (IPD) meta-analyses of diagnostic and prognostic modeling studies: guidance on their use. PLoS Med2015;12:e1001886. doi:10.1371/journal.pmed.1001886 pmid:26461078
    OpenUrlCrossRefPubMed
Back to top