Original Article
Missing data in a multi-item instrument were best handled by multiple imputation at the item score level

https://doi.org/10.1016/j.jclinepi.2013.09.009Get rights and content

Abstract

Objectives

Regardless of the proportion of missing values, complete-case analysis is most frequently applied, although advanced techniques such as multiple imputation (MI) are available. The objective of this study was to explore the performance of simple and more advanced methods for handling missing data in cases when some, many, or all item scores are missing in a multi-item instrument.

Study Design and Setting

Real-life missing data situations were simulated in a multi-item variable used as a covariate in a linear regression model. Various missing data mechanisms were simulated with an increasing percentage of missing data. Subsequently, several techniques to handle missing data were applied to decide on the most optimal technique for each scenario. Fitted regression coefficients were compared using the bias and coverage as performance parameters.

Results

Mean imputation caused biased estimates in every missing data scenario when data are missing for more than 10% of the subjects. Furthermore, when a large percentage of subjects had missing items (>25%), MI methods applied to the items outperformed methods applied to the total score.

Conclusion

We recommend applying MI to the item scores to get the most accurate regression model estimates. Moreover, we advise not to use any form of mean imputation to handle missing data.

Introduction

Missing data on multi-item instruments is a frequently seen problem in epidemiological and medical studies. Multi-item instruments can be used to measure, for example, quality of life, coping ability, or other psychological states. A multi-item instrument generally consists of several items that measure one construct [1], for example, the Pain Coping Inventory assesses active coping skills of people with pain complaints by 12 items [2]. Missing data on these kinds of instruments can occur as missing item scores, when several items are not completed or as missing data in total scores when the entire instrument is not filled out. Furthermore, missing item scores impair the calculation of the total score, which can lead to missing total scores as well. For missing data in item and total scores, different missing data-handling methods are available, with complete-case analysis (CCA) as the most frequently used method [3]. In general, CCA tends to perform well under the strict assumption that missing data are a completely random subsample of the data, in other words missing completely at random (MCAR) [4]. However, CCA reduces power caused by a decreased sample size. Single-imputation methods such as mean imputation of the total score and item mean imputation may be used to preserve the sample size by replacing the missing values by the mean score, but these methods reduce the variability in the data. Single stochastic regression imputation (SRI) uses observed data to predict the missing value and adds residual error to the imputed data to restore the variability in the data, but this method does not take the uncertainty of the imputed values into account.

Mostly, the probability of missing data depends on other observed variables, indicated as missing at random (MAR) [4]. In contrast to traditional methods such as CCA and mean imputation, more advanced methods such as multiple imputation (MI) produce reliable and unbiased results under the MAR mechanism and take missing data uncertainty into account [5], [6]. Both traditional and advanced methods can be applied either to the missing item scores or directly to the missing total scores.

The comparison between missing data methods for item-level and total score-level missingness in questionnaire data is seldom made in one study [3]. Other simulation studies have researched the performance of missing data methods applied to nonquestionnaire data [7], [8] or only studied methods applied to the item scores of a multi-item instrument [9], [10], [11], [12], [13]. For example, Burns et al. [13] studied the performance of MI of missing item scores but did not compare this with imputing at the total score level of their questionnaire. So far, it is still unclear if it is better to apply a missing data-handling method to the missing item scores or to the total scores when some or many items in a multi-item instrument are missing. Moreover, the impact on the study results of different missing data methods when multi-item data are missing on the covariate has not been researched extensively yet. The present study aimed to explore the performance of different missing data-handling methods designed for missing item scores and missing total scores in a multivariate regression model. This objective is considered in the following two aspects: (1) which missing data methods should be used to handle missing (item) data and (2) should this missing data method be applied to the item scores or to the total scores.

Section snippets

Simulation set up

To investigate the differences between several imputation methods, we used a simulation procedure comparable with the study performed by Marshall et al. [7]. We based our simulation on an empirical data set, which was previously used in a prospective cohort study investigating the prognosis of low back pain [14]. In this study, we used a cross-sectional part of these data that contained the multi-item variable active coping of the Pain Coping Inventory (PCI-active) [2]. The PCI-active consists

Results

In Table 2, the regression coefficient and SE estimates for the PCI-active total score under the three missing data mechanisms are presented. Not surprisingly, for the MCAR data, the coefficient estimate was the same as the true coefficient value, but the SE increased with higher missing data rates. A similar trend was seen in the MAR and MNAR missing data situations, however accompanied by much larger deviations in SEs.

Figures 1 and 2 present the effect of the missing data-handling methods on

Discussion

The results of our study are that missing item data are best handled by applying MI based on PMM or SR to the item scores regardless of how many subject scores and item scores are missing. Furthermore, single SRI also seems to yield acceptable results, and mean imputation of the total scores performs worst. Additionally, we showed that the underlying mechanism influences the performance of the missing data-handling method, especially when large amounts of data are missing. This is of concern

References (38)

  • S. van Buuren

    Item imputation without specifying scale structure

    Methodology

    (2010)
  • J.R. van Ginkel et al.

    Incidence of missing item scores in personality measurement, and simple item-score imputation

    Methodology

    (2010)
  • G. Hawthorne et al.

    Imputing cross-sectional missing data: comparison of common techniques

    Aust NZJ Psychiatry

    (2005)
  • P.L. Roth et al.

    Missing data in multiple item scales: a Monte Carlo analysis of missing data techniques

    Organizational Res Methods

    (1999)
  • M.W. Heymans et al.

    The effectiveness of high-intensity versus low-intensity back schools in an occupational setting: a pragmatic randomized controlled trial

    Spine

    (2006)
  • W.N. Venables et al.

    Modern applied statistics with S. Fourth Edition

    (2002)
  • J.P.L. Brand et al.

    A toolkit in SAS for the evaluation of multiple imputation methods

    Stat Neerlandica

    (2003)
  • R Core Development Team. R. A language and environment for statistical computing. Vienna,...
  • C.A. Bernaards et al.

    Influence of imputation and EM methods on factor analysis when item nonresponse in questionnaire data is nonignorable

    Multivariate Behav Res

    (2000)
  • Cited by (0)

    Funding: This work was financially supported by EMGO Institute of Health and Care Research.

    Conflict of interest: None.

    View full text