Introduction

The prevalence of foot ulcers in patients with diabetes ranges from 1.3% to 4.8% [1]. Patients with diabetes also have a 20-fold higher chance of suffering non-traumatic amputation than non-diabetic patients [2]. The diabetic foot is a multifactorial disorder. An awareness of all factors and how each one of them affects each patient should allow a more rational approach to prevention and treatment of complications. Prevention plays a key role in avoidance of ulcers, ulcer recurrence and amputation. Current guidelines emphasise the need for: (1) direct and sustained follow-up; (2) education of the patient and family; (3) treatment of non-ulcerative lesions; and (4) early identification of feet at risk [3, 4]. The last of these is essential when planning prevention in the light of resource allocation, which is typically limited [5].

Risk stratification should be based on a structured clinical evaluation of the patient, allowing the incorporation of characteristics from diagnostic tests and from clinical history or examination [6], and resulting in a clinical predictive model, which summarises and quantifies the probability of ulcer development in terms of a numerical index.

These models should allow us to analyse the impact of each variable as well as the effect of modifying them on ulceration risk. In 2006, Boyko et al. [7] developed a risk stratification model to predict foot ulceration in diabetic patients, using seven commonly available variables, with good results. However, to our knowledge, no external validation of the model has been performed to date. This study therefore sought to validate and optimise the diagnostic accuracy of the Boyko model in a different setting.

Methods

Type of study and selection of participants

We conducted a retrospective cohort study on all patients with diabetes attending the podiatry section of a diabetic foot clinic at a public tertiary hospital in Vila Nova de Gaia, northern Portugal from February 2002 to October 2008. Patients (n = 435) were previously diagnosed with diabetes and classified as type 1 or 2 according to the World Health Organization criteria [8]. Patients were excluded (n = 75) if they were unable to walk, if data were incomplete and/or if they had less than three podiatric appointments. This study was approved by the Ethics Committee of our institution. No adverse events occurred during the study.

Data collection

At the first podiatric appointment all variables were assessed and registered through a structured interview and thorough foot examination, performed by two podiatrists (with 10 and 6 years of experience, respectively, in evaluating diabetic foot). Predictive and outcome variables were then collected from the patients’ clinical files until May 2009.

The outcome analysed was foot ulcer development, i.e. a full-thickness skin defect distal to the malleolli that required more than 14 days to heal [7]. Follow-up on both limbs ended as soon as the first foot ulcer occurred. Given that this study sought to predict foot ulcers at the patient-level, all limb-specific measurements conducted in each participant were rated as present if they occurred in at least one foot. Investigators were blinded to stratification score during collection of predictive variables and outcome. Participants were re-evaluated at intervals of 1 to 12 months, according to the International Working Group on Diabetic Foot (IWGDF) guidelines [3, 4], but were instructed to return to the clinic if any ulcer or any complication appeared before the next scheduled appointment. The primary cause of the ulcer was also recorded.

Variables used in study by Boyko et al.

Prediction model variables included by Boyko et al. (HbA1c, visual impairment, monofilament insensitivity, tinea pedis, onychomycosis, and history of foot ulcer and amputation) [7] were collected in the following way in the present study. For HbA1c we used the value closest to the date of the first appointment. Visual impairment was assessed through interview and subjective analysis of the patient’s ability to inspect her/his own feet [5]. If the patient was unable to perform regular foot care and examine her/his feet due to visual inability, and/or if she/he presented with lesions caused by improper nail trimming, visual impairment was considered to be present. Sensory testing was evaluated, with the patient’s foot supported, by applying the Semmes–Weinstein monofilament perpendicularly and briefly (approximately 2 s) at four non-keratinised points on each foot, i.e. the hallux and the first, third and fifth metatarsal heads, while the patient’s eyes were closed. At each stage the patient was asked whether she/he felt the filament. A negative answer to one or more applications was classified as altered sensation [9]. The presence of tinea pedis and onychomycosis was assessed during the foot examination. A history of previous ulcer and/or foot amputation was obtained in the course of the visit.

Variables that were analysed in the original study but not included in the final model were also collected. Demographic data (age and sex), as well as information on diabetes type, treatment and duration, smoking habits, presence of intermittent claudication and previous laser photocoagulation were collected through a structured interview. During foot inspection, the presence of the following was also assessed: callus, foot deformity (any foot alteration augmenting the pressure in one or several sites of the foot and consequently potentially contributing to callus and/or ulcer development), hallux limitus and oedema.

Other variables

Other variables not analysed in Boyko’s study [7] were also collected at baseline due to their clinical relevance, availability and possible indication of an association with ulcer development. Among these variables were: eye problems due to diabetes [10, 11], physical impairment defined as patient experiencing difficulty in reaching her/his own feet [5], reported myocardial infarction [12] and stroke [10], educational level, nephropathy [12–14] and neuropathic symptoms [14, 15]. Peripheral vascular disease was assessed during the foot examination (defined as the presence of only two or less of the four pedal pulses [13]), as well as foot care habits [11, 16], i.e. nail care and skin moisturising.

Footwear, brought to the appointment, was evaluated and classified: (1) as low-risk if it covered the foot adequately, had laces and was of the correct size; (2) as moderate-risk for slippers, shoes without laces or shoes made from an inappropriately soft material; and (3) as high-risk if too small, or defined as sandals or flip-flops [13]. All patients were also asked to describe their most frequently used shoes. The attributed classification was based upon the footwear that presented the highest risk. Patients presenting with lesions, callosities or reddened sites due directly to footwear were immediately classed as using high-risk footwear, regardless of the shoes described or worn at the appointment.

Statistical analysis

Risk factor distribution was compared between the groups of patients with and without ulcers over the course of follow-up. Continuous variables (HbA1c, age and diabetes duration) were compared using Student’s t test or the Mann–Whitney U test for independent samples according to the distribution of the variable. The remaining dichotomous variables were compared using χ 2 test or Fisher’s exact test, when applicable. Normality of distribution was assessed by evaluating the histogram and applying the Kolmogorov–Smirnov test. Significance was defined as p < 0.05. All statistical analyses were performed using SPSS version 17.0 (SPSS, Chicago, IL, USA).

Of the variables not included in Boyko’s original model [7], those that were significantly associated with ulcer development in the multivariable analysis were included separately or in groups in the optimised model. To understand the impact of each alteration on model optimisation, the AUC for the receiver operating characteristic curve was recalculated for each variable included. Multivariable analysis and model modifications were performed using logistic regression. The same procedure was performed in reverse, removing each variable from Boyko’s original model [7] that was not significant in multivariate analysis, to see if it was possible to simplify it.

For the purpose of this study, the risk score was calculated using the downloadable spreadsheet from www.eric.seattle.med.va.gov/downloads.html [7] (accessed 1 June 2009). The risk score was defined as: score=0.0975 × (HbA1c in % units) + 0.7101 × (presence of monofilament insensitivity) + 0.3888 × (presence of visual impairment) – 0.3206 × (presence of tinea pedis) + 0.4579 × (presence of onychomycosis) + 0.7784 × (presence of previous foot ulcer history) + 0.943 × (presence of previous lower limb amputation).

Patients were stratified according to Boyko’s model [7], i.e. lowest risk if the score was lower than 1.48; next-to-lowest risk for score between 1.48 and 1.99; next-to-highest risk for score between 2.00 and 2.61; and highest risk if the score was greater than 2.61. The sensitivity, specificity, likelihood ratios (LR), predictive values and AUC with 95% CIs of the original and optimised stratification model were calculated. Due to the high dependence of predictive values on prevalence condition [17], they were calculated using the different prevalence values reported (ranging from 5 [5] to 34% [14]) in the five articles [4, 5, 7, 14, 18] identified by us as assessing the effectiveness of foot ulcer risk stratification systems.

Results

Description of participants

This study included 360 participants, with a median follow-up of 25 months (range 3–86). Patients had a median age of about 65 years, a median diabetes duration of 15 years, mainly type 2 diabetes (98%), with a low level of education (86% up to primary school). Of the 360 participants, 45% were men. Each participant had a median of eight appointments (minimum 3, maximum 20). Ulceration occurred in 94 (26%) participants. The main direct cause was inadequate footwear (60%), followed by self-treatment injury (9%) and direct trauma (7%). The median time between date of first appointment and HbA1c measurement was 2 months (range 0–3 months).

Association of variables with ulcer development

In our study, six of the seven variables included in Boyko’s original model [7] were statistically associated with ulcer development in univariate analysis: HbA1c, visual impairment, previous foot ulcer, previous amputation, monofilament insensitivity and onychomycosis (Table 1). However, some variables analysed, but not included in the original study, revealed association in our study: sex, intermittent claudication, foot deformity and hallux limitus. Of the variables not analysed in Boyko’s study [7] only eye problems due to diabetes, peripheral vascular disease, physical impairment and the use of moderate- to high-risk footwear were considered as possible predictive variables of foot ulceration (Table 1).

Table 1 Baseline characteristics with univariate and multivariate analysis for association with foot ulcer development
Full size table

Only variables with statistically significant association and those included in the original model were evaluated in multivariate analysis. Results show that only HbA1c, previous foot ulcer, tinea pedis, sex, hallux limitus and footwear were statistically significant in the multivariable analysis (Table 1). We therefore assessed whether the inclusion of sex, hallux limitus and footwear, and/or extraction of visual impairment, previous amputation, monofilament insensitivity and onychomycosis variables would improve the models’ accuracy for prediction of diabetic foot ulceration.

Optimisation of model

Only the inclusion of the hallux limitus (AUC 0.84) and use of moderate- or high-risk footwear (AUC 0.88) variables improved the accuracy of the original model, but this was not statistically significant. When included together, the improvement equalled that of the footwear variable alone (AUC 0.88). Exclusion of the variable monofilament insensitivity diminished the model’s accuracy (AUC 0.82), whereas upon exclusion of visual impairment, previous amputation and onychomycosis as variables, either separately, in pairs or all together, model accuracy remained the same (AUC 0.83) (Table 2).

Table 2 Original model AUC and optimisation process (through addition or removal of selected variables based on multivariable analysis results)
Full size table

Inclusion of the footwear variable with simultaneous exclusion of any or all the variables suggested for extraction of the original model maintained the model’s AUC at 0.88 (results not shown).

Our analysis of the resulting AUC (excluding the proposed variables) verified that accuracy of the model remained unchanged, while the respective confidence interval did not allow us to state how the model might have been affected. Therefore, we chose to adopt a conservative position and retain the proposed variables in our final model proposal, which resulted in the following model equation: score = 0.373 × (HbA1c in %) + 0.217 × (presence of visual impairment) + 2.037 × (presence of previous ulcer history) + 0.593 × (presence of previous amputation) + 0.637 × (presence of monofilament insensitivity) − 1.256 × (presence of tinea pedis) + 0.217 × (presence of onychomycosis) + 1.905 × (use of moderate- or high-risk footwear).

Best stratification was achieved using the following cut-off scores: under 3.87 (lowest risk); 3.87 to 5.66 (next-to-lowest risk); 5.67 to 6.81 (next-to-highest risk); above 6.81 (highest risk; Fig. 1).

Fig. 1
figure 1

Receiver operating characteristic curves (ROC) of the original and optimised models, with respective stratification. Dashed line, reference line (corresponds to an AUC of 0.5); blue line, ROC curve of original model; orange line, ROC curve of optimised model; black lines, best fit lines for original and optimised models (using cut-offs proposed in the study by Boyko et al. [7] and the selected cut-offs described in the text, respectively). It can be seen that the optimised model had a higher AUC, albeit without achieving statistical significance

Full size image

Accuracy of the original and optimised models

Ulcer development in each group of the original and optimised model risk stratification is shown in Table 3 (p < 0.001 for association, p < 0.001 for trend in both models). For the original model, the risk of foot ulcer development was fourfold greater in the next-to-lowest group (13%), sevenfold in the next-to-highest group (23%) and 11-fold in the highest group (39%) when compared with participants in the lowest risk group (3.5%). For the optimised model, patients in the next-to-lowest risk group had a sevenfold (11%), those in the next-to-highest a 19-fold (28%) and in those in highest group a 27-fold (41%) higher risk of foot ulcer development than patients in the lowest risk group (1.5%).

Table 3 Risk stratification of original and optimised models for ulcer development distribution and accuracy measures for different cut-offs
Full size table

In both models, the specificity value of the highest risk group and the sensibility value of the highest plus next-to-highest plus next-to-lowest risk groups are excellent (Table 3). The optimised model tended to produce higher specificity and positive LR at all levels, i.e. to show better results in prediction of patients who will develop foot ulceration (the most important property in a tertiary care context). It also tended to produce higher values for all diagnostic accuracy measures in Table 3 for detection of patients in or above the next-to-highest risk and in or above the next-to-lowest risk groups. However, none of these differences were statistically significant (Table 3).

Foot ulcer prevalence varies greatly from one study to another, which results in diverse predictive values. For a low prevalence (community setting), both models produced a 91% negative predictive value (NPV), with low positive predictive value (PPV; 20% in the original model, 25% in the optimised model). As expected, with an increase in outcome prevalence, NPV dropped up to 51% in a high-risk setting, where foot ulcer development occurred in 34% of patients. Conversely, PPV rose to 71% and 75% for the original and optimised model, respectively. With our study prevalence, both models had moderate NPV and PPV. The optimised model tended to generate higher PPV, although without statistical significance, and similar NPV to the original model (Table 4).

Table 4 Risk stratification of original and optimised models for predictive values for different foot ulcer prevalence
Full size table

Subgroup analysis

One of the limitations of the study by Boyko et al. [7] is that their sample consisted mainly of men. In our sample, we verified that sex was equally distributed and therefore performed a subgroup analysis. The original model stratification in men had an AUC of 0.84 (95% CI 0.78–0.90); in women the AUC was 0.80 (0.72–0.87). The optimised model stratification in men had an AUC of 0.89 (0.85–0.94), while in women the AUC was 0.83 (0.77–0.90). As can be seen, there were no statistically significant differences between sexes (although both models tended to perform better in men) or between the original model and the optimised model in both sexes (although the optimised tended to produce better results). Both models had good classification accuracy for prediction of foot ulcer development at the patient-level in both sexes. The optimised model tended to produce higher AUC values.

Discussion

An unhealed foot ulcer is the most frequent precursor of a diabetes-related lower limb amputation. Therefore ulcer prevention is vital. Ideally, specialised foot care should be given to all patients with diabetes. However, in reality preventive and treatment resources for the diabetic foot must be distributed in a cost-effective and rational manner [19].

It should be stressed that the population studied may be a biased sample, since this study was performed in a tertiary referral centre. However, in Portugal there are no secondary care institutions and resources allocated to prevention of diabetic foot complications in the primary care setting remain limited (as in several other countries). For now, therefore, it is the responsibility of hospitals to fill this gap, develop strategies and increase awareness at the remaining institutions of this extremely important health and social problem. Nonetheless, the outcome prevalence was superior in our study (26%) to that in Boyko’s study (17%) [7].

This study evaluated the validity of the first logarithmic model for risk degree stratification for development of foot ulcers in patients with diabetes. Logarithmic models have an advantage when compared with simple stratification systems. Knowing the effect of each variable in the risk degree simplifies the detection of risk variables, allows a more personalised approach and improves patient perception. For example, a patient with a high HbA1c can be shown the changes likely to occur in her/his risk group merely by achieving better glycaemic control. We confirmed that the commonly available and easy-to-collect variables chosen by Boyko et al. in their study [7] have excellent capacity for predicting foot ulceration in patients with diabetes. The AUC of 0.83 (95% CI 0.78–0.88) includes the value obtained in their study of 0.81 at 1 year [7].

We must highlight that 60% of ulcer occurrence resulted mainly from the use of inadequate footwear, clearly underlining the importance of this variable, which is simple to collect and ranked, moreover, as one of the five pillars of foot ulcer prevention by the IWGDF [3]; it was also significantly associated with foot ulcer development in several studies assessing inadequate footwear as a variable [13, 15]. Our results suggest that the inclusion of a footwear variable could improve the model (AUC 0.88). Yet it was not statistically significant, although this might be explained by the size of our sample. The same explanation can be applied to the variables that were considered for exclusion in the model. And so, we chose to adopt a conservative approach and retain them in the model. Ideally, 10 to 15 patients should be enrolled for each variable analysed [20] and therefore sample size should be between 300 and 450 participants. We included 360 patients, a figure comprising all patients available over the study period.

As mentioned above, all variables analysed in the study by Boyko et al. [7] were collected at the first appointment, except for race, which was not collected as all of our patients were Whites, and weight, which was not collected because of space, material and time restrictions (so we had no data for this variable). Nonetheless, both variables were not considered to be statistically significant in the prediction of ulcer development in the study by Boyko et al. [7]. Reproducibility was not evaluated because variables were collected through interview or blood analysis (HbA1c). For the variables collected through foot examination, the retrospective design meant that it was impossible to know which of the two podiatrists had collated a given variable.

Visual acuity data were collected in a less formal way than in the study by Boyko et al., in which visual impairment was assessed using a Snellen chart [7]. However, this procedure did not seem to affect the model’s discriminatory abilities. Although the method used is not the most exact form, it was the only viable way, given that ophthalmological observation was not possible and that the study was retrospective in character. Visual impairment data are often collected using specific eye evaluation charts. However, due to the impossibilities described in Methods, we chose to collect data for this variable in the same way as Leese et al. [5] in their study evaluating the effectiveness of the Scottish Intercollegiate Guidelines Network (SIGN) stratification system (the only alternative way of collecting this variable).

Consistency in the sensory testing procedure is very low and there is no selected standard method [21, 22]. We chose our cut-off, as proposed by Smieja et al. [9], since this four-site shortened version presented 94% accuracy for the detection of patients with abnormal 16-site monofilament examination. In addition, sites were selected on the basis of the literature and the resulting most common ulceration locations [9].

For assessment of peripheral vascular disease, we believe that the absence of two palpable pulses in one foot or the absence of pedal or posterior tibial pulses in both feet represent a noteworthy vascular supply alteration, and so we chose the designation used in the prospective cohort study of Abbott et al. [13].

In our study (as in Boyko’s [7]), tinea pedis was associated with foot ulcer development only in the multivariable analysis. Although no former studies have evaluated this association, Boyko et al. suggested that tinea pedis could be a clinical marker for intact autonomic function [7].

Participants unable to walk were excluded from this study in line with the exclusion factors from the study by Boyko et al. [7] and because they represent a specific subpopulation. This group included patients undergoing prosthetics adaptation, hospitalised patients and patients with severe physical limitations. We believe that these patients are at higher risk of developing pressure ulcer, which has a considerably different pathophysiology.

One of the limitations of the study by Boyko et al. [7] was that their sample consisted mainly of men. In our sample, sex was equally distributed. Subgroup analysis revealed no statistically significant differences in the AUC between sexes or between the original and the optimised model in both sexes. This indicates that the two models can be applied in women and men with good results. On the other hand, our population, like that of Boyko et al. [7], included mainly elderly (average age 65 years) patients with type 2 diabetes (98%), which may affect the generalisability of the models. Nonetheless, type 2 diabetes prevalence is around 90% [8] and we recommend that the model should be applied in a sample with similar distribution to that in the general diabetic population.

The selection of variables for multivariate analysis was based on an association having a p value of p < 0.05 and/or on it being present in the original model. The strict use of these criteria, rather than including all empirically based variables, may have contributed to a lack of information. On the other hand, we sought to simplify the optimised model to a maximum degree.

Another limitation of our study is the fact that we did not analyse the effect of time. Therefore statistical analysis was done by logistic regression, in spite of Cox proportional hazards modelling, due to the fact that very few patients were followed for 5 or more years (second end-line used by Boyko et al.) [7]. On the other hand, it is not possible to define a cut-off prevalence to designate low-, moderate- or high-risk contexts. If the cut-off is defined as superior to 5%, then all studies [4, 7, 14, 18] except one [5] that have assessed the effectiveness of a foot ulcer risk stratification system were performed in high-risk settings. In the paper by Peters and colleagues [4] the reported prevalence was similar to ours (25%) and in that by Lavery and co-workers [14] it was as high as 34%.

In comparison to the IWGDF stratification system [4], both models (original and optimised) had similar results in the detection of patients at highest risk of foot ulcer development. Nevertheless, the SIGN stratification system study [5] reported statistically significant better sensitivity, positive and negative LR for identifying patients at higher risk in comparison to the original model, and better sensitivity and negative LR in comparison to the optimised model. For the detection of patients at highest and next-to-highest risk, the IWGDF system showed statistically significant inferior specificity and positive LR in comparison to the optimised model. The SIGN system reported higher sensitivity than the original model, but similar results to those of the optimised model. The NPV value was higher in this study than in both models in our study. The PPV value was also higher, but statistical significance was not maintained when compared with the optimised model. However, these results were obtained in a dissimilar population, and neither the IWGDF nor the SIGN systems have been externally validated. A comparison with studies by Lavery and co-workers [14, 18] was not possible due to the impossibility to calculate diagnostic accuracy measures. Analysis of LR values (for both models) showed that most of them generate only small alterations in the outcome odds, although such alterations can sometimes be important [17]. It should, however, be stressed that the positive LR in the highest risk group had a moderate impact and the negative LR in the lowest risk group a large impact on the outcome odds (Table 3). Our results demonstrate that Boyko’s model [7] in its original or in the optimised form has a high discriminating ability with very low negative LRs and 91% NPVs, i. e. in a low-prevalence context participants in the lowest risk group have a 91% probability of remaining free of ulcer development (Table 4). In addition, both models resulted in a low proportion of false positives in the highest risk group, i.e. in a high specificity and moderate level of positive LRs, which is vital for tertiary care institutions.

Our results indicate that in countries with three different care levels, tertiary institutions should follow patients at the highest risk level (since this group presented a high specificity and positive LR), secondary institutions should follow those at next-to-lowest and next-to-highest risk, and primary institutions should screen all new patients and follow those at the lowest risk (given that this group had a high NPV). However, in countries with no intermediate care institutions, tertiary institutions should follow patients in the highest and next-to-highest risk groups, given that for them sensitivity is high (84% in the original model, 88% in the optimised model) at the expense of around 25% of patients who will not develop ulcers, but are included in those groups. Primary care institutions should screen and follow the remaining patients.

Although retrospective in design, this study represents the first attempt at external validation and optimisation of the Boyko model [7], therefore raising this clinical decision rule to level 2 of evidence. [23] As a future project, we wish to validate this model prospectively in a larger population, if possible with a higher proportion of patients with type 1 diabetes, and to establish whether inclusion of a footwear variable and of other variables resulting from a systematic review can in fact improve the model.

The stratification by risk of foot ulceration is a key issue in the allocation of scarce resources that represent everyday reality for podiatric services. The stratification studied is easy to use and available via internet, and can be used to identify patients at higher risk of foot ulceration, i.e. patients for whom the most specialised care, orthotic resources, structured educational programmes and more frequent examinations should be provided [4, 5, 24]. We conclude that Boyko’s model [7], based on seven easily available variables, is an excellent discriminating instrument for foot ulcer prediction in patients of both sexes with diabetes and that inclusion of a footwear variable may improve the model.