Summary of research into the validity of adverse event measures of healthcare quality
| Study | Dimension of validity | Methods | Results and conclusions |
|---|---|---|---|
| PRO = peer review organisation; DRG = diagnosis related group | |||
| Panniers and Newlander (1986)30 | Construct | Applied adverse patient occurrence inventory to sample of 426 patients in DRGs 121 and 122 (myocardial infarction) from one hospital, and collected details of length of stay and treatment costs | Significant correlation between adverse patient occurrence inventory score and both length of stay (p<0.001) and treatment costs (p<0.001) |
| Panniers (1987)31 | Construct | Applied adverse patient occurrence inventory and AS-SCORE to sample of 354 patients in DRGs 121 and 122 (myocardial infarction) from one hospital, and also collected details of length of stay and treatment costs. | Significant correlation between adverse patient occurrence inventory score and AS-SCORE (p<0.001) |
| Schumacher et al (1987)32 | Construct | Applied adverse patient occurrence inventory and severity of illness index to 752 patients from 7 hospitals. | Significant correlation between APO inventory score and severity of illness index (p<0.0001) |
| Barnes and Moynihan (1988)33 | Criterion related | Used 3252 records from SuperPRO project, drawn from PROs all over USA. Applied PRO generic screens for primary review; used single physician assessment for secondary review | Primary screening had sensitivity of 48.5%, specificity of 72.8%. Missed adverse events tended to be less clinically significant ones. Some adverse event definitions were more valid than others |
| Richards et al (1988)28 | Face/content | Used panel of 3 doctors to rate “adversity” of each element of adverse patient occurrence inventory | Results not specifically reported. All ratings obtained were negative and “generally agreed” |
| Richards et al (1988)28 | Criterion related | Applied adverse patient occurrence inventory, severity of illness index, and condition specific quality measures to 25 000 patients from 45 hospitals, and also collected administrative/demographic and clinical details of cases | Adverse patient occurrence inventory score correlated with criterion based quality measures for more serious conditions |
| Richards et al (1988)28 | Construct | Same data set | Adverse patient occurrence inventory score correlated with length of stay and with severity of illness index. Some variations in adverse patient occurrence inventory score observed across different types of hospitals. |
| Brennan et al (1989)34 | Criterion related | Applied own adverse event measure twice each to 360 cases selected to contain a high proportion of cases with adverse events; undertook a separate single physician assessment of cases | For general population of cases, estimated measure had sensitivity of 21% and specificity of 99.5% |
| Institute of Medicine (1990)35 | Criterion related | Used 6 309 839 cases reviewed by PROs up to June 1989 to compare percentages of cases with an adverse event identified at primary screening and percentage of those adverse events subsequently confirmed as quality problems | Between 7.5% and 71.3% of adverse events were confirmed as quality problems—rate varied by type of adverse event |
| Brennan et al (1991)36 | Construct | Used 31 000 patient records drawn from 51 hospitals to compare adverse event rates and negligent adverse event rates across hospitals | Adverse event rate and negligent adverse event rate varied substantially across hospitals. Primary teaching hospitals had higher adverse event rates; rural hospitals had lower adverse event rates. Proportion of adverse events due to negligence lower in teaching and proprietary hospitals; higher in non-profit and governmental hospitals |
| Bates et al (1995)37 | Criterion related | Applied adverse event measure to 3137 admissions; undertook a separate clinician review of those screened positive to assess presence, seriousness and preventability of adverse event | Sensitivity of screening criteria ranged from 68% to 4%; specificity ranged from 56% to 99%. No criteria had both high sensitivity and high specificity |
| Walshe (1998)15 | Face/content | Surveyed 150 doctors in public health and clinical medicine in UK on validity of a general adverse event measure. Interviewed separate sample of 6 doctors | Broad support for validity, though many suggestions for improvements to the detail of the measure. No significant differences between public health and clinical doctors' views. Interview study showed support for adverse event measure but some concerns about need for other measures of quality to be used too. |
| Walshe (1998)15 | Construct | Used data on 12 676 admissions in 8 specialties all screened for adverse events using generic and specialty specific measures | Adverse event rates varied significantly among specialties, were associated with length of stay, mode of admission, and death in hospital |