There is growing international interest in harnessing patient dissatisfaction and complaints to address problems with quality in health care.¹ The value of complaints as a marker of threats to patient safety depends on the answers to several questions. Do complaints track injuries, or are they prompted by more subjective concerns? Are complaints the “tip of the iceberg” in terms of quality of care problems and, if so, how representative are they of broader quality problems? The longstanding obstacle to addressing these questions is methodological in nature and concerns the elusiveness of an appropriate metric against which to measure the prevalence and reasonableness of complaints.

In New Zealand, injury compensation and complaints against healthcare professionals are dealt with in distinct settings. An innovative accident compensation system compensates injured patients on a “no fault” basis. An independent health ombudsman, the Health and Disability Commissioner (“the Commissioner”), has statutory responsibility for resolving patient complaints about quality of care, acts as a gatekeeper to disciplinary proceedings, and strives to use complaints as a catalyst for improving patient safety.^2–4

In this study we linked information on quality of care complaints lodged with the Commissioner with adverse event data gathered in the New Zealand Quality of Healthcare Study (NZQHS). Together, these two datasets permit estimation of how frequently adverse events led to complaints, and a description of the characteristics of patients who did and did not complain to the Commissioner.

METHODS

The Wellington Ethics Committee and the Harvard Institutional Review Board approved the study.

Baseline data on a random sample of patients who had experienced adverse events came from the NZQHS. As previously described,⁵ NZQHS used a two stage sampling process to develop a representative sample of 6579 medical records of patients discharged from publicly funded acute care hospitals in 1998, excluding psychiatric and same day discharges. Trained reviewers assessed each episode of care for the presence of an adverse event and, when an adverse event was detected, rendered a judgment on whether it was preventable. Following previous research in the United States,^6,7 adverse events were defined as unintended injuries caused by healthcare management, rather than the underlying disease process, that resulted in disability. The study included all adverse events detected during or responsible for the index admission, as well as those occurring during the index admission that were detected on a subsequent admission. Serious adverse events were those which caused death or permanent disability.

The Commissioner provided data on all complaints received by his office that were associated with a public hospital admission in 1998 (n = 398). To determine which of these complaints involved adverse events and preventable adverse events, an investigator with medicolegal expertise (MMB) reviewed the relevant administrative information using the NZQHS process of structured implicit review. A second investigator (TAB), also a medicolegal expert, independently reviewed a random subsample of 98 complaints, a quarter of the sample. Inter-rater reliability for determination of whether an adverse event had occurred was high (κ = 0.84); reliability for preventability judgment was moderate (κ = 0.50), although slightly higher than previous estimates of reliability for preventability determinations.^7,8

We matched the NZQHS records to complaints probabilistically using national hospital number, date of birth, sex, and date of injury. Clinical data from the medical record review was then compared with claims data to confirm that the NZQHS patient and complainant was the same person and that the complaint related to the same episode of care. Two investigators (MB, DS) reviewed potential matches and reached consensus on whether sufficient information existed to confirm a match with a high degree of certainty. Among 18 candidate matches, nine were determined not to be matches and two were excluded on the grounds that there was insufficient information to confirm a match.

The analyses are descriptive. We compared characteristics of patients who lodged complaints with the Commissioner after experiencing an adverse event (“complainants”) with characteristics of patients from the NZQHS who experienced an adverse event and did not file complaints (“non-complainants”). Matched cases—that is, patients from the NZQHS sample whose care involved an adverse event and who subsequently complained about that instance of care—were classified as complainants. Figure 1 shows the derivation of the two populations which were combined for the multivariate analysis.

Data were analysed using the SAS 9.0 statistical software package (Cary, NC, USA) Stata 8.0/SE (Stata Corp, College Station, Texas). We used t tests and χ² tests to conduct bivariate comparisons of the characteristics of complainants and non-complainants. We investigated predictors of failure to complain despite having experienced an adverse event using multivariate logistic regression. The dependent variable in the regression analysis distinguished complainants from non-complainants. The independent variables were sex, ethnicity (Maori, Pacific, Non-Maori/non-Pacific), patient age (<1 year, 1–17 years, 18–44 years, 45–64 years, ⩾65 years), disability due to adverse event (temporary, permanent with <50% impairment, permanent with >50% impairment, death), and whether or not the event was preventable. An additional covariate provided a measure of the patient’s socioeconomic status using the New Zealand Index of Deprivation Score.^9,10 This index, based on mesh blocks, combines nine census variables reflecting aspects of material and social deprivation; following previous studies,¹¹ index scores were separated into quintiles for analysis.

To account for the stratified two stage cluster sampling design in the NZQHS,¹² the bivariate and multivariate analyses were weighted. The weighting had negligible effects on our estimates.

RESULTS

The Commissioner received 398 complaints related to care delivered in public hospitals in 1998, 254 of which (64%) related to an episode of care in which the patient had experienced an adverse event. For 51% of the complaints the adverse event was judged to be preventable. The NZQHS review identified 850 adverse events, as previously reported,⁵ of which 315 were preventable, 124 were serious, and 48 were both serious and preventable.

Patients themselves instigated 105 (41%) of the 254 complaints involving adverse events (fig 2). Third party complaints were commonly laid by family members, primarily the patient’s child (17%), parent (16%), or spouse (13%). A total of 79% (313/398) of complaint letters and 75% (191/254) of the adverse event complaint letters expressed concern about a health professional’s attitude or communication either during the index admission or after the adverse event.

There were seven matches between the complaint sample and the full NZQHS sample (n = 6579). NZQHS reviewers judged three of these matches to involve adverse events (one preventable death, one preventable permanent disability, and one unpreventable temporary injury); the rest did not involve adverse events. (Besides injuries, the Commissioner also has jurisdiction to hear complaints relating to informed consent, discrimination, and a variety of other bases of dissatisfaction with care.) Hence, 0.4% (3/850) of the patients in the NZQHS sample who experienced adverse events complained. Among NZQHS patients judged to have experienced adverse events that were serious and preventable, 4% (2/48) complained.

Box 1 provides case studies of three patients. The first patient complained following an adverse event. The second patient suffered an adverse event and did not complain. The third patient complained but, because the poor outcome he experienced was attributable to the condition for which care was sought rather than the medical care itself, it was not an adverse event.

Table 1 shows the characteristics of the patients in the full NZQHS sample, the subset of patients who experienced adverse events (except for the three who complained), and the complainants who experienced adverse events. The average age of the complainants was 47 years and 59% were female. Among complainants for whom ethnicity data were available, 14% were Maori and 3% were Pacific. In general, complainants’ injuries involved were quite severe, with 31% resulting in permanent injury and 28% in death. Most of the injuries (79%) were preventable. 44% (110/254) of complaints involved an injury that was both permanent and preventable.

Bivariate analyses showed several significant differences between injured complainants and injured non-complainants (table 1). Compared with complainants, non-complainants were significantly older (52 v 47 years, p = 0.003) and more likely to live in deprived socioeconomic areas. Complainants, on the other hand, were significantly more likely to have sustained injuries that led to permanent disability or death, and preventable adverse events were twice as common in this group (79% v 37%, p<0.001).

These differences persisted in multivariate comparisons of the complainants and non-complainants (table 2). Injury severity was a strong predictor of complaining, with odds of complaining increasing with injury severity. Compared to patients with temporary disability, the odds of complaining for patients with a permanent disability resulting in >50% impairment were 11.4 times greater (95% CI 5.9 to 22.1), and for patients who died they were 17.9 times greater (95% CI 9.3 to 34.2). There was also a strong independent correlation between preventability and odds of complaining (odds ratio (OR) 7.6, 95% confidence interval (CI) 5.0 to 11.6).

In addition, several sociodemographic factors were associated with propensity to complain after an adverse event. Odds of complaining for patients in the most deprived quintiles were one third those for patients in the most privileged quintile (OR 0.3, 95% CI 0.2 to 0.6). Elderly patients were significantly less likely than their younger counterparts to complain following an adverse event (OR 0.2, 95% CI 0.1 to 0.4). Odds of complaining for patients of Pacific ethnicity were lower than for non-Maori/non-Pacific patients (OR 0.3, 95% CI 0.1 to 0.9). The difference was not statistically significant for Maori patients, perhaps because of the relatively small numbers in this category.

DISCUSSION

Principal findings

This study is the first to match epidemiological data on medical injuries to complaints about quality of care lodged with a national health ombudsman. Three findings are noteworthy. Firstly, while the right to an appropriate standard of care is only one of 10 patient rights enforced by the Commissioner, most complaints involved an adverse event, often a serious one. Secondly, our results suggest that approximately one in 200 injured patients complain about their care to the Commissioner. Among patients who experience injuries that are both serious and preventable, one in 25 complain. Thirdly, the “under-complaining” phenomenon was not spread uniformly across the patient population: elderly patients and socioeconomically disadvantaged patients were especially unlikely to complain despite having suffered an injury, and propensity to complain increased steeply with the severity of the injury sustained.

Legitimacy of complaints

Although New Zealand doctors appear to support the use of complaints as a quality assurance tool, concerns abound in the medical community about the prevalence and reasonableness of many complaints brought before the Commissioner.^13–15 The 2001 survey by Cunningham and colleagues¹³ of doctors’ attitudes about complaints found general support for a forum for hearing and investigating complaints, but considerable scepticism about the legitimacy of complaints actually lodged with the Commissioner. Only one in 10 doctors agreed with the statement that most complaints were warranted, and half disagreed with the statement that most complaints are about errors and actual wrongdoing.

The second of these opinions is partially correct as a matter of law. The Commissioner’s obligations extend beyond classic violations of quality. New Zealand law sets forth a variety of other rights, including rights to be treated with respect and to be free from discrimination or financial exploitation. Perceived breaches of all such rights are legitimate bases for complaint.¹⁶

Nonetheless, our analysis suggests that doctors’ attitudes about the reasonableness of complaints are at odds with reality, at least among the subset of complaints related to public hospital care. Nearly two thirds of the complainants in the study sample had experienced adverse events, of which 79% were preventable and 60% involved permanent injury or death; 93% of the adverse events were either preventable or serious injuries.

It is incorrect to interpret these results as evidence that complaints are usually triggered by doctors’ wrongdoing. The causes of adverse events in medicine are often multifactorial, involving a complex interplay between individual and system factors. On the other hand, the prevalence of adverse events—especially preventable and serious ones—refutes the notion that most complaints over quality of care are groundless.

Complaint rates

Complaint rates were low. The finding that only 0.4% of adverse events and 4% of serious preventable adverse events triggered complaints is consistent with crude estimates of 0.3% (254/85 000) and 2.3% (110/4800) obtained using the number of complaints as the numerator and an extrapolation of adverse event rates to the national level as the denominator.

In an earlier analysis of the same survey data, Cunningham and colleagues¹⁷ found that one in three doctors had experienced a complaint at some time in their career; the study estimated a complaint rate of 5.7% per doctor per year and used these data to conclude that there was a “high incidence” of complaints. From the perspective of busy practitioners, this may be so. But from a health policy perspective, the prevalence of poor quality care rather than the number of physicians is the appropriate baseline against which to measure complaint frequency. Using this metric, we reach the opposite conclusion: complaints are rare in the sense that the vast majority of preventable adverse events never trigger one.

Why are complaint rates so low? Felstiner and colleagues’ conceptualisation of the evolution of disputes through a process of “naming, blaming, and claiming” helps organise the most likely explanations.¹⁸ Firstly, many patients may not be aware that they have sustained an injury from medical care. Disentangling medical injury from the progression of underlying illness is not straightforward, especially in the inpatient setting where that illness may be severe.

Secondly, patients may recognise their injury but be unaware of the Commissioner’s services, or unwilling to commit the time and energy needed to take action. In theory, the complaints process should pose few barriers—patients can lodge a complaint by writing a letter or by making a free phone call (0800 11 22 33); a lawyer’s assistance is not required. However, in perception or reality, it may not be straightforward for some aggrieved patients. The complaints process has been described as “confusing, cumbersome, difficult to access, and costly, both financially and emotionally.”¹⁹ Moreover, health professionals are provided with a copy of the letter of the complaint, including the patient’s name, so some patients may hold back out of concern that such action will bring tension into their relationship with their doctor.

Thirdly, despite both recognising their injury and understanding the complaint option, some patients may simply adopt the attitude that “what’s done can’t be undone”, put the event down to bad luck, and move on. Alternatively, they may elect to take action, but not with the Commissioner. Several other options are available to injured patients seeking redress or accountability following an adverse event. Monetary compensation is available through the national no-fault compensation scheme. Patients seeking an apology, an explanation, or system change to protect others from suffering a similar harm can have those interests met by bringing their concerns directly to the attention of their healthcare provider by using free independent patient advocacy services, or by lodging a complaint with the hospital.

Disparities in use of complaints

A study by Tapper and colleagues²⁰ of complaints against surgeons found that they were more likely to be brought by women and patients in older age groups. Again, analyses of complaints that do not calibrate their frequency to the underlying rate and patterns of injury can be misleading. Women and the elderly are leading users of the healthcare system;²¹ they are also over-represented among injured patients.⁵ Calibrating complaints to baseline data on injury, we found no sex differences in complaint behaviour. Elderly patients, on the other hand, were one quarter as likely as their younger counterparts to complain following an adverse event.

Similarly, socioeconomic disparities in complaint behaviour are not readily apparent from a discrete analysis of complaint data. The incidence of complainants is fairly evenly distributed across the five deprivation quintiles, but multivariate regression analysis controlling for the presence and severity of injury showed that patients from the most socioeconomically deprived areas were significantly less likely to complain. These results echo studies of malpractice litigation in the United States in which both old age^22,23 and lower socioeconomic status^23,24 have been correlated with lower propensity to sue.

Severity of injury

The strong relationship we identified between severity of injury and propensity to take legal action is also consistent with findings from medicolegal research from the United States.^23,24 Although the Commissioner’s complaint processes attract only a small proportion of adverse events, the odds that an injury will materialise as a complaint increase steeply with severity of the injury; the odds are also substantially greater if the injury is preventable. There is thus a clear “bias” in the severity and types of injuries that come before the Commissioner. Complaints data should not be construed as representative of general patterns of medical injury. On the other hand, the skew towards serious and preventable events is precisely what policymakers might hope for from a system whose goals are to protect consumers from the most serious safety hazards and identify opportunities for quality improvement.

Limitations of study

Our study has several limitations. Firstly, complaints relating to episodes of care in 1998 may have been (or might be) lodged later than 30 June 2004, the date our complaints sample was drawn, although this is unlikely because virtually all complaints are filed within 2 years of the date of the alleged injury (mean 10 months, median 5 months). The 5.5 year window for complaints that we allowed is therefore conservative.

Secondly, estimating adverse event rates through medical record review has recognised limitations.^25,26 In our review of complaint files, inter-reviewer agreement on the preventability judgment was only fair. To the extent that complaints were judged not to involve preventable adverse events and they did, our regression analysis will underestimate the predictive value of preventability in complaining.

Thirdly, several of the other variables used in our analyses were suboptimal. Ethnicity data were missing for one quarter of complainants, and misclassification of ethnicity is a recognised problem.²⁷ The use by the New Zealand Index of Deprivation of small area-based measures to assign socioeconomic characteristics at the individual level creates the potential for measurement error.¹⁰ The direction and magnitude of potential biases stemming from these data limitations are unknown, but we know of no reason why they would differ systematically between complainants and non-complainants and thus affect the results of our analyses.

Conclusion

Given the absence of tort remedies and the availability of a free independent complaints mechanism, it might be expected that patients in New Zealand would frequently lodge complaints following adverse events. Indeed, some physicians in New Zealand feel under siege by complaints processes and the medicolegal environment has been described as one of the “most hostile” in the world.²⁸ Yet when complaints to the Commissioner are set against the underlying rate of injury, it becomes apparent that they represent only the tip of an iceberg of adverse events. That tip misrepresents what is beneath the surface in two important ways. Firstly, the relatively low propensity to complain among patients who are elderly, socioeconomically deprived, or of Pacific ethnicity suggests troubling disparities in access to and utilisation of complaints processes. Further research is required to better understand and address these disparities. Secondly, the probability of a complaint increases steeply with severity of injury, and preventable events are much more likely to lead to a complaint than unpreventable ones. In this regard, complaints offer a valuable portal for observing serious threats to patient safety and may facilitate efforts to improve quality.