Methods

INCLUSION AND EXCLUSION CRITERIA

The search included all quality system activities in nursing homes but excluded research on the effectiveness of treatment interventions. The scope of the study was limited to publications in the English, Dutch and German languages. To answer the research question, only empirical studies of quality system activities in nursing homes were included. Furthermore, randomised controlled trials could not really be expected and it had been decided a priori that other research designs would also be included such as pre-test/post-test studies, time series, and post-test only studies. Studies which included no description of the impact of the quality system activity on process or outcome measures (resident satisfaction and health outcomes of residents) were excluded.

LITERATURE SEARCH

Literature pertaining to the effectiveness of quality systems in nursing homes was identified by several methods (table 1).

The first searches in Medline, Cumulative Index to Nursing & Allied Health Literature (CINAHL) journal base, Cochrane Library controlled trials register, Cochrane EPOC reviews register, and the Netherlands Institute of Health Services Research (NIVEL) literature database for the years 1985 to the end of 1997 were based on a combination of the free text keywords “quality systems”, “quality assurance”, “quality improvement” or “quality control” and the free text keywords “long term care” or “nursing home”. In addition, specialist journals that could not been found in one of the above mentioned journal bases were searched by hand. Secondly, additional information was obtained from the references of identified articles and from colleagues (snowball method).

The selected literature was analysed by the authors on methodological characteristics (research design, sample size, quality management activity, and measurement instruments) and process and outcome measures resulting from implementation of the quality management. The selected articles were grouped by the authors according to the five focal areas of a quality system (table 2).¹⁵

Results

SELECTED STUDIES

The first search resulted in 231 relevant publications but, after analysis, only six met the inclusion and exclusion criteria. The other reviewed articles were not selected for the following reasons: they only described the quality systems activity but not the effects (n=101), the setting appeared to be a hospital or home care agency (n=20), or the author(s) only expressed an opinion on quality improvement or long term care (n=99). Five references could not be traced. An additional 15 articles were identified from the references of articles and from colleagues, four of which were published shortly after the search period.

Of the 21 selected studies, two were based in the UK, one in Canada, and the remainder in the USA. Process and outcome measures were used to evaluate effectiveness and, in three studies, both measures had been used. Of the 13 studies which examined the effects on the process of care, as measured by the development of policies for care, all reported some improvements. All of the 11 studies which assessed resident outcome also reported some improvements. The results are shown in table 3 in which the controlled trials are indicated.

RESEARCH DESIGN OF THE SELECTED STUDIES

The research design and the data differed between the studies. Controlled studies described: (1) the effectiveness of specific educational programmes^20,21; (2) the effectiveness of two different facility and resident assessment processes on the overall quality of care and the detection of problems²²; and (3) the impact of working with quality assurance cycles.²³

In 10 studies the researchers used a pre-test/post-test design with no control groups in which the interval between the pre-test and the post-test period was less than 1 year.^5,24–32 There were two longitudinal studies in nursing homes with a follow up of several years,^33,34 and four studies with a post-test design only.^35–38 The number of participating nursing homes involved in the selected studies ranged from one to 268, with 13 studies of 1–16 nursing homes, two of 60 nursing homes, and six with more than 200 participating nursing homes.

SELECTED STUDIES AND THE VARIOUS FOCAL AREAS

The controlled trials were found in the focal areas “process improvement by quality improvement procedures” and “human resources management”. There were no studies in the “quality assurance documents” focal area. One study described the impact of a quality system.³³ In the focal area “process control based on standards” five studies evaluated the effectiveness of the implementation of a new guideline—for example, for the prevention of pressure ulcers, the adequate use and reduction of indwelling catheters, and the reduction of psychotropic drug use—all of which reported a decline in the prevalence of adverse events.^{25,26,34,36,37}

The studies related to the focal area “process improvement by quality improvement procedures” described three different activities: (1) feedback of information on patient related quality indicators, (2) a resident assessment instrument (RAI) to analyse the needs of residents and support the care planning process, and (3) clinical audits. Characteristically, all three activities made use of a quality cycle whereby caregivers assess, evaluate, and improve the quality of care when necessary. Information on the present situation was obtained and subsequently compared with the desired situation. If there was any discrepancy between the desired and the actual situation, changes were implemented. The results of one controlled study²³ showed that there was a decrease in the occurrence of a number of adverse outcomes (constipation and hazardous mobility), but for other outcome measures (urinary incontinence and potential skin breakdown) the prevalence of adverse outcomes was the same as in the control group. Another controlled study²² showed that the assessment method helped to detect problems and that many process improvements would not occur without external review. In the other studies the authors admit that it is not entirely clear to what extent the results can be attributed to the intervention since there was no control group and various other changes were also made at the time. Three studies in this focal area made use of internal audits to improve the satisfaction and health status of residents. The effectiveness, for example, of the Continuous Assessment Review and Evaluation (CARE) scheme has been investigated in pilot studies.^5,39 The audit process was applied to the procedures in nine mainly clinical fields—for example, decubitus, urine incontinence, drug consumption, and the autonomy of the resident. As a consequence of the audits, existing agreements and guidelines in institutions were improved or new ones developed. The improvement in processes was assessed on the basis of the number of institutions which had an explicit policy concerning the nine fields in question—for example, standard policies were made more comprehensive (missing components reduced from 38% to 19%).

The professionals in an institution determine the quality of the care that they provide, as well as the quality of their mutual collaboration. The improvement of expertise in the form of tuition, supervision of work, or peer review can have an influence on the quality of the care the residents receive. Of the 21 studies, four described and evaluated the implementation of training programmes for nurses and doctors. The object of the training was to reduce the number of protective measures,²⁸ the prescription of psychoactive drugs (controlled studies),^20,21 and the prevalence of incontinence in residents.²⁴ The results of the two controlled studies show that, with the help of specific training, significant improvements can be achieved in comparison with the control group.

In many countries it has already been agreed that residents can play an important part in the execution and improvement of care. One study investigated whether an ombudsman for residents in nursing homes would lead to better outcomes and more compliance with standards.³⁵ The researchers concluded that the presence of an ombudsman who can visit residents and resolve disputes on their behalf might improve the outcomes of care, but there were no indications of better compliance with standards. No control group was included in this study.

Discussion

Systematic improvement in the quality of care provided for the elderly has been high on the agenda in the USA and Europe for a number of years. The research question addressed in this review was: which quality system activities have measurably improved delivery of care in nursing homes and resident related outcomes?

Over 200 publications were identified using keywords associated with quality assurance and long term care or nursing homes, but only 21 of these described the implementation of quality system activities and their effects on the quality of care provided for residents, and only four included a control group. From these studies it can be concluded that specific training, specific resident assessment procedures, and the use of quality assessment cycles with the assistance of a quality assessment consultant can be effective in improving the quality of care in specific aspects of the care process and certain health outcomes for residents. The results of the studies without a control group can only serve as an indication of what should be the subject of further research. The link between quality system activities at the process level and the effect on the health outcomes of residents has not yet been proved conclusively. In line with evidence-based medicine, it is important that more proof of the effectiveness of quality management is found in the near future or it might be debatable whether so much money should be spent on activities that have failed to demonstrate any improvement in the quality of care or quality of life of residents.

This considerable difference in the number of publications and the number of effect studies suggests that the effects of most initiatives are not reported or their effects are not evaluated. The fact that none of the identified studies reported predominantly negative results might imply that there is some publication bias. This means that valuable information and experience is not made available for other nursing homes, and it could therefore be beneficial to investigate and survey such unpublished activities. It is possible that, because of the search strategy, some studies evaluating quality system activities which do not mention “quality” in the title or the abstract have been missed.

In general, the studies which were identified concerned quality system activities such as the implementation of guidelines, providing feedback, assessment of the needs of residents by means of care planning, internal audits and tuition, and an ombudsman for residents. One study reported on the effects of a quality system. The opinion of the residents is apparently seldom used to evaluate the effectiveness of a quality system. In view of the high prevalence of demented people in nursing homes, this approach is difficult to implement. One possibility is to ask the opinion of relatives (proxies), but it is questionable whether relatives can really speak from the perspective of the residents. Participant observation of demented residents could therefore be an adequate though time consuming method of measuring satisfaction with the quality of care. However, there are studies which have measured “consumer satisfaction”,⁴⁰ but not in relation to quality system activities.

A number of key issues emerge from critical appraisal of the selected studies. In most cases the design of the studies is such that it is not possible to attribute the results entirely to the newly implemented quality systems activity—for example, studies on the effects of the RAI.^29–31 The results are often presented without detailed quantification. Moreover, the intervention is often not described in sufficient detail to allow an institution manager or carer to repeat it in order to achieve the same effects. There is also often no indication of which factors or elements were essential in achieving the (positive) effects.

It is concluded from the literature that, at present, there is no clear answer to the question as to which quality system activity should be used in nursing homes to improve the care provided for residents. From the controlled studies there is some proof that activities which are directly associated with the ability of the professional, such as training and guidelines, can influence the outcomes at patient level. It could be that carers in nursing homes lack training in the specific knowledge and abilities which are needed in caring for the increasing number of elderly residents who have more disabilities and greater comorbidity. Additional training and the use of guidelines can reduce the uncertainty of carers. In addition, there are indications that the RAI has some positive effects on the health outcomes of residents, but the study design does not allow further conclusions to be drawn. The RAI provides carers with a systematic approach that incorporates treatment suggestions and guidelines. In order to obtain more certainty about the effectiveness of quality system activities, future studies should at least include a pre-test/post-test design and a control group. The results show that, for some quality system activities, it is possible to set up a randomised controlled trial. It seems more difficult, in practice, to design a randomised controlled study which evaluates a more complex integral quality system. Future research into the effectiveness of quality systems could therefore also be based on a multivariate or multilevel approach—for example, to determine initially which resident related outcomes (corrected for case mix of residents) vary among institutions and subsequently to investigate in more detail what the differences between the institutions are with regard to structure and process characteristics. This would make it possible to identify influential organisational and environmental factors, or patterns of factors, and to determine which elements of a quality system are most effective and in which situations or circumstances. Another possibility is a combination of process and effect evaluation with randomisation of nursing homes instead of residents. Finally, it might also be necessary to evaluate quality systems with a combination of qualitative and quantitative research methods. Examples of qualitative methods include participant observations, site visits combined with interviews, or an accreditation procedure. The data gathered by these methods could be combined and analysed at a more conceptual level to help us understand the mechanisms between the organisation structure, the care process, and resident outcomes.