Methods

The audit loop which identifies those components presented in this paper is shown in fig 1.

SAMPLING AND DATA COLLECTION

All trusts in England, Wales, and Northern Ireland were invited to participate in the audit through a letter to the Chief Executive. They were asked to identify a lead clinician and clinical audit coordinator to be responsible for the audit locally. Auditors were trust staff of any discipline involved in the management of stroke or from the clinical audit department.

An organisational audit describing the services available within the trust was completed in each audit round. The process audit included the first 40 cases admitted during a 3 month time frame (between 1 January and 31 March 1998 and between 1 August and 31 October 1999). The first audit included 6 month follow up data about care provided in the community after discharge and therefore data were collected in October 1998. For funding reasons the second audit did not include 6 month follow up data and therefore trusts submitted data in December 1999.

AUDIT TOOL

The audit tool was developed at the Royal College of Physicians Clinical Effectiveness and Evaluation Unit under the guidance of the Intercollegiate Stroke Working Party.¹⁴ It was based on standards for service organisation and care agreed by the different professional colleges and patients' organisations represented. The standards were confirmed by a formal consensus Delphi survey¹⁵ of 90 individual experts in stroke and 10 patients' organisations external to the Working Party. The audit proformas were piloted in 14 volunteer trusts including teaching, district general and community rehabilitation hospitals, and in two general practices. Detailed explanatory notes were developed to help in the completion of the audit proformas by different members of the multidisciplinary team. Some examples of the audit standards and the explanatory notes for both the organisational audit and the process audit are given in appendix 1. Examples of the audit questions are provided in appendix 2. To enable data to be scanned the questionnaire was designed in conjunction with optical mark recognition software (Teleform) which works with the SPSS statistical package.

The audit covered organisation and facilities for treating stroke (table 1), processes of care (table 2), and case mix and outcome of admitted patients.⁸ Organisational factors which had been identified by the Cochrane reviewers as being important to the success of stroke care³ such as staff expertise, team meetings, staff training, and involvement of carers formed the basis of the organisational audit. Case mix measures were included to help identify systematic differences between participating trusts in the types of patients treated that might explain differences in the organisation or process of care. Measures included premorbid disability as assessed by the Barthel score, age, and stroke type. Outcome measures included 7 and 30 day mortality, length of hospital stay, accommodation before and after discharge, and Barthel score on discharge.

For each process question the auditor was asked to identify whether the standard had been met or whether predefined characteristics about the patient meant that the standard was not applicable, in which case the response was “No, but ...”—for example, if patients were unconscious they could not be assessed for speech problems. An inter-rater reliability study during the first audit round showed moderate to good agreement between auditors of varying disciplines.⁹

First round of audit

Details of the first round of the audit conducted in 1998 have been described previously.¹⁰¹¹ The Chief Executives and lead clinicians for all trusts treating patients with acute stroke in England, Wales, and Northern Ireland were invited to participate in the audit, providing data on the organisation of stroke services and a retrospective analysis of the notes of up to 40 patients admitted with stroke to the trust during the previous 6 months. 197 trusts (80% of all trusts) participated, reporting data on 6894 patients. Results for each trust were returned within 3 months of data submission, benchmarked against the national averages.

Confidential reports showing individual trust data benchmarked against national averages were sent to the participating trusts after each audit. In both rounds the audit results were returned to the lead clinician and the audit department in participating trusts within 3 months of data being submitted. Regional Directors of Public Health received a report showing regional variations in care and outcome, and publication of the national picture¹⁰ after the first audit provoked considerable media interest both in the lay press and the journals of professions allied to medicine.

Following the first audit round a series of 17 multidisciplinary workshops was held around the country. Between 50 and 100 clinicians attended each meeting, in all cases including representatives from the medical, nursing, and therapy professions. At these the main messages from the national and regional findings were presented, followed by local presentations on action taken since the audit. Breakout groups were also held to discuss key issues in changing practice—for example, development of a specialist stroke service, involvement of patients in the design of services, improving documentation, use of standardised assessment measures, sharing specialist skills—to stimulate discussion among local clinicians about possible strategies for implementing change.

The principal changes being implemented were as follows:

• Care in stroke units was being broadened to develop comprehensive coordinated stroke services from one-stop clinics for transient ischaemic attacks to support and follow up in the community. Many were using specialist nurse coordinators to effect these changes.

• Many centres were developing multidisciplinary records and integrated care pathways. There was consensus that collaboration at a regional level would have been beneficial as individual hospitals were spending a lot of time developing their own systems.

• There was a widespread view that a range of agreed assessment measures would be useful and that, to allow comparisons, some national guidance on this would be helpful.

• The need for sharing skills between professionals working with stroke patients was forming the basis for staff training in some units—for example, screening for swallowing disorders should be performed by nurses and doctors (with appropriate training) for appropriate referral to speech and language therapists for in-depth assessment. To identify the reasons for changes between the first and second round, a questionnaire asking about changes in the organisation of stroke care, staffing levels, documentation of care, and systems introduced to enhance information provision was sent to trusts shortly before the second round (box 1). Information was sought as to whether changes had occurred as a result of the audit.

Second round of audit

The methodology for the second round of the audit conducted in 1999 was identical to that used for the first round, except for adaptation of the questionnaire to remove the requirement for 6 month follow up data and the inclusion of some questions within the organisational audit form to establish staffing levels provided for stroke care within the trust sites.

In a separate questionnaire conducted after the second round, trusts were asked for their views as to how much more or less representative of all eligible cases their sample was in the second round compared with the first. Since data collection in the second audit was done only 6 weeks after the last admission, some cases admitted during the audit time frame might still be in hospital. If trusts coded their cases some weeks after discharge, then this could impact on patient identification as well as gaining access to notes when patients attended outpatients or visited other departments.

DATA ANALYSIS

The intention was to describe changes in the organisation and processes of care. Relevant data came from trusts taking part in both national audits of 1998 and 1999 and from the supplementary questionnaires. Changes in regional stroke unit provision were also described. In addition, the extent of selection bias was explored by comparing the results with those of the first audit and by comparing the results of the supplementary questionnaire on changes between trusts that participated and those that did not participate in the second round.

The presentation of the results is primarily descriptive through the use of tables and percentages. The percentage change in results between audits for different subsets of clinical standards is presented as median (range). Inferential methods of analysis such as tests of significance and confidence intervals to compare the results of the two audits are of limited value for the organisational audits as these come close to being a census of the organisation of care of all trusts in England, Wales, and Northern Ireland. Such methods are of more relevance to the process audits, given the differences in intake of stroke patients by sites over time, since audit patients represent a much lower proportion of all stroke patients admitted to all trusts during the period of data collection. The confidence intervals in table 4 present a range of plausible differences in results between audits, but they remain unadjusted for these (unknown) sampling fractions.

Results

AUDIT SAMPLE

A total of 197 trusts took part in the first round of the audit and, of these, 157 (64% of eligible trusts in England, Wales and Northern Ireland in 1999) submitted data for both audits—5589 cases in 1998 and 5375 cases in 1999. The results from the 1998 audit have been reported previously.¹⁰ Some of these data are repeated here to provide the baseline from which to interpret the data from the 1999 audit.

COMPARISON OF TRUSTS PARTICIPATING IN THE FIRST OR BOTH ROUNDS OF THE AUDIT

A comparative analysis of first round results was performed to see if there was any suggestion of selection bias in those volunteering a second time; the analysis indicated that this was not a significant issue. The 40 trusts that did not participate in the second round had a similar case mix and level of performance in their first round to the 157 trusts participating in both rounds.

The two groups of trusts had categorical results similar to within 3% with respect to sex, age category, mortality within 7 and 30 days, Barthel score category before stroke and after discharge, accommodation before stroke and after discharge, length of stay, and whether more than 50% of a patient's stay was in a stroke unit, a rehabilitation unit, or in a general ward. Information for organisational domains (table 1) enabled domain scores and a composite score to be computed. The mean domain scores for the two groups were similar, as were the mean (SD) composite scores for trusts participating in both rounds (62 (17)) and for those participating only in the first round (67 (15)). Three quarters (n=53) of compliance rates to the 71 process standards were within 5% of one another, with 63% (n=45) being within 3%. Trusts taking part in both audits tended to have lower compliance rates for clinical diagnosis (9%, 10%, 11% lower for the three standards relating to likely site of cerebral lesion, type of lesion, and whether a brain scan was performed within 24 hours); for whether the patient became the responsibility of a multidisciplinary team of clinicians specialising in stroke (11% lower), and for whether the GP was informed by the day of discharge or by the day following death (13% lower). On the other hand, these trusts tended to have higher compliance rates in some aspects of management planning (9% higher for agreement of rehabilitation goals by multidisciplinary team, 16% higher for evaluation of patients progress on target dates for the team rehabilitation meeting).

ORGANISATIONAL AUDIT RESULTS FOR PARTICIPANTS IN BOTH ROUNDS

Of the 38 standards measured, 29 had improved between the 1998 and 1999 audits (range 1–20%, median 5%). Seven standards had deteriorated, mainly in the areas of provision for staff knowledge and skills for the various therapies (occupational therapy, physiotherapy, and speech and language therapy), availability of guidelines on care of pressure area, access to psychiatric services, and social worker support. Two had remained unchanged. The area where change was most frequently seen was in the introduction of formal assessment measures and improvements in the use of multidisciplinary records. The results for the 14 items considered by the authors to be of greatest interest to practising clinicians are shown in table 3.

PROCESS AUDIT RESULTS FOR PARTICIPANTS IN BOTH ROUNDS

Of the 71 standards measured, 64 had improved (range between trusts 1–20%, median 8%) with two remaining unchanged and five deteriorating (range 1–20%, median 4%). For the main domains covering acute assessment and care, rehabilitation, secondary prevention, and discharge planning, improvements were evident between 1998 and 1999. The results for the 23 items considered by the authors to be of greatest clinical interest are shown in table 4. Standards for assessment, rehabilitation, and discharge planning improved equally on stroke units and general wards, but in many aspects remained poor (41% formal cognitive assessment, 46% weighed once during admission, 67% physiotherapy assessment within 72 hours, 24% plan documented for mood disturbance, 36% carers' needs assessed separately). In the 1999 audit, as in the 1998 audit,¹⁰ compliance with the standards was significantly more likely if the patients were managed on a stroke unit than in alternative settings. For example, between 1998 and 1999 the overall proportion of patients screened for dysphagia increased from 54% to 66%. For patients managed on a stroke unit the increase was from 67% to 79%, while in those managed on general or rehabilitation wards only 61% were being screened by 1999. Failure to identify patients with swallowing problems puts them at risk of aspiration pneumonia which is one of the commonest causes of death following stroke. The proportion of people having a brain scan where there were clear clinical indications that this was required urgently rose from 56% to 68% on stroke units and from 48% to 61% in other ward settings between the two audits. Adequate quality of care would require near to 100% compliance with all these standards.

REASONS FOR CHANGE

Table 5 lists the changes in service organisation that had occurred between the two cycles of audit, indicating how many of these changes had occurred as a direct result of the audit process. The most frequently cited initiatives that had resulted from the audit process were changes in documentation (30 trusts introducing interdisciplinary care pathways and 39 using multidisciplinary notes) and information provision (52 trusts improving their services). Among the trusts participating in the audit, 31 new stroke units had opened, eight of which were considered to have occurred as a result of participating in the audit process. The response rate to this supplementary questionnaire was 100% for those trusts who participated in both rounds.

The extent of selection bias was explored by comparing responses to this questionnaire between trusts that participated in the second audit round and those that did not. These results also suggest that selection bias was not a significant factor. Of those 40 trusts that did not participate in the second round, 27 responded and reported improvements including: new stroke unit in four (two as a result of first audit), stroke team started in four (two as a result of first audit), specialist nurse for stroke in five (two as a result of first audit), interdisciplinary care pathways in eight (four as a result of first audit), multidisciplinary documentation in 10 (five as a result of first audit), better social work involvement in four (two as a result of first audit), and improved information for patients and relatives in eight (six as a result of first audit). There were very few instances of negative change.

SAMPLE REPRESENTATIVENESS

In response to the later questionnaire about the representativeness of the sample (response rate 94% of trusts participating in both rounds), 75 of 148 (50%) stated that their patient sample was “about the same” in both rounds, 27 (18%) felt it was “more representative”, 36 (24%) “less representative”, and 10 did not know. The overall neutrality of these results adds greater confidence to the validity of the comparisons made between the two audits.

REGIONAL VARIATIONS IN STROKE UNIT PROVISION

Results showing major regional variations in relation to stroke unit provision are given in table 6. The proportion managed for more than 50% of their stay on a stroke unit in 1998 ranged from 10% to 37% (overall 19%). In 1999 this figure ranged from 16% to 52%. All the 10 regions studied increased the proportion managed in stroke units and in eight the proportion managed on general wards also fell. In six regions much of the increase in care in the stroke unit appeared to be at the cost of care on general rehabilitation units, with the proportion being managed on such units falling by as much as 12%.

Discussion

The main findings of this study are that the standards of care for patients with stroke in England, Wales, and Northern Ireland have improved in nearly all of the domains audited between 1998 and 1999. However, it has highlighted many areas where considerable changes are still needed. It has shown that national audit, when combined with an active implementation strategy, can be a powerful force for change.

THE AUDIT PROCESS

This is the first large scale national audit of stroke in the UK to complete the audit cycle. It involved a large proportion of hospitals providing care for patients with stroke in England, Wales, and Northern Ireland and covered service organisation and aspects of care including acute management, rehabilitation, secondary prevention, and discharge planning.

For an audit tool to be of value it needs to be reliable, sensitive to change, feasible to complete, and acceptable to the clinicians involved. The audit tool used here was extensively piloted during the development phase. An inter-rater reliability study showed that this exercise improved data quality to levels as good as any of the large published audit studies.⁹ The tool was able to detect a series of improvements in both the organisation and the process of care, consistent with the findings in the change questionnaire. It would therefore seem to be sensitive to change. The audit tool does appear to identify poorly performing units and centres of excellence and therefore may be used to direct guidance for service development and clinical governance.

The case note audit was feasible to complete but was reported by clinicians participating in the regional workshops on the audit to be time consuming, taking at least 30 minutes per case. With each trust being asked to complete up to 40 cases, this was clearly a significant burden. There was a need to consult the notes of several professionals including, for the 1998 audit, obtaining information from GPs about care provided for the 6 months after discharge. The audit was also cross-sectoral, involving both health and social services. An alternative to this form of wide ranging audit would be to identify those audit items that were most discriminating of good and bad units and to produce a shortened version. While this would provide information needed for clinical governance, it would reduce the value of the audit to local clinicians wanting a comprehensive description of their services to be able to have clear direction as to the areas most in need of change. Using electronic patient records to automatically collect data for audit prospectively is the obvious development that needs to be made, but the technology available has not yet been shown to be feasible on a large scale. The audit received broad support from the clinicians who took part, and many instances were reported of the audit results providing the ammunition needed to stimulate change. The high proportion of trusts voluntarily participating in the second round also suggests that it was considered to be of value.

STANDARDS OF CARE

While not all the developments in stroke care observed between the two audits can be attributed to the audit, the results of the change questionnaire suggest that it was a significant factor in many cases. Thirty one new stroke units had opened in the 18 months between the audits and the proportion of patients receiving most of their treatment on a stroke unit rose from 19% to 26% in trusts participating in both audits. The organisational audit indicated that an extra 12 consultants had specific responsibility for stroke (table 3). The use of interdisciplinary care pathways and multidisciplinary documentation had increased significantly, often as a direct result of the audit. The standards of care, as measured by the audit, are considerably better for patients on stroke units than in other settings including generic rehabilitation units.¹⁰ The single most important feature of stroke care must remain the continuing development of stroke units of sufficient size to enable virtually all patients to receive the benefit. Although standards of care off stroke units are worse, it is nevertheless encouraging that improvements of a similar magnitude to those seen on the stroke units were observed between the two audits.

USING AUDIT TO EFFECT CHANGE

Encouraging change to occur requires effective dissemination and feedback of results. Trusts indicated that the confidential report detailing their performance against the national benchmark was valuable. Similarly, feedback from the 17 regional workshops between the audit rounds suggested that they were a stimulating arena for sharing ideas on good practice at a local level. Requests from participants for these to be repeated after the second round were acceded to as an important means of promulgating the recently published national clinical guidelines.¹⁶ We cannot prove that change would not have occurred with feedback of results alone, but we believe that regional workshops were an important additional factor in giving local clinicians new ideas for change and the confidence to promote those ideas.

USING GUIDELINES TO EFFECT CHANGE

The King's Fund first produced stroke guidelines,¹⁷ with more recent contributions from the Scottish Intercollegiate Guidelines Network^18–21 and the Royal College of Physicians of Edinburgh consensus conferences on medical management of stroke.²² The Royal College of Physicians of London published the national clinical guidelines for stroke in 2000,¹⁶ providing an overview of the evidence base linking the process of care with outcome. Extracts from the guidelines are given in box 2 to illustrate some of the key recommendations and format of the document. Development of the guidelines was carried out in parallel with the audit. They were, however, not published until after completion of the second cycle of the audit and would therefore not have been directly responsible for changes in service provision found in this audit. The impact of the introduction of the national guidelines will be explored in association with future cycles of national audit.

Stroke care in the UK is improving but there is evidence that outcomes are worse than in other countries in Europe, both in mortality and morbidity.²³ Continuing to improve the process of care, particularly by increasing the proportion of patients managed on specialist stroke units, must become a top priority. Solving the problem of the major regional variations in the standards and outcomes of care is also fundamental in overcoming the perception that health care in the UK is second rate.

In conclusion, comparative national audit does appear to have a positive effect by informing and cajoling healthcare providers into modifying entrenched and outdated systems of care. Further national comparative data collection has the potential to drive improvements in care, especially if linked to clinical governance, the National Service Frameworks, and the modernisation agenda. Measurement of processes of care that research has shown are linked to outcome is the only realistic way of assessing the effectiveness of individual clinical services. Few hospitals will have sufficient patients to enable valid measurement of outcome, particularly with a disease such as stroke that is as variable in its pathology and clinical presentation. Investment in process audit therefore seems justifiable.