Objectives—To develop, validate, and test the reliability of an audit instrument that measures the extent to which patient records describe important aspects of nursing care.
Material—Twenty records from each of three hospital wards were collected and audited. The auditors were registered nurses with a knowledge of nursing documentation in accordance with the VIPS model—a model designed to structure nursing documentation. (VIPS is an acronym formed from the Swedish words for wellbeing, integrity, prevention, and security.)
Methods—An audit instrument was developed by determining specific criteria to be met. The audit questions were aimed at revealing the content of the patient for nursing assessment, nursing diagnosis, planned interventions, and outcome. Each of the 60 records was reviewed by the three auditors independently and the reliability of the instrument was tested by calculating the inter-rater reliability coefficient. Content validity was tested by using an expert panel and calculating the content validity ratio. The criterion related validity was estimated by the correlation between the score of the Cat-ch-Ing instrument and the score of an earlier developed and used audit instrument. The results were then tested by using Pearson's correlation coefficient.
Results—The new audit instrument, named Cat-ch-Ing, consists of 17 questions designed to judge the nursing documentation. Both quantity and quality variables are judged on a rating scale from zero to three, with a maximum score of 80. The inter-rater reliability coefficients were 0.98, 0.98, and 0.92, respectively for each group of 20 records, the content validity ratio ranged between 0.20 and 1.0 and the criterion related validity showed a significant correlation of r = 0.68 (p< 0.0001, 95% CI 0.57 to 0.76) between the two audit instruments.
Conclusion—The Cat-ch-Ing instrument has proved to be a valid and reliable audit instrument for nursing records when the VIPS model is used as the basis of the documentation.
(Quality in Health Care 2000;9:6–13)
- audit instrument
- nursing care plans
- quality assurance
Statistics from Altmetric.com
The patient record is a principal source of information in which the nursing documentation of patient care is an essential part. Traditionally, nurses have written down their performance of the medical interventions or observations ordered by the physician. The rationale behind this was to show that the instructions had been adhered to and to inform other nurses or physicians.
Recently, the nursing profession has moved towards a more independent practice with a clear recognition of nursing care. With increasing recognition of these nursing components the documentation of nursing care must include not only timely and accurate recording of the performed interventions—medical and nursing—but also the decision process, explaining and evaluating why a specific nursing action was chosen.
In 1967, Yura and Walsh first described the nursing process model (fig 1) as a structured, problem solving approach to nursing practice and its evaluation.1 This process originally contained four steps: assessment, planning, implementation, and evaluation. In a later version, the nursing diagnosis was included. The nursing process model is a central and widely accepted concept,2 both for nursing practice and documentation. The nursing process model is based on the scientific approach of investigation and goal oriented action. The primary purpose of the model is to relate individualised nursing care to the individual patient rather than generalised care based on routines.
The nursing care plan is an essential tool in the delivery of modern nursing care. The care plan is a document containing the three nuclear parts of the nursing process model: a nursing diagnosis, describing the problem or need; the aim of the nursing care; and the interventions that have been planned to achieve the aim. The purpose of the nursing care plan is to have a reference easily accessible in the clinical setting that describes the patient's needs and wishes and the nursing interventions that have been planned for the patient. It is used to ascertain the continuity of care among caregivers. The nursing care plan is part of the permanent patient record.
Since 1980, major health related organisations and some western countries have begun to develop standards, laws, and regulations stating that the nursing process should be included with nursing documentation. The World Health Organisation,3 the International Council of Nursing,4 the American Joint Commission on Accreditation of Hospital Nursing Service Standards,5 and the United Kingdom Central Council6 all promoted the use of the nursing process in nursing care. The Swedish law on this subject was passed in 1986,7 and was further clarified specifically for nursing by the National Board of Health and Welfare in 1993 (box 1).
Box 1. Regulations about nursing documentation as stipulated by the Swedish National Board of Health and Welfare
Regulations about nursing documentation
“The patient record shall include a distinct and clear nursing documentation. The nursing documentation shall, from the patient's individual needs, describe the planning, implementation and effects of the nursing care. The documentation shall be designed in such a way that it contributes to secure patient safety and provides a basis for continuous evaluation and revision of nursing interventions. The nursing care shall be summarised in a patient discharge note at the time of discharge.”8 (Authors' translation.)
The development of written care plans has been slow, however, and in Sweden nurses have only recently started to produce a more structured documentation of nursing. Nurses continue to document care retrospectively rather than document prospective care. Ehnfors showed in 1993 that 90% of the audited patient records lacked identified nursing problems, goals, and nursing discharge notes.9 In two thirds of the records, planned interventions were not stated. In 1996, Nordström and Gardulf stated that the nursing assessment was insufficiently described in 60% of records10; only 10% contained identified nursing problems and goals; and less than 45% of the records contained planned nursing interventions. Even in 1999 Ehnfors and Ehrenberg showed that only one of 120 patient records contained a comprehensive description of a patient problem, as prescribed by Swedish law.11
Benefits of documentation
The main benefit of the documentation is improvement of the structured communication between healthcare professionals to ensure the continuity of individually planned patient care. Without an individualised care plan, nursing care tends to become fragmentary and based predominantly on institutional routine and schedules. The care plan defines the focus of nursing care not only to the nursing staff but also to the patient and his relatives.12 By documenting the agreement between patient and nurse, an opportunity is provided for the patient to participate in the decision making about his own care.13, 14 Moreover, the documentation of expert nursing provides an important source of knowledge to the novice nurse and a potential instigation of the further development of nursing theory.2 The care plan yields criteria for reviewing and evaluating care, financial reimbursement,12 and staffing. Furthermore, a correlation between care plans and positive patient outcomes, such as a reduced stay in hospital, has been described.15
In 1992, a new documentation model was developed and tested by Ehnfors, Thorell-Ekstrand, and Ehrenberg.16, 17 The model is called VIPS, an acronym formed from the Swedish words for wellbeing, integrity, prevention, and security, which are seen as the major goals of nursing care (fig 2). This model is designed to be used in the documentation of the nursing process and therefore includes a nursing care plan. The model also includes a nursing discharge note. The purpose of the model is to guide the nurse in the sequences of assessment, problem identification, aim, planning of interventions, implementation, and evaluation of results and thereby to make nursing documentation structured, adequate, and easy to use in clinical care.
In the VIPS model, 13 keywords are used for classifying the information collected by the nurse about the patient's situation and status into categories, for example communication, nutrition, and psychosocial status. Ten keywords classify the nursing interventions into categories such as information, support, and environment. The use of keywords simplifies information retrieval, although, to retrieve the information asked for, a consensus about definitions of categories must be reached.18 The VIPS model provides such a lexicon, in which each category, labelled by a keyword, has a definition, a description, and prototypical examples given in a manual. Keywords may be seen as a first step towards a unified nursing language for patient care.
The VIPS model has been received with interest and appreciation by nurses in Sweden and is now the most commonly taught and used model for nursing documentation in hospitals and primary health care.17
Auditing patient records
Audit has to be distinguished from traditional review (box 2). The audit of patient records may be done for several reasons. The most common reason is because the audit is part of an ongoing process of quality improvement. A clinic may have agreed to document in a certain way, possibly with a minimum data set as a standard, or wants to evaluate specific criteria for quality health care.19–21
Box 2. The difference between audit and review
“Differences between audit and traditional review:
Use of explicit criteria for measurement rather than implicit judgments
Numerical comparison of current practice patterns against these criteria
Formal identification of action required to resolve any discrepancies disclosed
Recording the process to retain information and increase impact of audit on future management.”19
Evidence exists that a continuously performed audit of patient records, combined with discussions about improvements, is one way to improve the quality of the records and to change certain behaviours of healthcare professionals.22–24 Another benefit of auditing documentation is that it makes comparisons possible over time and among wards or hospitals, provided that a reliable audit instrument is used to put a numerical value on the written content.25, 26 Audit is also used to evaluate the effects of quality management27 by identifying the necessary professional strengths and the weaknesses that need to be addressed and corrected. Craig reports that nurses using her audit tool acquired a better understanding of what was expected of them for recording of care and patient care itself, and also focused on the areas that they specifically needed to concentrate for improvement.28
It is important to differentiate between auditing records for the sake of measuring the quality of record keeping and auditing records for the sake of measuring the quality of given care. There is important criticism in the literature about the auditing of patient records for the purpose of checking patient care, the argument being that patient records do not necessarily reflect the reality of the given care.29, 30 This raises the question whether it is possible to claim that audit measures the quality of care. Whether better documentation can also influence and improve patient care is another question, not dealt with in this article. Well written records, however, may be seen as a step towards a process of quality assurance, as a structured element in the nursing care. Donabedian is careful to stress that good structure only increases the likelihood of a good process in the actual care given and that the correlation between process and outcome has yet to be shown.31
Two Swedish audit instruments have been developed earlier, one by Ehnfors9 and the other one by Gardulf and Nordström.10 Both instruments are based on the nursing process and evaluate the record in its quantitative aspect; is there documentation for each function or is there not? In addition, Ehnfors evaluates, for each patient problem, the flow of information in accordance with the nursing process. Consequently, neither of the instruments includes a qualitative evaluation of the written content in the sense of the amount of information, wording, pertinence, etc. Several other audit instruments described in international journals29, 32–36 were also inadequate with regard to the quality and quantity aspects of auditing.
Improving nursing documentation is an urgent issue. Poor documentation is an indication that further investigation is needed to judge whether or not the given care is less than optimal. Audit instruments for nursing records are therefore needed to identify poor assessment, poor structure, and the lack of a plan for the patient's care. They are also needed for evaluating the effects of interventions aimed to improve the documentation.
The aims of this study were to develop an audit tool to measure both the quantitative and the qualitative aspects of nursing documentation, based on the VIPS model, and to evaluate the validity and reliability of that tool.
DEVELOPMENT OF THE INSTRUMENT
Before the instrument was constructed by two of the authors (CB, IT-E), a set of criteria was identified to determine what questions needed to be answered about nursing documentation in the patient record. These criteria were derived from the following sources:
The Swedish law that stipulates that nursing documentation should include the steps of the nursing process as described above, the signing and dating of each entry, a minimum degree of legibility, and a nursing discharge note
The VIPS model which includes the nursing process, the use of specified keywords, the correct classification of the keywords in accordance with the user manual, and a nursing discharge note
Common hospital policies that prescribe that each patient should have a named nurse with the primary responsibility for the patient's nursing care and care plan documentation.
At this stage, 19 questions were formulated to determine whether this information was documented in the patient record. Each question was constructed to reveal both the quantity and the quality of the written content on a rating scale. A manual was designed to explain how to score each question.
The quality and quantity values were scored on a rating scale from zero to three, zero indicating “poor” and three indicating “very good”. The quantity value is expected to measure whether or not there is a written note and, if so, how much is written. For example, for the patient's nursing status, a certain minimum number of nursing areas, represented by keywords in the VIPS model and relevant to surgical care, should be described for a patient in a surgical ward. The quality value is used to measure to what degree the written notes are clear and concise, without superfluous text, and include all relevant nursing information with a correct use of language. If all notes fulfil these criteria, a full score of three is given; if more than 50% of the notes, but not all of them, fulfil the criteria, a score of two is given; if less than 50% fulfil the criteria, though some notes still fulfil the criteria, a score of one is given, etc. Furthermore, the instrument is expected to measure the extent to which it is possible to follow a patient problem through the nursing process. That is, whether the problem is properly assessed and described in a diagnosis, with the expected outcome, planned and implemented interventions, and an evaluation. The instrument was named Cat-ch-Ing.
To test usability for understanding questions and phrasing of the instrument, five patient records collected from one hospital ward were independently reviewed by three nurses using the new instrument. The instrument was revised after each of the three audits. The revisions concerned the clarification of definitions in the manual and the deletion or rephrasing of questions. Two questions were omitted, one about the evaluation of nursing care, which was already covered by other questions, and the other about the use of keywords other than those stipulated by the VIPS model. One question about the discharge note was rephrased.
TESTING OF RELIABILITY AND VALIDITY
Inter-rater reliability was tested by comparing different reviewers' total Cat-ch-Ing scores given to the same record. Twenty patient records from each of three hospital wards at a university hospital in Stockholm, Sweden were used for this part of the development. The records were selected from the registers of the wards and were coded to protect patient identity. The specialty wards were surgery, neurology, and rehabilitation. The criteria for the collection of the records were that they should concern the first 20 patients from each ward who were admitted for five days or more during a specific time period. The collected records were audited three times, each time by a different reviewer. The auditors were nurses knowledgeable and experienced in nursing documentation and in the use of the VIPS model. Before the audit, a calibrating process was undertaken, which means that the use of the instrument was taught and discussed with the reviewers.
The inter-rater reliability was statistically investigated by calculating the inter-rater reliability coefficient37 between raters' total scores of each record. Additionally, score differences between reviewers, on each question in the same patient record, were compared and calculated as percentages of agreement.
The content-validity ratio was calculated as a means of quantifying the degree of consensus in a panel of 10 experts, who made judgments about the instrument's content validity. Each expert was asked to judge whether or not the 10 questions in the instrument, meant to measure the nursing process, were indeed essential in measuring the parts of the nursing process documented in a patient record. The method, developed by Lawshe,38 is described by the formula:
where CVR is the content-validity ratio, ne is the number of panellists indicating “essential” about a specific question and N is the total number of panellists.
The criterion-related validity was estimated by the degree of correlation between the score of the Cat-ch-Ing instrument and the score of the audit instrument developed by Ehnfors9 and used in previous research. The Ehnfors instrument was constructed to measure whether each part of the nursing process (and thereby also the VIPS model) was documented for each nursing problem identified in the patient record. The nursing process was the chosen criterion in both the Ehnfors and the Cat-ch-Ing instrument. The Ehnfors instrument has a score from zero to five, giving one point for each documented part of the nursing process: assessment, goal and diagnosis, planned intervention, implemented intervention, and a discharge note, concerning each specified nursing problem. The Ehnfors instrument scores mainly the quantity; the quality aspect is only present for evaluating the flow of information in the nursing process for each patient problem.
A mean Ehnfors score of all identified nursing problems in a record was calculated and compared with the total score given by the Cat-ch-Ing instrument for the same record. The results were then tested by the use of Pearson's correlation coefficient.
The research protocol was approved by the regional ethical committee of the Karolinska Institutet.
The final version of the Cat-ch-Ing instrument (appendix), which was completed in December 1996, consists of 17 questions: 10 reflecting the presence of each step of the nursing process; four about dating, signatures, and legibility; one about keywords; and one asking about the existence of the individual patient's named nurse.
The total score ranges from zero to 80 points. Sixty eight per cent of the total score may be achieved by questions that are posed to measure the content of the nursing process; 15% relate to questions that judge legibility, signing, and dating; 7% correspond to keywords; 5% to the nursing discharge note; and 5% to the identification of a primary nurse.
Nine of the questions may be rated for quantity and quality. Five questions can be rated only for quantity, for example “are all entries signed?”, and one question about legibility is rated only for quality. The two remaining questions have “yes” or “no” answers (fig 3).
The inter-rater reliability coefficients were calculated to be 0.98, 0.98, and 0.92 for each group of patient records from the three wards.
The content validity ratio between the expert panellists (table 1) ranged between 0.20 and 1.0. Of the 12 items measuring the nursing process in the instrument, all but three were judged to be essential by the expert panellists.
The criterion related validity for the Cat-ch-Ing instrument was illustrated by the significant correlation (r = 0.68, p = <0.0001, 95% CI 0.57 to 0.76) between the scores of the Ehnfors and the Cat-ch-Ing instruments (fig 4).
On examining the score differences between the three reviewers on each question (n=4680 comparisons) we found no differences in scores in 64% of the comparisons. Thirty two per cent of the comparisons differed by one point and 4% differed by two or three points. The largest score differences (two or three points) occurred on the items of qualitative judgment of nursing assessment and nursing interventions. Larger records (for example, 56 pages of text) had a greater discrepancy among raters scoring the same record than less extensive records.
This study has resulted in a new instrument for auditing nursing documentation of the patient record. The instrument has proved to be valid for measuring information pertinent to the nursing process, and to possess a high degree of reliability when used by different auditors.
In the few records in which an increased discrepancy among auditors was noted, the patient records proved to be comprehensive. The reasons for the discrepancy may be that it is harder to keep track of pertinent information in a large mass of nurses' notes, or simply that the auditor loses concentration after reading the same record over a long period.
The Cat-ch-Ing instrument has been thoroughly investigated for validity and reliability. Two types of validity have been confirmed, and using three reviewers in the reliability testing strengthens the results. The Cat-ch-Ing instrument showed a strong validity for measuring whether the nursing process existed in the patient record, which is one of the main strengths of the instrument. It clearly gives a measure not only of the amount of written text but also, most importantly, of the quality of the information that has been documented. Furthermore, the Cat-ch-Ing is only a one page instrument with a two page manual, whereas other instruments usually have more extensive instructions, and this may be a facilitating factor.
Various limitations to the study should be highlighted, however. Firstly, the auditors were selected because of their knowledge and experience in documentation, as well as in nursing. This was thought to be a necessary prerequisite when developing a new instrument. We have not tested the instrument among nurses in general. Secondly, the study dealt with records from the wards of a major university hospital that provides somatic acute care and short term rehabilitation. The testing we did could be considered valid for this type of record only. Thirdly, the weighting of the scores between the various questions may have to be adjusted; as much as 32% of the score can be achieved by dating and signing correctly, by recording the named nurse, by using a typewriter, and by using the keywords of the VIPS model correctly. None of this indicates the nursing process.
The development and testing of the Nordström and Gardulf audit instrument for nursing documentation has not been scientifically described in the literature. The Phaneuf Nursing Audit tool has been described in numerous papers. Neither of these produces a clear result of validation and reliability testing. Also, the Phaneuf instrument claims to measure the quality of care by auditing the patient record and has received criticism for this.29, 32, 36 The Ehnfors instrument,9 used as a comparison in this study, has an obvious, high, face validity and inter-rater reliability when tested by Kohen's κ ( κ = 0.93).
Group level comparisons with inter-rater reliability coefficients in the vicinity of 0.70 show sufficient reliability.39 Thereby the reliability of the Cat-ch-Ing instrument, with a coefficient of 0.98, proved to be very satisfactory. It is a known problem that auditing patient records involves subjective judgments.39 Less inference is required of the auditor when reviewing the documentation of demographic information compared with that required when assessing the adequacy of documentation related to the patient specific needs and specific nursing skills, such as educational strategies and information giving.40
According to Lawshe,38 the minimum value of the content validity ratio to ensure that agreement is unlikely to be due to chance, with 10 panellists, is 0.62 per identified item. This indicates that the Cat-ch-Ing instrument to a large degree measures the documented nursing process in the patient record, as intended. Seven out of 10 items in the instrument received satisfactory values. The three items that received a lower value will be considered for exclusion. The reasons for the lower value for the questions, “is the underlying information for the nursing diagnosis described in the nursing status?” and “is there a new nursing status at the time of discharge?”, may be because the questions were thought to be already covered by other questions in the instrument. Why a lower value was given to the question, “is the expected outcome (goal) documented in the care plan?” is more difficult to explain. One reason may be that the expert panellists all work with the nursing process model in a theoretical setting where the patient outcome is not an explicit part of the model, whereas in the practical setting it is an explicit part of the model.
The fact that yet another nursing audit instrument has been developed and tested implies to nurses that the auditing of nursing performance is an important subject, possibly making more nurses familiar with auditing and quality improvement. One approach to increasing the awareness and knowledge of the auditing of nursing documentation and care planning is to encourage the use of a peer review system. By using an instrument like the Cat-ch-Ing, peer review of patient records may be a means not only of improving patient records but also of instigating a discussion and thereby possibly reaching a consensus on best nursing care in specific situations, which may improve direct care.
The criterion based audit is a concept used in medicine8 which may be applicable to nursing also. In this study, the nursing process was used as the evaluated criterion because this is what Swedish law prescribes, and may be seen as the short term goal—to improve nursing documentation and record keeping. However, the Cat-ch-Ing instrument is constructed so that it could be modified to measure specific criteria of nursing care quality, as documented in the patient record. Modification would then be described in the user manual, for example what interventions are expected in the nursing care plan for a patient with a specific problem to get a full score, or what specific information will be expected under the keyword of nutrition for a patient with newly discovered diabetes in order to get a full score. This may be seen as a long term goal of auditing within nursing care.
It can be concluded that the Cat-ch-Ing instrument proved to be a valid and reliable audit instrument for nursing documentation in patient records when the VIPS model was used as the basis of the documentation. Nursing, as a growing scientific discipline, is constantly adding new knowledge to clinical care and thereby increasing the need to be able to detect whether patient care was documented in accordance with scientific findings. The next step, once record keeping is improved, will be to evaluate the effect that it has on patient care.
We are grateful to the Stockholm County Council, whose generous grant made this study possible. We also thank Anders Sjöberg for statistical advice.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.