Article Text
PDF
Abstract
Objective The Maturity Matrix (MM) comprises a formative evaluation instrument for primary care practices to self-assess their degree of organisational development in a group setting, guided by an external facilitator. The practice teams discuss organisational development, score their own performance and set improvement goals for the following year. The objective of this project was to introduce a translated and culturally adapted version of the MM in Denmark, to test its feasibility, to promote and document organisational change in general practices and to analyse associations between the recorded change(s) and structural factors in practices and the factors associated with the MM process.
Setting MM was used by general practices in three counties in Denmark, in two assessment sessions 1 year apart. First rounds of MM visits were carried out in 2006–2007 in 60 practice teams (320 participants (163 GPs, 157 staff)) and the second round in 2007–2008. A total of 48 practice teams (228 participants (117 GPs; 111 staff) participated in both sessions.
Method The MM sessions were the primary intervention. Moreover, in about half of the practices, the facilitator reminded practice teams of their goals by sending them the written report of the initial session and contacted the practices regularly by telephone reminding them of the goals they had set. Those practice teams had password-protected access to their own and benchmark data.
Results Where the minimum possible is 0 and maximum possible is 8, the mean overall MM score increased from 4.4 to 5.3 (difference=0.9, 95%, CI 0.76 to 1.06) from first to second sessions, indicating that development had taken place as measured by this group-based self-evaluation method. There was some evidence that lower-scoring dimensions were prioritised and more limited evidence that the prioritisation and interventions between meetings were helpful to achieve changes.
Conclusions This study provides evidence that MM worked well in general practices in Denmark. Practice teams appeared to be learning about the process, directing their efforts more efficiently after a year's experience of the project. This experience also informs the further improvement of the facilitation and follow-up components of the intervention.
- Communication
- continuous quality improvement
- family medicine
- management
- organisation
Statistics from Altmetric.com
Introduction
Formative organisational assessment has been proposed as a means for quality improvement (QI) in primary care, but there is little evidence about whether such approaches successfully promote or achieve improvement. Family practices are multidisciplinary organisations where teamwork is essential for optimal care.1 There is some evidence that including staff in decision-making is positively correlated with a higher quality of care,2 because influence and shared knowledge bring encouragement and work satisfaction.3 ,4
The Maturity Matrix (MM) is a formative evaluation instrument designed for primary care practices to self-assess their degree of organisational development in a group setting, guided by an external facilitator.5 The aspects covered by MM include a wide range of organisational issues (table 1). At the facilitated MM session, the whole practice team should be present. Without prior exposure to MM, each participant is given a blank profile and instructed to assess the level of organisational development for each dimension in their practice. After this, the facilitator conducts a discussion, taking every dimension in turn, achieving an agreement score based on the lowest level at which consensus occurs. Finally, the practice team may choose goals for future organisational development. In the Danish setting, practice teams were encouraged to appoint an anchor person responsible for each goal set.
Danish version of the Maturity Matrix used
Apart from the Danish version reported here, adjusted versions have been made and tested in other parts of Europe, including UK, Germany, Holland, Switzerland and Slovenia, and recently the MM has been adapted for use in the international setting.5 ,6
The objective of this project was first to introduce a translated and culturally adapted version of MM in Denmark, to test its feasibility to promote and document organisational change in general practices. A secondary objective was to analyse associations between the recorded change(s) and structural factors within practices and factors associated with the MM process itself.
Method
Setting
MM was introduced in Denmark in 2004, and adapted in the years 2004–2006 through a pilot in four practices. Adjustments were made and a final version tested on a larger scale in 2007–2008 in three counties: Aarhus and Copenhagen, where facilitating units were already established, and Frederiksborg, where facilitators were recruited for this specific project.
First rounds of MM sessions were carried out in the period from September 2006 to December 2007 with 60 practice teams (163 GPs, 157 staff) and the second round from November 2007 to November 2008 with 48 of these practice teams (228 participants (117 GPs; 111 staff)).
Participants
An invitation to the MM project was sent to all general practices in the three participating counties, and the first to come forward were included. Participation was free of charge, and practices were reimbursed for the time spent on the sessions by county funds for QI.
MM instrument and MM sessions
The MM instrument comprised 11 out of 12 described dimensions of general practice organisation (table 1). Each dimension has eight consecutive levels, from lower to higher attainment of organisational maturity. (The practices from Frederiksborg County were randomised to one of two different versions of the MM, because of a whish to test the influence of exposure to the dimension significant events on later reporting of events. That led to eight practices using a MM version where significant events were exchanged with learning from patients.)
Facilitators were GPs with prior experience from outreach visits and QI in general practice. All facilitators were trained for the specific purpose of MM. In all cases but one, the facilitators were the same in the first and second visit.
Interventions between meetings
In two of the counties involved (Aarhus and Frederiksborg) interventions in the 1-year period between first and second MM sessions was also performed. Interventions included password-protected access to the practice's own data, including benchmarking to the whole project distribution of scores, on the project website. Facilitators reminded the practice team's contact person of the results and goals recorded at the first MM session by telephone and email. In the third county (Copenhagen), there were no interventions between meetings. The decisions to perform interventions between meetings or not were in the hand of the local county authorities.
Non-participants in second round
Among the 12 practices that did not participate in the second MM session, two had their first MM sessions so late that it was not possible to schedule a second session within the project period. The following reasons (sometimes multiple) for not participating in second sessions were: lack of time and resources (six practices), lack of expectations of benefit of a second session (four practices) and change in GPs and/or staff since last session (three practices).
Measures
Basic features of practices, such as number of GPs, staff composition and list size, etc were recorded. At the first and second MM sessions, facilitators recorded the consensus score of each MM dimension (11 per practice each time). They also noted whether the dimension was prioritised by the practice team, if an anchor person was appointed and what the developmental goals were for each prioritised dimension.
Analytical methods
Data were coded twice in Access and linked together so corrections could be made by consulting original data. Differences in scores between the MM sessions and average number of prioritised dimensions (with and without anchor person) were tested by paired t tests. The change in prioritisation for each dimension was tested by Pearson χ2 tests.
The influences of the various practice and process characteristics on the MM development are inter-related. Therefore, consideration has to be given to the associations that are to be tested. To unravel the various hypotheses of interest (table 2), we postulated an influence diagram7 (figure 1), a graph structure where the various factors are depicted by nodes and causal influences by arrows. To test the influence postulated by a particular arrow, for example, between ‘interventions between meetings’ and ‘change,’ we have to adjust for influences from factors that may confound and mediate this influence; these are read from the graph as the nodes that lie on a path from ‘interventions between meetings’ to ‘change’ (mediating factors, eg, prioritisation) and the nodes that have arrows or edges into both follow-up and change (confounding factors, eg, practice size).7 Using the influence diagram (figure 2), we test the hypotheses described in table 2 using partial γ coefficients.8 Note that by investigating the influence of a process factor on the score at the second meeting adjusted for the score at the first meeting, we effectively test for the effect of this factor on the change between these scores.
Associations tested through graphical modelling
Influence diagram. The associations tested through graphical modelling are pictured here. The influences of the various practice and process characteristics on the MM development are inter-related. To unravel the various hypotheses of interest (table 2), we postulated this influence diagram,7 a graph structure where the various factors are depicted by nodes and causal influences by arrows. To test the influence postulated by a particular arrow, we have adjusted for influences from factors that may confound and mediate this influence. Using the influence diagram, we test the hypotheses described in table 2 using partial γ coefficients.8 Note that by investigating the influence of a process factor on the score at the second meeting adjusted for the score at the first meeting, we effectively test for the effect of this factor on the change between these scores. The questions and interpretation of the results of the graphical modelling are displayed in table 5.
Results
The data set comprised the 48 practices that participated in both MM sessions. At both the first and second session, 11 dimensions were processed, making a total of 528 processed dimensions per session (table 3).
Oversight of the data obtained at first and second Maturity Matrix (MM) sessions
At the second MM sessions, practices scored themselves higher than on the first MM sessions. The mean MM score across dimensions increased from 4.4 to 5.3 (difference=0.9; 95% CI 0.76 to 1.06; variation from 0.2 learning from patients to 1.8 emergencies) from first to second sessions (table 3). The largest changes were in emergencies, significant events, human resource management, practice meetings and continuing medical education.
At the first MM sessions, practices prioritised 260 out of 548 (49%) dimensions for their next year's development, and at the second MM sessions 212 (40%) of dimensions were prioritised (table 3).
The number of prioritised dimensions where practice teams appointed an anchor person increased from 128 out 260 (49%) at the first to 142 out of 212 (67%) at the second MM sessions (table 3).
The results of the graphical modelling are displayed in table 4. Interpretations are based on a combination of whether the results show patterns of positive or negative scores, and the statistical significance (table 5).
Questions and interpretation of results of the graphical modelling
Discussion
Principal findings
At the second MM sessions, practices scored themselves higher than on first MM sessions, showing a change in global score and across all dimensions. The number of dimensions that were prioritised by practices for their next year's development plan dropped considerably from first to second sessions, whereas the number of prioritised dimensions where practice teams appointed an anchor person increased. The results of the graphical modelling provided some evidence that lower scoring dimensions were prioritised and more limited evidence that the prioritisation and interventions between meetings was helpful to achieve changes, but that such changes were independent of practice size.
Strengths and weaknesses
These are the first data that indicate a correlation between a formative QI intervention in general practice and reported development of the organisation. Perhaps the reasoning behind this correlation is circular, as the tool was both the basis of the intervention and the tool for evaluation. However, the findings are consistent with the results of a survey, distributed to all participating staff members and GPs in the project just before the second visit round, and carried out independently from the project itself:9 15% of practices stated that they planned changes, 22% of participants reported taking on responsibilities to be an anchor person, and free text comments indicated that MM may have contributed to new working routines in 55% of participating practices. A qualitative interview study, carried out among participants in this study, concluded that successful change was associated with: a clearly identified anchor person within the practice, a shared and regular meeting structure and an external facilitator who provides support and counselling during the implementation process. Failure to implement change was associated with: a high patient-related workload, staff or GP turnover (that seemed to affect small practices more), anchor persons who were either not clearly identified or ineffective, unsustained support from an external facilitator and no formal commitment to working with agreed changes.10
The study group comprised practices from a range of urban and rural settings, but self selection of practices sharing a common interest in organisational development cannot be ruled out. The study provides longitudinal data on progress in the project, with the project functioning both as evaluation and as an agent of the change. However, the MM scores are based on self-report and may reflect more favourable assessments than would be made by external assessors.11 The improved MM scores at second visit may represent a positive (social) response bias for the facilitators. Moreover, the changes reported were perceptions of influences on the organisation, not actual or observed changes, so until further data are available, caution is required in interpreting the findings. In the absence of a control group, the higher score at second sessions may be due to a general trend in practices. It is difficult to construct a valid control group in this type of intervention, partly due to selection bias of participation by practices already interested in this topic area.
The change vectors tested through the graphical modelling were chosen so that there would be a good chance that the results would provide an insight into the details of the interrelated influences of the factors. Both factors that were structural (practice size), practice process related (prioritising) and external process related (anticipation of interventions between meetings) were tested. The negative results—that prioritising was not associated with change, and that the size of practices covaries with neither prioritising nor change—were surprising and not fully in line with the results of the qualitative data from the interview study. These results may be false-negative.
Interpretation of results in context of current literature
The relationship between effective organisation and good quality in patient care is widely accepted, resulting in a growing interest for organisational QI tools.11–13 There is some evidence that internally led QI initiatives provide insight into the process of QI and work well as a stepping stone for higher levels of QI activity such as EPA,11 Quality Practice Award,14 ,15 Quality Framework for Australian General Practice and16 Joint Commission on Accreditation.17
Analysis of the original UK-based MM instrument revealed a high face validity.5 A criterion validity study combining and comparing MM (formative self-assessment) with EPA showed that practices scored similarly on both instruments but with a tendency to self rate more favourably using MM.18 (EPA is an externally led approach based on both practice visits to check premises and equipment, and patient surveys leading to accreditation when set standards are met, making benchmarking on a national basis and between European countries possible.) It has also been identified that practice teams develop differently across different dimensions in different counties or health systems. The finding in this study that practice teams tended to prioritise dimensions where they scored themselves lower at the first meeting suggests that they have also engaged with the method and ethos of self-assessment.
Geboers et al evaluated the feasibility of an internally led formative intervention based on the ‘quality cycle’ in terms of acceptance and its continued application at the end of the study period.19 Similar to MM, this approach involves all staff in a facilitated setting with designation of a coordinator analogous to the ‘anchor person’ in MM. Sixty-five per cent of projects chosen were completed, and half of the practices continued applying the model after the end of the study.
The visitation instrument for practice management (VIP) is a widely practised accreditation method in The Netherlands.18 ,20 Analogous to MM, VIP evaluates practice organisation on different parameters with feedback from an external observer. Although it is based on external assessment, there is also a discussion of the evaluations, and goals for QI are set. A randomised controlled trial enrolling 49 Dutch practices with VIP implementation showed significant effects of this QI method on initiation and completing own QI projects. In terms of the VIP dimensions, the differences between intervention and control practices were non-significant.20
The findings in this study are consistent with those from the external assessment with VIP in Netherlands21 and support the value of continuous QI initiatives, showing some indication of development occurring after the intervention. The MM intervention of two practice meetings, a year apart, ideally with interventions between meetings to remind the practice team of their goals, is not a large intervention in the context of organisational change. Geboers et al showed that more intensive support was necessary to implement and maintain continuous QI in small organisations,19 and the limited effects of both the prioritisation process and follow-up by facilitator shown in this study concord with this. An initial high level of commitment is no guarantee for success in working with the method, after the facilitator has left the practice.9
Implications for practice
The lower percentage of dimensions prioritised at second than at first MM sessions and the higher percentage of prioritised dimensions where an anchor person was appointed at second meetings indicates that facilitators and practice teams have learnt that they may have more likelihood of success with this strategy. This experience should be integrated into the facilitation process for future participants, arguably with more specific direction at this stage of the MM process to focus on fewer objectives, and with clearly identified project leads.
Further research
The feasibility of MM as an intervention to promote QI and achieve development requires evaluation in other healthcare systems. Attention is required into whether the facilitation and follow-up can be enhanced.
Conclusions
This study provides evidence that General Practices in Denmark engaged with the method of formative self-assessment and QI. They appeared to be learning about the process, directing their efforts more efficiently after a year's experience of the project. This experience also informs the further improvement of the facilitation and follow-up components of the intervention. Even so, the evidence to date is that the MM method can be used to achieve reported organisational development in individual general practices.
References
Footnotes
Funding The Danish Maturity Matrix project was financed by county/regional public means for quality improvement in primary care.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.