Article Text
PDF
Abstract
Introduction Well-organised practices deliver higher-quality care. Yet there has been very little effort so far to help primary care organisations achieve higher levels of team performance and to help them identify and prioritise areas where quality improvement efforts should be concentrated. No attempt at all has been made to achieve a method which would be capable of providing comparisons—and the stimulus for further improvement—at an international level.
Methods The development of the International Family Practice Maturity Matrix took place in three phases: (1) selection and refinement of organisational dimensions; (2) development of incremental scales based on a recognised theoretical framework; and (3) testing the feasibility of the approach on an international basis, including generation of an automated web-based benchmarking system.
Results This work has demonstrated the feasibility of developing an organisational assessment tool for primary care organisations that is sufficiently generic to cross international borders and is applicable across a diverse range of health settings, from state-organised systems to insurer-based health economies. It proved possible to introduce this assessment method in 11 countries in Europe and one in Africa, and to generate comparison benchmarks based on the data collected. The evaluation of the assessment process was uniformly positive with the view that the approach efficiently enables the identification of priorities for organisational development and quality improvement at the same time as motivating change by virtue of the group dynamics.
Conclusions We are not aware of any other organisational assessment method for primary care which has been ‘born international,’ and that has involved attention to theory, dimension selection and item refinement. The principal aims were to achieve an organisational assessment which gains added value by using interaction, engagement comparative benchmarks: aims which have been achieved. The next step is to achieve wider implementation and to ensure that those who undertake the assessment method ensure linkages are made to planned investment in organisational development and quality improvement. Knowing the problems is only half the story.
- Family medicine
- general practices
- microsystem
- organisation
- quality of care
Statistics from Altmetric.com
Background
Well-organised practices deliver higher-quality care.1 Evidence-based medicine and the introduction of clinical practice guidelines have been influential, particularly if linked to target setting. However, there has been very little emphasis on process management or cultural change methods to help primary care organisations achieve quality improvement. No attempt has been made to achieve organisational comparisons—as a stimulus for further improvement—at an international level. There is, however, evidence of pressure on primary care organisations to be accountable to external bodies, such as the Quality Outcomes Framework in the UK2 or to meet accreditation standards.3 However, most of these methods are summative exercises. The methods also either tend to be automated (coded computer data) or require collection by trained external agents (eg, practice visits4). There are very few methods which provide formative assessments and which aim to support practice development and the generation of iterative action plans.5 An exception is the Quality Team Development (QTD) method, developed by the Royal College of General Practitioners in the UK.6 However, the QTD approach has a high cost, requires multiple meetings with trained facilitators and is labour-intensive.7 Moreover, despite increasing interest in achieving international comparisons, efforts to facilitate practice development in primary care have, apart from the European Practice Assessment,4 been country-specific.
Primary care organisations have not traditionally invested effort in organisational development: there are many competing priorities. An efficient yet effective approach to practice assessment is needed.8 ,9 In response, we have developed an approach known as the Maturity Matrix10 ,11 which also aims to promote communication and learning.12 ,13 The method involves a facilitator visiting a practice in order to assess the achieved degree of organisational development. The facilitator convenes a multidisciplinary group meeting, typically involving three to 10 individuals. Each individual is asked to complete the Maturity Matrix instrument—without conferring. When individuals have completed their assessment, the facilitator leads a discussion where participants compare their evaluations, aiming to arrive at a consensus in a group discussion of approximately 90 min. A consensus score is agreed and the data entered onto an online database. The scores enable the identification of priorities and support the creation of a practice action plan. The UK Maturity Matrix (2003) was evaluated10 and adapted for international use (MS Buch, A Adrian Edwards, T Eriksson, submitted, 2009) in a series of evaluations in Slovenia, Switzerland, The Netherlands and Germany10 ,11 ,14 ,15 and Denmark (MS Buch, A Adrian Edwards, T Eriksson, Submitted, 2009; T Eriksson, VD Siersma, L Løgstrup, et al, Submitted, 2009).16
However, concerns were noted, when the UK Maturity Matrix was used in other countries about dimensions choice and the lack of an underpinning theoretical model, that the scale development had not ensured that each step required the completion of the former step,14 ,15 ,17 that is that items had an incremental sequence (a design known as a Guttman scaling18). We decided to develop a new version of the Maturity Matrix instrument which would be underpinned by a theoretical model and designed for international use. Our literature review had confirmed the need for this approach.8 ,9 The aim of our work was to design this instrument and to evaluate the feasibility of this approach in different countries by forming a collaboration with the European Association for Quality in Family Medicine (EQuiP).
Methods
Development of the International Family Practice Maturity Matrix (IFPMM)
The development of the IFPMM (http://www.maturitymatrix.co.uk) took place in three phases:
selection of organisational dimensions;
development of incremental scales;
testing the feasibility of the IFPMM.
Selection and refinement of organisational dimensions
A literature review of organisational factors and frameworks related to high-quality care primary care was conducted12 ,13 ,19–22 and presented to a workshop of EQuiP delegates (Turkey, 2006) in order to generate a list of organisational factors. In each country, a principal investigator (PI) was asked to form an expert panel (10 individuals) of primary care practitioners (clinical or managerial) who had experience of evaluation. The members were asked to use an online system to rank the organisational factors in order of importance to the delivery of high-quality care. These ranked factors formed the basis for the IFPMM dimensions. This process ensured fidelity to organisational issues, and not to the performance of individual clinicians, and that dimensions could be applicable in different healthcare settings.
Development of incremental scales based on a recognised theoretical framework
Instrument development followed the following steps: alignment to theory, prototype generation and refinement by online consultation with the country-specific panels. We noted that the Manchester Patient Safety Framework (MAPSAF)23 had used a similar approach to the Maturity Matrix to assess patient safety. MAPSAF had adapted Westrum's work on the typology of organisational culture: a three-level classification.24 At the lowest level, labelled ‘pathological,’ organisations are power-oriented and demonstrate low cooperation and poor communication, where failure leads to blame, and innovation is discouraged. At the second ‘bureaucratic’ level, organisations are rule-based and have modest levels of cooperation and communication, failure is analysed, and innovation proposals require formal approval. At the third ‘generative’ level, organisations are orientated around high performance, cooperation and communication, failure leads to constructive enquiry, and innovation is valued as having potential for improvement.
We used this model as the basis for instrument development by ensuring that the dimensions and items were stratified attributed to the Westrum model. The final English (see figure 1) version was translated using two independent forward-translations, followed by adjustment to country-specific settings and, finally, two independent backward translations, and agreement of a final version.25
International Family Practice Maturity Matrix, copyright-free.
Testing the feasibility of the IFPMM: an international approach
Members of EQuiP were invited to become PIs in an implementation feasibility study and to recruit a minimum of five practices that provided routine primary care services. In a workshop, training materials (handbook, DVD and web-based materials) were provided. Practices were recruited by using an initial familiarisation meeting, followed by a consensus meeting, where the IFPMM assessment was completed and, finally, a feedback meeting.
IFPMM evaluation
Questionnaires were used to evaluate the IFPMM Consensus meeting process and its contribution to practice development, the skills of the facilitator and views about the overall process. In addition, MJB conducted semistructured telephone interviews, covering the following issues: advantages and disadvantages of using the IFPMM process, any difficulties encountered and suggestions for improvement. The interviews were recorded and transcribed, and a thematic content analysis was performed.
The IFPMM scores were entered into an online database. The automated output (see figure 2) provides a visual representation of the practice score and two comparative benchmarks: (1) the average practice score benchmark (a mean of all other practice assessments) and (2) comparison with organisations with scores at the leading edge score benchmark (ie, the aggregated top 25%). Dimension scores were rescaled (0 to 100), with the IFPMM global score being the average of the seven dimension scores, and score variation within and between countries was examined. Dimension cohesiveness was assessed by Spearman correlations, on an intercountry (ecological) basis using country means and on an intracountry basis by calculating residuals.
International Family Practice Maturity Matrix consensus matrix showing practice score, average and leading edge benchmark.
Results
Selection and refinement of organisational dimensions
The literature review provided evidence that the following changes made a difference to patient care: the implementation of guidelines and recall systems, increasing consultation length and the use of targeted incentive systems:19 with this input, the workshop generated a list of 69 factors. We decided that the most appropriate organisational framework was the Dutch adaptation of the European Framework for Quality Management (EFQM):20 it is divided into the following aspects: leadership, policy and strategy, people management, resources, processes, appreciation by customers, by people and by society, and lastly, business results. The brainstorm factors were organised according to this framework and ranked by the country-specific panels (Austria, Belgium, Czech Republic, Denmark, Finland, Ireland, The Netherlands, Norway, Poland, Portugal, Slovenia, Spain, Sweden, Turkey and the UK). Full results are published elsewhere.19
The IFPMM dimensions were based on the highest ranked aspects. However, although highly ranked, the financial management and leadership dimensions were excluded on the grounds that it is not possible to debate financial management in a group where discussion about profit sharing would not be possible and, similarly, that a debate on leadership would be too contentious. Two aspects, ‘Appreciation by staff’ and ‘People management,’ were merged into a dimension called ‘Managing staff.’ table 1 summarises the process of arriving at the final seven dimensions included in the IFPMM.
Results of the ranking process and dimension ranking and refinement
Development of incremental scales
The development of the IFPMM instrument involved five iterations. Feedback from the EQuiP workshop in Barcelona (2006) suggested shorter item phrasing, and 120 (75%) of the 159 panel members participated. There was a high agreement (82%) that the items could be applied in each country. Thirteen translations were produced (Albanian, Croatian, Dutch, French (Walloon), German, Greek, Italian, Norwegian, Portuguese, Slovene, Spanish, Swedish and Turkish).
Testing the feasibility of the IFPMM: an international approach
At the end of a 12-month feasibility study, data were available from 73 practices in 12 countries: Belgium (four practices), Croatia (five), Germany (22), Greece (five), Kosovo (five), Nigeria (six), Norway (four), Portugal (three), Slovenia (five), Spain (five), The Netherlands (five) and the UK (four). Nineteen practices were single-handed; a further 17 had two doctors, and other practices ranged from three to nine practitioners. Only seven practices had no nurse as part of the staff. The PIs in Nigeria (YY), Kosovo (IM) and Germany (CK) joined the study independently of EQuiP.
IFPMM evaluation
The evaluation of the IFPMM consensus meeting process was positive: there was a high agreement from 70 practices that there was sufficient time to discuss organisational development (99%), that the process facilitated talk about organisational development (90%), that respondents would like to use the IFPMM again (97%) and that the comparative benchmark was useful (94%). Similarly, positive evaluations of the facilitator were received from 70 practices: that the process was effective (99%), that clear aims were stated (90%) and that there was a willingness to recommend the facilitator to others (99%). Positive evaluations were received for the identification of improvement priorities (97%) and that the process added value (95%). In summary, the evaluation confirmed that the IFPMM was viewed as a valuable method for assessing organisational development in primary care across a range of settings. The two benchmarks were considered to be a novel and useful way to motivate quality improvement.
A total of 14 individuals from Belgium, Croatia, Germany, Greece, The Netherlands, Nigeria, Portugal, Norway, UK, Slovenia and Spain were interviewed using a semistructured schedule. Thematic analysis showed that the method, the clarity of the instrument design and the team-building group process aspects were consistently appreciated. In addition, the benchmarking feature helped prioritise areas for change and generate enthusiasm for action. There were concerns that the assessment process requires adaptation for use in very large or solo practices. The interviewees also felt that the IFPMM process needs to be embedded in an organisational development system where support for change management should be linked to wider system of quality improvement at a group or regional level.
FPMM scores and comparative benchmarks
IFPMM scores
Note that practice samples are small and cannot be representative because participation was voluntary and by invitation. Nevertheless, the global IFPMM scores had a wide range across the 73 practices, from 14 to 94. There was significant variation (p<0.001) in the mean global scores from the 12 countries, from 21 (Nigeria) to 79 (UK); see table 2 and figure 3. Rank correlations between scores for the seven dimensions ranged from 0.4 to 0.7 (median 0.5). Broadly similar correlations were obtained from residuals (0.3 to 0.7, median 0.4) and on an ecological basis (0.2 to 0.8, median 0.6), indicating that the moderately large positive correlations between the dimensions are driven by both country- and practice-level factors.
Maturity Matrix Global Scores (all practices, country level)
Consensus matrix: global and dimension scores.
Average and leading edge benchmark
Figure 2 illustrates the IFPMM output. The yellow line provides the consensus IFPMM score for the practice. The blue line shows the average benchmark. The red line illustrates the leading edge benchmark.
Discussion
This study demonstrates the feasibility of developing an organisational assessment tool for primary care organisations. We believe the assessment is valid at the organisational level and leads to improvements in annual timescales: it also has the potential to create valuable comparison data where it is appropriate to examine similar structures and processes, and the process can be applied across a range of health settings. It was possible to introduce this assessment method in 11 countries in Europe and one in Africa, and to generate comparison benchmarks based on the data collected.
We caution against intercountry comparisons, however: the samples are too small, voluntary and therefore subject to selection bias. There are also local, regional and contextual factors that need to be taken into account in order to interpret the results. In ideal circumstances, further training of the facilitators would ensure greater assessment consistency. Nevertheless, evaluations of the assessment process were uniformly positive: participants felt that the approach efficiently identifies organisational development priorities and motivates efforts to achieve change, especially at the practice level.
The Maturity Matrix concept was initiated in 1996, was the subject of a doctorate thesis,17 and was adapted for use in Denmark (MS Buch, AA Edwards, T Eriksson, Submitted, 2009; T Eriksson, VD Siersma, L Løgstrup, et al, submitted, 2009).16 Criticism of the initial version led to this study in which a new instrument was developed for international use. We are not aware of any comparable work in primary care. In terms of research methods, we would have preferred to assess more practices and for facilitators to have had more training (MS Buch, AA Edwards, T Eriksson, submitted, 2009).11 We attempted to include practices from the US and Canada, in order to enhance the validity, and hope that this will take place in future studies.
It is important to note that the aim was not to create a summative measure or a ‘league table’ of practice quality but to generate an assessment, albeit ‘a soft fuzzy’ assessment, that engages participating practices and enables comparisons against peers using the concept of benchmarks without confronting sensitive issues such as financial issues in group settings.
Findings in context
We could not identify similar studies. Most assessment methods in this context have used a checklist approach to provide external assessments (Y Engels, S Campbell, M Dautzenberg, et al, submitted, 2004)8 often aligned to accreditation efforts.3 A notable exception is the QTD method developed by the RCGP,7 which is a team-based formative assessment. Macfarlane's in-depth interview study noted that, although the process was valued, it required significant additional resources, and the assessment is not widely implemented.7 Work in Denmark, on an adapted version of the UK Maturity Matrix, reports that participants were positive about the facilitated group-based assessment approach (MS Buch, AA Edwards, T Eriksson, submitted, 2009; T Eriksson, VD Siersma, L Løgstrup, et al, submitted, 2009).16
Conclusions
We are not aware of any other organisational assessment methods for primary care that have an international aim or that have used an organisational framework and a theoretical model to guide their development. We recognise that formal validation has not been achieved, but the purpose is not to achieve high reliability: the principal aim is to achieve an organisational assessment that is capable of identifying priorities for change and at the same time motivating the participants to take action. We think it may be possible to generate a generic version that could be used in organisational unit that Batalden refers to as microsystems.26 However, our next aim is to achieve wider implementation and to ensure linkages are made between the IFPMM results and planned investment in quality improvement. Knowing the problem is only half the story: improving the organisation so that the problems are tackled and managed is the more difficult task, especially when this involves changing cultures and communication processes as well as introducing management tools such as a documented interdisciplinary meetings and developing standardised operating procedures. An international database of practice-based assessments is being developed, and we would welcome initiatives to join this effort.
Acknowledgments
We thank the European Association for Quality in General Practice/Family Medicine (EQuiP) for their support and cooperation and EQuiP delegates, R Glehr, L Seuntjes, B Seifert, T Eriksson, E Ketola, M O'Riordan, K in't Veld, J Thesen, ZJ Król, L Pisco, J Kersnik, JM Turón, P Stenström and N Dağdeviren for convening the country-specific panels. Most of the PIs in the feasibility study were recruited from EQuiP ranks. In addition to already named authors, we are grateful to the following PIs: T Schofield (UK) and S Uka (Kosovo). Full translations for the IFPMM instrument were achieved by all named PIs as well as by A Campanini (Italy), P Stenström (Sweden) and N Daĝdeviren (Turkey). In addition to PIs conducting consensus meetings, IFPMM facilitation was provided by K Knops (Belgium), J Warns and K Pralle (Germany), K Prestegaard and J Falhof (Norway), H Covita (Portugal), B Oliván and Y Lopez (Spain), and L Roebroeck-van de Zande (The Netherlands). Finally, we wish to acknowledge the support of participating practices, Oscom Solutions (database design) and the support provided by E Hussein, S Evans and A Watkins. EQuiP (http://www.equip.ch/) is a constituent part of the World Organization of Family Doctors (WONCA), and the members have expertise in quality improvement in primary care.
References
Footnotes
Funding This study was achieved without external funding by collaboration between Radboud University Nijmegen Medical Centre and Cardiff University. Other Funders: internal.
Competing interests The IFPMM instrument is available copyright-free. A licence is required to gain access to other IFPMM resources and the benchmark database, under the administration of Cardiff University on a not-for-profit basis. No author benefits financially from the licensing arrangements.
Provenance and peer review Not commissioned; externally peer reviewed.