Article Text
PDF
Abstract
Aims To investigate internal consistency and factor structure of a questionnaire measuring learning capacity based on Senge's theory of the five disciplines of a learning organisation: Personal Mastery, Mental Models, Shared Vision, Team Learning, and Systems Thinking.
Design Cross-sectional study.
Setting Substance-abuse treatment centres (SATCs) in The Netherlands.
Participants A total of 293 SATC employees from outpatient and inpatient treatment departments, financial and human resources departments.
Main outcome measures Psychometric properties of the Questionnaire for Learning Organizations (QLO), including factor structure, internal consistency, and interscale correlations.
Findings A five-factor model representing the five disciplines of Senge showed good fit. The scales for Personal Mastery, Shared Vision and Team Learning had good internal consistency, but the scales for Systems Thinking and Mental Models had low internal consistency.
Conclusions The proposed five-factor structure was confirmed in the QLO, which makes it a promising instrument to assess learning capacity in teams. The Systems Thinking and the Mental Models scales have to be revised. Future research should be aimed at testing criterion and discriminatory validity.
- Learning organisations
- teams
- questionnaire
- feedback
- health professions education
- healthcare quality improvement
- organisation
- statistics
- teams
Statistics from Altmetric.com
- Learning organisations
- teams
- questionnaire
- feedback
- health professions education
- healthcare quality improvement
- organisation
- statistics
- teams
Introduction
Organisational learning is believed to be essential for quality improvement processes, and it is thought to be crucial for increasing performance and effectiveness of healthcare.1–3 The concept of organisational learning emphasises expanding capacity at a personal and organisational level.4 Treatment teams are key components in the delivery of high-quality patient care, and learning capacity is a characteristic of individuals and teams in maintaining and improving quality of professional care. Teams with learning capacity are able to handle complexity, organisational change and rising expectations in today's healthcare.
Organisational learning, benchmarking and routine outcome monitoring
It was Peter Senge who introduced the term ‘learning organisation’ in his book The Fifth Discipline4 to a large audience. He defined a learning organisation as one that possesses five core-learning disciplines: Personal Mastery, Mental Models, Shared Vision, Team Learning, and Systems Thinking (see box 1). The disciplines for organisational learning were originally conceptualised for, and became very popular in business organisations, but they are also applied in public sector and healthcare organisations.1–3 5 6
Senge's five disciplines
Personal Mastery—the process of expanding personal capacity and continually improving one's level of expertise in order to achieve desired goals
Mental Models—internal pictures, assumptions, and generalisations of the world that influence perceptions, reactions and decisions
Shared Vision—the capacity to develop commitment to and in an organization or a group, by establishing the principles and guiding practices to develop shared pictures of the future desired by members
Team learning—the capacity of groups to acquire collective thinking skills through dialogue, in order to develop intelligence and results that would not be reached individually
Systems thinking—the ability to use knowledge and instruments in order to perceive the greater patterns in systems and organisations and to change them effectively
The theory of Senge contains elements that fit current developments in healthcare, where emphasis shifts from ‘individual craftsmanship’ to ‘professional accountability.’ Performances are shared and discussed in teams in order to acquire a clear vision how healthcare should be delivered to patients (eg, by evidence-based or practice-based guidelines). Professionals in these teams learn both individually and collectively. Applying routine outcome monitoring (ROM) and benchmarking of performance and outcomes are examples of this shift.
Therefore, concepts such as Team Learning and Shared Vision are extremely important concepts in today's healthcare. In addition, Senge stresses that organisations learn only through individuals: if there is no individual learning, there will be no organisational learning.4 It is these features—together with the reputation in popular management theories—that makes Senge's theory an appealing one for the development of a questionnaire for the assessment of learning capacity.
Empirical studies and instruments
From a theoretical perspective, it is important to know whether the concept of learning organisations exists in Senge's proposed dimensionality and how it can be measured. From a practical perspective, higher management or human resource professionals can benefit from a tool that measures the concept properly in order to assess and influence strengths and weaknesses of organisations and teams. However, the body of literature is small. Pang5 developed a questionnaire for learning organisations in elementary schools and showed that, in accordance with Senge's fifth discipline discourse, Systems Thinking has the strongest associations with the other four scales. Jeong1 developed the Learning Organization Scale (LOS). It was used in healthcare settings and revealed positive associations between learning capacity and organisational commitment and job satisfaction of nurses in South Korean hospitals. However, no data were presented on the factor structure, and it is unknown whether factor structure of the questionnaire fits Senge's theory. Both questionnaires are limited to Asia, and to our knowledge no translations exist. This situation led us to develop and test the Questionnaire for Learning Organizations (QLO).
Aim of the study
The primary aim was to assess whether it was possible to develop an instrument, based on the theory on organisational learning of Senge, that assesses the learning capacity in individuals and teams in a sample of employees in substance-abuse treatment centres (SATCs).
Method
Subjects
Subjects were 720 employees of four Dutch SATCs in urban and suburban areas. Employees worked in different departments—for example, clinicians and social workers in outpatient and inpatient treatment departments, and professionals and administrators from financial and human resources departments. All groups of employees were included because we expected the disciplines of the learning organisation to be general.
Of the subjects, 46% returned a completed questionnaire, resulting in 331 respondents in the study. In 38 cases, one or more answers were missing, leaving questionnaires of 293 respondents for analysis. Cases with missing values were deleted—instead of replacing missing values through imputation—in order to exclude alternative explanations for potential deviations from the proposed factor structure. Excluded respondents did not differ from the remaining sample in terms of age, gender, educational level and SATC.
Instrument and procedure
In the first development stage, a team of two psychologists, two methodologists and a management scientist generated 44 items according to the five disciplines of Senge's theory. Items that did not question actual facts, opinions or behaviour were dropped, resulting in 38 items in the concept questionnaire. Subsequently, potential respondents commented on the clarity and formulation of the items. This resulted in reformulation of unclear items and deletion of two items that could not be reformulated. In the next stage, 73 respondents filled out the questionnaire. Two items generated ‘outliers,’ and those items were discussed with three respondents who scored an outlier on one or both of these items. It turned out that these items could be interpreted in more than one way. These were reformulated, resulting in the final QLO with 36 items analysed in this study.
The scales include: Personal Mastery (PM; nine items), Mental Models (MM; four items), Shared Vision (SV; five items), Team Learning (TL; 13 items), and Systems Thinking (ST; five items). Items employ a six-point Likert-type scale ranging from 1 to 6, denoting ‘not applicable at all’ for 1 and ‘very applicable’ for 6. Higher scale scores represent a larger capacity on the represented discipline, and a higher total score represents a larger learning capacity.
Respondents were asked to rate the items according to their personal work situation. The questionnaire contained items about the individual situation of employees and items about the department of the employee. Since some employees carried out a number of tasks in different positions and departments, respondents were asked to indicate their primary position and department on the questionnaire. Respondents were instructed to fill out the department-related items about this department.
Data preparation and analysis
Skewness and kurtosis coefficients, means and standard deviations of items were assessed, together with the Cronbach α of each of the five scales. Values above 0.70 were regarded as acceptable.7 Next, by deleting items with the lowest interitem correlation, scales were limited to a maximum of five items, resulting in the final version of the QLO for further analysis.
To investigate the factor structure as depicted in figure 1, a confirmatory factor analysis (CFA) was conducted using LISREL 8.8 for Windows. The original five-factor structure was investigated together with one alternative model. The response format of the items is regarded as ordered categorical. Since categorical scales with more than five categories can be treated as continuous in CFA, we analysed the asymptotic covariance matrix using the Maximum Likelihood estimation procedure with the Satorra–Bentler Scaled χ2 8 9 for non-normal data. The absolute χ2 value as reported by LISREL is not evaluated for fit because large samples (generally above 200) will almost invariably produce significant values.10 The present study used the 90% CI for the root mean square error of approximation (RMSEA) with the suggested cut-off value of 0.05 indicating a close fit. Next, it is suggested that the p value to test the interval should be >0.50.9 In addition, the Adjusted Goodness-of-Fit Index (AGFI) was evaluated, ranging from zero to 1.00, with values close to 1.00 being indicative of good fit. Finally, Bentler's Comparative Fit Index (CFI) was also calculated where values >0.90 indicate an acceptable fit.
Model 1; five correlated factors; Model 2: two correlated factors. IL, Individual Learning; MM, Mental Models; PM, Personal Mastery; SV, Shared Vision; TL, Team learning.
Results
Descriptive statistics, reliability and data reduction
Table 1 presents descriptive statistics of items and scales of the initial QLO for the total sample (N=293). Most items were right-skewed and had significant peaked kurtosis, indicating deviations from normality. No differences in internal consistency of scales were found between employees from treatment departments (N=213) and support departments (N=80), corroborating the suggestion that the five disciplines are common for all groups of employees (data not shown).
Psychometric properties of the items and the five subscales of the Questionnaire for Learning Organizations for the total sample (n=293)
The high mean and low standard deviations of three out of five scales (PM, MM and ST) and the total score suggest ceiling effects. Cronbach's α for PM and TL is acceptable, and eliminating items with lowest interitem correlation resulted in the final 22-item QLO with higher internal consistencies, leaving two scales with low internal consistency: MM and ST (table 2). However, scale scores still showed deviations from normality as expressed in the skewness and kurtosis coefficients, and the mean scores PM, MM and ST became even higher.
Psychometric properties of the five subscales of the shortened Questionnaire for Learning Organizations (total n=293)
Confirmatory factor analysis (CFA)
The factor loadings and correlations between the factors were positive and significant for the model 1 as depicted in figure 1. The CFI indicates a satisfactory fit, but AGFI and RMSEA and the associated confidence interval suggest a less satisfactory fit (table 3). Modification indices provided by LISREL for associated measurement error between items a10, ‘My present job is extremely important to me,’ and a11, ‘I feel closely involved with the organisation,’ suggest shared variance over and above the measured concepts. Permitting this association results in a significantly better fit (Model 1b; Δχ2(1) 42.85; p<0.05). The squared multiple correlations for item a2 (‘The results of my work are determined in part by the efforts of staff members on my team’) and for item a7 (‘The results of my work are determined in part by the efforts of staff members outside my team’) indicate that almost no variance of these items is explained by the factor.
Goodness-of-fit indices for models 1 and 2
Table 4 presents correlations between the scales, showing weak associations between TL and all other factors, suggesting that a two-factor model with an Individual Learning factor (factor IL, integrating PM, MM, SV and ST) and a Team Learning factor (factor TL, a replication of the original TL factor) might fit the data as well (Model 2 in figure 1). This resulted in a model with inferior fit (Model 2 in table 3) and with associated measurement errors for items originally designated to the PM and the SV scale, justifying the postulated model with five correlated factors. Therefore, Model 1b is the best alternative.
Correlations between factors*
Discussion
Improving items and scales is the first step to be taken in the development of the QLO. The next stages consist of establishing criterion and discriminatory validity and the assessment of the nomological network. Examples are the comparison of learning capacity assessed with the QLO with learning capacity assessed by organisational experts, and the effects of the dimensions of organisational learning on performance indicators as patient safety and clinical outcomes. For the assessment of the nomological network, associations between learning capacity and characteristics such as educational level, years of employment or gender and age should be established. In addition, comparisons with other available instruments—based on organisational learning theories such as those of Watkins & Marsick, Argyris or Pedlar—developed in other fields are necessary. One of the few instruments with known psychometric properties11 is the Dimensions of the Learning Organisations Questionnaire based on the Dimensions of the Learning Organisations.12 13
Another point is the relation between organisational learning to other concepts in organisational theory like team climate; with the Team Climate Inventory being the most used questionnaire.14–18 A positive team climate is assumed to result in creative and innovative behaviour that enhances team effectiveness, mainly in terms of job satisfaction and consumer satisfaction.17 19 Organisational learning has a larger emphasis on personal development and flexibility, and is more of concern for organisations dealing with the current reorganisations in the addiction treatment field.
A limitation for this study is sample size. To apply CFA on ordinal or non-normal data, the use of asymptotic distribution-free (ADF) estimators is recommended. A practical limitation of ADF estimators is the need for very large sample sizes of 1000 subjects or more. Most researchers, therefore, treat the data as continuous by using ML estimators and correct this by using the Satorra–Bentler scaled statistic.8 9 20 Suggestion for future research is to test the QLO in larger samples. This study is also limited in setting. It took place in the Dutch substance abuse treatment sector (SATS), and one must consider whether these results hold for other settings, such as mental healthcare or healthcare in general. The sector is considered as less developed than other health facilities—for example, having fewer educated employees, less transparent treatment processes and being slow to adopt computerised information systems.21 22 However, we expect the factor structure of the QLO to be more or less stable in different healthcare settings, since Senge's theory makes no statements on the applicability of his theory to specific sector. We expect the above differences to be reflected in scores on distinctive scales. In order to assess whether this hypothesis is true, future research in different settings has to be conducted. Finally, Senge stresses that teams form the core of learning, and not individuals. In this context, it is questionable if Team Learning can be assessed in individual subjects. Although we tackled this by making the team the object of evaluation in the scale Team Learning, it is still an individual assessment, and it remains questionable if it is appropriate to calculate and summarise individual scores on a shared object of evaluation.
Conclusion
This is the first study on the QLO and one of the few empirical studies examining the operationalisation of the concept of learning organisations. This study was conducted in substance abuse treatment, to test whether Senge's five disciplines could be measured with a questionnaire. The presence of the five disciplines was confirmed, although the present quality of the QLO requires improvement.
For Systems Thinking, revision of items is necessary into items that are focused on actual actions instead of the respondents' opinion on their skills. For instance, the two items (a9 and a14) may invoke social desirability, as they focus on self-reported skills on solving problems. Since almost no variance of the items on the determination of work results (a2 and a7) is explained by the factor, they have to be rephrased as well—preferrably into items that are more distinguishable from each other in phrasing—in order to configure a unity with the other items in the scale.
Acknowledgments
We would like to express our gratitude to the management and treatment professionals at the various centres for taking part in the study. The Jellinek (now the Jellinek division of Arkin), Brijder Substance Abuse Treatment Center (now the Brijder Substance Abuse Treatment Division at ParnassiaBavo Group) and Novadic-Kentron made it possible for us to conduct the study. We also thank M Spits, at the AIAR for research assistance, and A Buchholz, for her advice.
References
Footnotes
Funding This study was funded by The Netherlands Organization for Health research (ZON-MW grant no 310000050).
Competing interests SCCO and GMS hold research positions at the substance abuse treatment centre participating in the study; SCCO and MES hold management positions at the substance abuse treatment centre under study.
Provenance and peer review Not commissioned; externally peer reviewed.