Methodological and conceptual issues confronting a cross-country Delphi study of educational program evaluation

https://doi.org/10.1016/j.evalprogplan.2008.02.005Get rights and content

Abstract

Although the Delphi is widely used, research on certain methodological issues is somewhat limited. After a brief introduction to the strengths, limitations, and methodological challenges of the technique, we share our experiences (as well as problems encountered) with an electronic Delphi of educational program evaluation (EPE) in the Asia-Pacific region. The study is described followed by a discussion of the difficulties in participant recruitment and selection, sample size, instrumentation, data collection and analysis and attempts to resolve them. Some of these problems are generic to the Delphi whereas others related to the specifics of the investigation. What we learned should be useful for future Delphis with a similar focus.

Introduction

The Delphi technique (referred to as the Delphi) has been recognized as an effective method for reaching consensus and/or forecasting future events. It is a good vehicle for collecting opinions from geographically dispersed experts who cannot meet face to face (Delbecg, Van de Ven, & Gustafson, 1975; Linstone & Turoff, 1975). Delphi has been widely used in numerous settings (industry, government, and academe) and in education the procedure has been employed for curriculum development, institutional planning, and other similar matters (Clayton, 1997). Delphi as key word generates about 188,000 records on Google scholar search and over 3600 on the EBSCO academic electronic search. The numbers indicate that it is well-accepted across many disciplines.

While there are numerous Delphi studies that report answers to specific questions, there are fewer that deal with methodological issues such as recruiting the panel, survey administration, and other challenges as well as use in evaluation studies. Limited information pertaining to cross-country applications is available. Thus, the main purpose of this paper is to share our experiences with an electronic Delphi in the Asia-Pacific region. The focus is not on research findings, but on issues encountered and their resolution. We also describe some of the difficulties that arose in the researchers’ collaboration in conducting this research.

Section snippets

The Delphi technique

The development of the Delphi is attributed to Dalkey and Helmer (1963) of the Rand Corporation in the 1950s (Eggers & Jones, 1998; Franklin & Hart, 2007; Garavalia & Gredler, 2004; Gordon, 1994; Hasson, Keeney, & McKenna, 2000; Hsu & Sandford, 2007; Linstone & Turoff, 1975; Milkovich, Annoni, & Mahoney, 1972; Mitchell, 1991; Skulmoski, Hartman, & Krahn, 2007; Spinelli, 1983; Williams & Webb, 1994; Witkin & Altschuld, 1995). Predicated on the logic that “two heads are better than one” (Dalkey,

Strengths and limitations of the technique

The many varied applications of the technique in the literature will not be examined here. Strengths and limitations have also been discussed extensively (Eggers & Jones, 1998; Franklin & Hart, 2007; Gordon, 1994; Hartman, 1981; Hsu & Sandford, 2007; Lang, 1994; Linstone & Turoff, 1975; Mitchell, 1991; Powell, 2003; Price, 2005; Williams & Webb, 1994; Yousuf, 2007). They are summarized in Table 1.

Criticisms of the Delphi have been voiced since the mid-1970s. Sackman (1975) viewed it as

Methodological considerations

Some authors provide recommendations for improvement and better use of the Delphi (Eggers & Jones, 1998; Hasson et al., 2000; Keeney, Hasson, & McKenna, 2006; Lang, 1994; Skulmoski et al., 2007). The following observations are based on a number of concerns found in the literature.

The study of educational evaluation in Asia-Pacific countries

The purpose of our Delphi study was to describe the nature of educational program evaluation (EPE), training needs for professionals, and problems/issues that EPE might encounter 5 years from now in Asia-Pacific countries. Expert opinions were sought and a classical Delphi was employed with three iterations delivered by e-mail and web.

There were two principles for recruiting the sample—the experts were individuals with acknowledged expertise in EPE in their countries and with the ability to

Issues involved in the example Delphi study

In our work, we encountered dilemmas related to some of the methodological issues addressed earlier and others pertinent to the specific focus of the study. Detailed discussion on some of these and the steps taken to address them follows.

Lessons learned/conclusion

Methodological and conceptual issues encountered reflect aspects of the Delphi technique and the unique nature of the project. We believe that our experiences will be informative for others tackling analogous types of research. Lessons learned from this project may help future Delphi research with a similar focus.

The problems related to recruitment, panel composition and size, and participant drop out are common in the Delphi technique. Locating participants in each country and uneven

Acknowledgments

This paper emanated from a project on educational program evaluation in the Asia-Pacific region conducted at the Center for Research on Educational Evaluation and Development (CREED) at National Taiwan Normal University. The first author would like to thank the CREED and its director, Dr. Samuel S. Peng, for providing resources for her post-doctoral fellowship program. The authors appreciate the panelists for their contribution to the study.

Hsin-Ling Hung, PhD, is a Research Associate at the Evaluation Services Center, University of Cincinnati. Her research interests include educational program evaluation, needs assessment, inclusive education and research on special needs population.

References (39)

  • L. Garavalia et al.

    Teaching evaluation through modeling: Using the Delphi technique to access problems in academic programs

    American Journal of Evaluation

    (2004)
  • G. Rowe et al.

    The Delphi technique as a forecasting tool: Issues and analysis

    International Journal of Forecasting

    (1999)
  • F. Woudenberg

    An evaluation of Delphi

    Technological Forecasting and Social Change

    (1991)
  • M.J. Clayton

    Delphi: A technique to harness expert opinion for critical decision-making tasks in education

    Educational Psychology

    (1997)
  • N.C. Dalkey

    The Delphi method: An experimental study of group opinion

  • N.C. Dalkey et al.

    An experimental application of the Delphi method to the use of experts

    Management Science

    (1963)
  • A.L. Delbecg et al.

    Group techniques for program planning

    (1975)
  • R.M. Eggers et al.

    Practical considerations for conducting Delphi studies: The oracle enters a new age

    Educational Research Quarterly

    (1998)
  • K.K. Franklin et al.

    Idea generation and exploration: Benefits and limitations of the policy Delphi research method

    Innovative Higher Education

    (2007)
  • Gordon, T. J. (1994). The Delphi Method. AC/UNU Millennium Project, Futures Research Methodology. Retrieved July 6,...
  • Green, J. P. (1982). The content of a college-level outdoor leadership course. Paper presented at the Conference of the...
  • A. Hartman

    Reaching consensus using the Delphi technique

    Educational Leadership

    (1981)
  • F.T. Hartman et al.

    Using technology to improve the Delphi method

    Journal of Computing in Civil Engineering

    (1995)
  • F. Hasson et al.

    Research guidelines for the Delphi survey technique

    Journal of Advanced Nursing

    (2000)
  • Hsu, C. C., & Sandford, B. A. (2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research &...
  • S. Keeney et al.

    Consulting the oracle: Ten lessons from using the Delphi technique in nursing research

    Journal of Advanced Nursing

    (2006)
  • Lang, T. (1994). An overview of four futures methodologies. Retrieved July 6, 2007 from...
  • H.A. Linstone et al.

    The Delphi method: Techniques and applications

    (1975)
  • K.G. Loughlin et al.

    Using Delphi to achieve congruent objectives and activities in a pediatrics department

    Journal of Medical Education

    (1979)

    • Cited by (0)

    Hsin-Ling Hung, PhD, is a Research Associate at the Evaluation Services Center, University of Cincinnati. Her research interests include educational program evaluation, needs assessment, inclusive education and research on special needs population.

    James W. Altschuld, PhD, is a Professor Emeritus at Ohio State University. He has presented and published extensively on evaluation topics, especially on needs assessment and the evaluation of science and technology education.

    Yi-Fang Lee, PhD, is an Assistant Professor at National Chi Nan University in Taiwan. She has published and presented in the area of needs assessment and Science, Technology, Engineering, and Mathematics retention with underrepresented minorities.

    1

    Tel.: +1 513 556 7004; fax: +1 513 556 5112.

    2

    Tel.: +886 49 291 0960x2611; fax: +886 49 291 5306.

    View full text
    Copyright © 2008 Elsevier Ltd. All rights reserved.