Skip to main content
SpringerLink
Log in

Metacognitive scaffolds improve self-judgments of accuracy in a medical intelligent tutoring system

  • Published:
Instructional Science Aims and scope Submit manuscript

Abstract

In this study, we examined the effect of two metacognitive scaffolds on the accuracy of confidence judgments made while diagnosing dermatopathology slides in SlideTutor. Thirty-one (N = 31) first- to fourth-year pathology and dermatology residents were randomly assigned to one of the two scaffolding conditions. The cases used in this study were selected from the domain of nodular and diffuse dermatitides. Both groups worked with a version of SlideTutor that provided immediate feedback on their actions for 2 h before proceeding to solve cases in either the Considering Alternatives or Playback condition. No immediate feedback was provided on actions performed by participants in the scaffolding mode. Measurements included learning gains (pre-test and post-test), as well as metacognitive performance, including Goodman–Kruskal Gamma correlation, bias, and discrimination. Results showed that participants in both conditions improved significantly in terms of their diagnostic scores from pre-test to post-test. More importantly, participants in the Considering Alternatives condition outperformed those in the Playback condition in the accuracy of their confidence judgments and the discrimination of the correctness of their assertions while solving cases. The results suggested that presenting participants with their diagnostic decision paths and highlighting correct and incorrect paths helps them to become more metacognitively accurate in their confidence judgments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Azevedo, R., & Hadwin, A. F. (2005). Scaffolding self-regulated learning and metacognition: Implications for the design of computer-based scaffolds. Instructional Science, 33, 367–379.

    Article  Google Scholar 

  • Azevedo, R., & Lajoie, S. (1998). The cognitive basis for the design of a mammography interpretation tutor. International Journal of Artificial Intelligence in Education, 9, 32–44.

    Google Scholar 

  • Azevedo, R., Moos, D., Greene, J., Winters, F., & Cromley, J. (2008). Why is externally-facilitated regulated learning more effective than self-regulated learning with hypermedia? Educational Technology Research and Development, 56(1), 45–72.

    Article  Google Scholar 

  • Azevedo, R., & Witherspoon, A. M. (2009). Self-regulated use of hypermedia. In A. Graesser, J. Dunlosky, & D. Hacker (Eds.), Handbook of metacognition in education (pp. 319–339). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Berner, E. S., & Graber, M. L. (2008). Overconfidence as a cause of diagnostic error in medicine. The American Journal of Medicine, 121(5A), S2–S23.

    Article  Google Scholar 

  • Bull, S. (2004). Supporting learning with open learner models. In Proceedings of 4th Hellenic conference with international participation: Information and communication technologies in education. Keynote. Athens, Greece.

  • Bull, S., & McKay, M. (2004). An open learner model for children and teachers: Inspecting knowledge level of individuals and peers. In R. Luckin, K. R. Koedinger, & J. E. Greer (Eds.), Intelligent tutoring systems (pp. 232–251). Berlin: Springer.

  • Chi, M. T. H., de Leeuw, N., Chiu, M. H., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18, 439–477.

    Google Scholar 

  • Chi, M. T. H., Siler, S., & Jeong, H. (2004). Can tutors monitor students’ understanding accurately? Cognition and Instruction, 22, 363–387.

    Article  Google Scholar 

  • Chi, M. T. H., Siler, S., Jeong, H., Yamauchi, T., & Hausmann, R. (2001). Learning from human tutoring. Cognitive Science, 25, 471–534.

    Article  Google Scholar 

  • Clancey, W. (1987). Knowledge-based tutoring: The GUIDON program. Cambridge: MIT Press.

    Google Scholar 

  • Corbett, A. T., Koedinger, K. R., & Hadley, W. H. (2002). Cognitive tutors: From the research classroom to all classrooms. In P. S. Goodman (Ed.), Technology enhanced learning: Opportunities for change (pp. 199–224). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Croskerry, P., & Norman, G. (2008). Overconfidence in clinical decision making. The American Journal of Medicine, 121(5A), S24–S29.

    Article  Google Scholar 

  • Crowley, R., & Gryzbicki, D. (2006). Intelligent medical training systems (guest editorial). Artificial Intelligence in Medicine, 38(1), 1–4.

    Article  Google Scholar 

  • Crowley, R. S., Legowski, E., Medvedeva, O., Tseytlin, E., Roh, E., & Jukic, D. (2007). Evaluation of an intelligent tutoring system in pathology: Effects of external representation on performance gains, metacognition, and acceptance. Journal of the American Medical Informatics Association, 14(2), 182–190.

    Article  Google Scholar 

  • Crowley, R. S., & Medvedeva, O. (2006). An intelligent tutoring system for visual classification problem solving. Artificial Intelligence in Medicine, 36(1), 85–117.

    Article  Google Scholar 

  • Crowley, R. S., Medvedeva, O., & Jukic, D. (2003a). SlideTutor: A model-tracing intelligent tutoring system for teaching microscopic diagnosis. In J. Kay, F. Verdejo, & U. Hoppe (Eds.), Proceedings of the 11th International Conference on Artificial Intelligence in Education, Sydney.

  • Crowley, R. S., Naus, G. J., Stewart, J., & Friedman, C. P. (2003b). Development of visual diagnostic expertise in pathology—An information processing study. Journal of American Medical Information Association, 10(1), 39–51.

    Article  Google Scholar 

  • Crowley, R. S., Tseytlin, E., & Jukic, D. (2005). ReportTutor—An intelligent tutoring system that uses a natural language interface. In Proceedings of American Medical Informatics Association Symposium, Austin, TX (pp. 171–175).

  • El Saadawi, G., Azevedo, R., Castine, M., Payne, V., Medvedeva, O., Tseytlin, E., et al. (2010). Factors affecting feeling-of-knowing in a medical intelligent tutoring system: The role of immediate feedback as a metacognitive scaffold. Advances in Health Sciences Education, 15(1), 9–30.

    Article  Google Scholar 

  • El Saadawi, G. M., Tseytlin, E., Legowski, E., Jukic, D., Castine, M., Fine, J., et al. (2008). A natural language intelligent tutoring system for training pathologists: Implementation and evaluation. Advances in Health Sciences Education: Theory and Practice, 13, 709–722.

    Article  Google Scholar 

  • Feyzi-Behnagh, R., Khezri, Z., & Azevedo, R. (2011). An investigation of accuracy of metacognitive judgments during learning with an intelligent multi-agent hypermedia environment. In L. Carlson, C. Hoelscher, & T. F. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 96–101). Austin, TX: Cognitive Science Society.

    Google Scholar 

  • Freidman, C. P., Gatti, G. G., Franz, T. M., Murphy, G. C., Wolf, F. M., Heckerling, P. S., et al. (2005). Do physicians know when their diagnoses are correct? Implications for decision support and error reduction. Journal of General Internal Medicine, 20, 334–339.

    Article  Google Scholar 

  • Goodman, L. A., & Kruskal, W. H. (1954). Measures of association for cross classifications. Journal of the American Statistical Association, 49(268), 732–764.

    Google Scholar 

  • Graber, M. L., Franklin, N., & Gordon, R. (2005). Diagnostic error in internal medicine. Archives of Internal Medicine, 165(13), 1493–1499.

    Article  Google Scholar 

  • Graesser, A. C., Bowers, C. A., Hacker, D. J., & Person, N. K. (1997). An anatomy of naturalistic tutoring. In K. Hogan & M. Presley (Eds.), Effective scaffolding of instruction (pp. 145–184). Cambridge, MA: Brookline Books.

    Google Scholar 

  • Graesser, A. C., Wiemer-Hastings, K., Wiemer-Hastings, P., Kruez, R., & The Tutoring Research Group. (2000). AutoTutor: A simulation of a human tutor. Journal of Cognitive Systems Research, 1, 35–51.

    Article  Google Scholar 

  • Johnson, A. M., Azevedo, R., & D’Mello, S. K. (2011). The temporal and dynamic nature of self-regulatory processes during independent and externally assisted hypermedia learning. Cognition and Instruction, 29(4), 471–504.

    Article  Google Scholar 

  • Juslin, P., Olsson, N., & Winman, A. (1996). Calibration and diagnosticity of confidence in eyewitness identification: Comments on what can be inferred from the low confidence–accuracy correlation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22(5), 1304–1316.

    Google Scholar 

  • Katz, J. (1984). Why doctors don’t disclose uncertainty. The Hastings Center Report, 14(1), 35–44.

    Article  Google Scholar 

  • Kelemen, W. L., Frost, P. J., & Weaver, C. A. (2000). Individual differences in metacognition: Evidence against a general metacognitive ability. Memory & Cognition, 28(1), 92–107.

    Article  Google Scholar 

  • Koedinger, K., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review, 19(3), 239–264.

    Article  Google Scholar 

  • Koedinger, K., & Corbett, A. (2006). Cognitive tutors: Technology bringing learning science to the classroom. In R. K. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences (pp. 61–78). New York: Cambridge University Press.

    Google Scholar 

  • Koriat, A., Sheffer, L., & Ma’ayan, H. (2002). Comparing objective and subjective learning curves: Judgments of learning exhibit increased underconfidence with practice. Journal of Experimental Psychology: General, 131, 147–162.

    Google Scholar 

  • Lajoie, S. P. (2009). Developing professional expertise with a cognitive apprenticeship model: Examples from avionics and medicine. In K. A. Ericsson (Ed.), Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments (pp. 61–83). New York: Cambridge University Press.

    Chapter  Google Scholar 

  • Lajoie, S. P., & Azevedo, R. (2006). Teaching and learning in technology-rich environments. In P. Alexander & P. Winne (Eds.), Handbook of educational psychology (2nd ed., pp. 803–821). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Lepper, M. R., Drake, M. F., & O’Donnell-Johnson, T. (1997). Scaffolding techniques of expert human tutors. In K. Hogan & M. Pressley (Eds.), Scaffolding student learning: Instructional approaches and issues (pp. 108–144). Cambridge, MA: Brookline.

    Google Scholar 

  • Maki, R. H., Shields, M., Wheeler, A. E., & Zacchilli, T. L. (2005). Individual differences in absolute and relative metacomprehension accuracy. Journal of Educational Psychology, 97(4), 723.

    Article  Google Scholar 

  • Mann, D. (1993). The relationship between diagnostic accuracy and confidence in medical students. Atlanta, GA: American Educational Research Association.

    Google Scholar 

  • Maries, A., & Kumar, A. (2008). The effect of student model on learning. In I. Aedo, N. Chen Kinshuk, D. Sampson, & L. Zaitseva (Eds.), ICALT’09, 8th IEEE International Conference on Advanced Learning Technologies (pp. 877–881). Piscataway, NJ: IEEE Computer Society Press.

    Google Scholar 

  • McGinnis, K. S., Lessin, S. R., Elder, D. E., DuPont Guerry, I. V., Schuchter, L., Ming, M., et al. (2002). Pathology review of cases presenting to a multidisciplinary pigmented lesion clinic. Archives of Dermatology, 138(5), 617–621.

    Article  Google Scholar 

  • Medvedeva, O., Chavan, G., & Crowley, R. S. (2005). A data collection framework for capturing ITS data based on an agent communication standard. In Proceedings of the 20th Annual Meeting of the Association for the Advancement of Artificial Intelligence (AAAI) (pp. 23–30). Pittsburgh, PA: AAAI Press.

  • Metcalf, J., & Dunlosky, J. (2008). Metamemory. In H. Roediger (Ed.), Cognitive psychology of memory (Vol. 2, pp. 349–362). Oxford: Elsevier.

    Google Scholar 

  • Nelson, T. O. (1984). A comparison of current measures of the accuracy of feeling-of-knowing predictions. Psychological Bulletin, 95, 109–133.

    Google Scholar 

  • Nelson, S. B. (1996). Why study? How reasons for learning influence strategy selection. Educational Psychology Review, 8(4), 335–355.

    Google Scholar 

  • Nietfeld, J. L., Enders, C. K., & Schraw, G. (2006). A Monte Carlo comparison of measures of relative and absolute monitoring accuracy. Educational and Psychological Measurement, 66(2), 258–271. doi:10.1177/0013164404273945.

    Article  Google Scholar 

  • Obradovich, J. H., Smith, P. J., Guerlain, S., Rudmann, S., & Smith, J. (2000). Field evaluation of an intelligent tutoring system for teaching problem-solving skills in transfusion medicine. In Proceedings of the IEA 2000/HFES 2000 Congress. 44th Annual Meeting of the Human Factors and Ergonomics Society, San Diego, CA. London: Taylor & Francis.

  • Pintrich, P. R. (2000). The role of goal orientation in self-regulated learning. In M. Boekaerts, P. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 451–502). San Diego, CA: Academic Press.

    Chapter  Google Scholar 

  • Rogers, E. (1995). VIA-RAD: A blackboard-based system for diagnostic radiology. Artificial Intelligence in Medicine, 7, 343–360.

    Article  Google Scholar 

  • Schraw, G. (2009). A conceptual analysis of five measures of metacognitive monitoring. Metacognition and Learning, 4(1), 33–45. doi:10.1007/s11409-008-9031-3.

    Article  Google Scholar 

  • Sharples, M., Jeffery, N., du Boulay, B., Teather, B., Teather, D., & du Boulay, G. (2000). Structured computer-based training in the interpretation of neuroradiological images. International Journal of Medical Informatics, 60, 263–280.

    Article  Google Scholar 

  • Shute, V. J., & Zapata-Rivera, D. (2012). Adaptive educational systems. In P. Durlach (Ed.), Adaptive technologies for training and education (pp. 7–27). New York: Cambridge University Press.

    Chapter  Google Scholar 

  • Smith, P., Obradovich, J., Heintz, P., et al. (1998). Successful use of an expert system to teach diagnostic reasoning for antibody identification. In B. P. Goettl, H. M. Halff, C. L. Redfield, & V. J. Shute (Eds.), Proceedings of the 4th International Conference on Intelligent Tutoring Systems, ITS ‘98 (pp. 54–63). San Antonio, TX: Springer.

    Google Scholar 

  • Troxel, D. B. (2006). Medicolegal aspects of error in pathology. Archives of Pathology and Laboratory Medicine, 130(5), 617–619.

    Google Scholar 

  • VanLehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16(3), 227–265.

    Google Scholar 

  • VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197–221.

    Article  Google Scholar 

  • Voytovich, A. E., Rippey, R. M., & Suffredini, A. (1985). Premature conclusion in diagnostic reasoning. Journal of Medical Education, 60, 302–307.

    Google Scholar 

  • Winne, P. H. (2001). Self-regulated learning viewed from models of information processing. In J. Douglas, J. D. Hacker, & A. C. Graesser (Eds.), Self-regulated learning and academic achievement: Theoretical perspective (pp. 153–190). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Winne, P., & Hadwin, A. (1998). Studying as self-regulated learning. In D. Hacker, J. Dunlosky, & A. Graesser (Eds.), Motivation in educational theory and practice (pp. 227–304). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Winne, P., & Hadwin, A. (2008). The weave of motivation and self-regulated learning. In D. Schunk & B. Zimmerman (Eds.), Motivation and self-regulated learning: Theory, research, and application (pp. 297–314). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Woo, C. W., Evens, M. W., Freedman, R., et al. (2006). An intelligent tutoring system that generates a natural language dialogue using dynamic multi-level planning. Artificial Intelligence in Medicine, 38(1), 25–46.

    Article  Google Scholar 

  • Woolf, B. P. (2009). Building intelligent interactive tutors: Student-centered strategies for revolutionizing e-learning. Boston: Morgan Kaufmann/Elsevier.

    Google Scholar 

  • Zimmerman, B. (2006). Development and adaptation of expertise: The role of self-regulatory processes and beliefs. In K. A. Ericsson, P. Charness, P. Feltovich, & R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 705–722). New York: Cambridge University Press.

    Chapter  Google Scholar 

Download references

Acknowledgments

The authors gratefully acknowledge support of this research by the National Library of Medicine through Grant 5R01LM007891. We thank Ms. Lucy Cafeo for her expert editorial assistance. This work was conducted using the Protégé resource, which is supported by Grant LM007885 from the United States National Library of Medicine. SpaceTree was provided in collaboration with the Human-Computer Interaction Lab (HCIL) at the University of Maryland, College Park.

Author information

Authors and Affiliations

  1. Department of Educational and Counselling Psychology, McGill University, 3700 McTavish St., Montéal, QC, H3A 1Y2, Canada

    Reza Feyzi-Behnagh & Roger Azevedo

  2. Department of Biomedical Informatics, University of Pittsburgh, 5607 Baum Boulevard, BAUM 423, Rm 523, Pittsburgh, PA, 15206-3701, USA

    Elizabeth Legowski, Kayse Reitmeyer, Eugene Tseytlin & Rebecca S. Crowley

  3. Department of Pathology, University of Pittsburgh, Pittsburgh, PA, USA

    Rebecca S. Crowley

Authors
  1. Reza Feyzi-Behnagh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Roger Azevedo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Elizabeth Legowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kayse Reitmeyer
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Eugene Tseytlin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Rebecca S. Crowley
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rebecca S. Crowley.

Appendix

Appendix

See Tables 6 and 7.

Table 6 FOK contingency table
Full size table
Table 7 FOK accuracy statistics and equations
Full size table

Rights and permissions

Reprints and permissions

About this article

Cite this article

Feyzi-Behnagh, R., Azevedo, R., Legowski, E. et al. Metacognitive scaffolds improve self-judgments of accuracy in a medical intelligent tutoring system. Instr Sci 42, 159–181 (2014). https://doi.org/10.1007/s11251-013-9275-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11251-013-9275-4

Keywords

Search

Navigation