Learning collaborative reasoning

Richters, Constanze (2024): Learning collaborative reasoning: foundations of adaptive reflection support in agent-based simulations. Dissertation, LMU München: Fakultät für Psychologie und Pädagogik

Abstract

As a complex skillset, collaborative diagnostic reasoning is crucial in various professional contexts. Professionals (e.g., physicians or teachers) engage in collaborative diagnostic activities, which include individual activities—such as generating and evaluating evidence and hypotheses and drawing conclusions—and collaborative activities—such as eliciting and sharing evidence and hypotheses. High-quality diagnostic outcomes such as accurate diagnoses with well-supported, evidence-based justifications require collaborating professionals to apply different types of knowledge such as content knowledge and collaboration knowledge. Recently, simulation-based learning and scaffolding have been found to be effective instructional means for developing complex skills such as collaborative diagnostic reasoning in higher education. However, a major challenge that educational and psychological researchers have emphasized in light of recent technological advances is how to support learners on the basis of their individual needs. Understanding how learner characteristics such as prerequisites, behavior, or performance are related to their needs for support is critical for effectively adapting instructional support. Various coarse and fine-grained approaches can be used to provide foundations for adaptation. Researchers have frequently used conventional product data, such as prior knowledge data, to investigate the effects of scaffolding for learners with different prior knowledge levels. A newer direction involves analyzing computer-system-generated process data, which can help researchers understand problem-solving processes and their relationships with task outcomes. With help of machine learning, process data may facilitate finer adjustments in real time. Addressing both approaches, the present PhD dissertation aims to lay foundations for adaptive instructional support for learning collaborative diagnostic reasoning. Previous studies have demonstrated that agent-based simulations, which enable a highly standardized training of collaborative processes, effectively enhance collaborative diagnostic reasoning when combined with collaboration scripts that additionally facilitate collaborative processes. The research in this dissertation builds on and extends previous research by proposing reflection guidance, which encourages learners to reflect on their own activities and performance, as a new effective type of scaffolding in collaborative diagnostic reasoning. The dissertation comprises three studies conducted in the same agent-based medical simulation where participants in the role of internists diagnosed diseases for several patient cases while collaborating with an agent-based expert radiologist to gather further evidence for the cases. Experimental Studies 1 and 2 investigated conditions under which various types of scaffolding—notably reflection guidance—enhanced the learning of collaborative diagnostic reasoning. The effectiveness of different forms of reflection guidance, tailored to different collaborative diagnostic activities and providing different levels of structure, was examined on the basis of a priori hypotheses. Study 3 used machine learning to analyze collaborative diagnostic reasoning processes and their relationships to the diagnostic outcome. Study 1 examined the effects of reflection guidance addressing individual activities and collaboration scripts as a function of learners’ prior content and collaboration knowledge on collaborative diagnostic reasoning. Collaborative diagnostic reasoning was operationalized as performance in evidence and hypothesis sharing (collaborative activities) and diagnostic accuracy and justification (diagnostic outcomes). Furthermore, Study 1 explored how reflection and collaboration affected the accuracy of suspected diagnoses throughout the reasoning process. Medical students were given questions to help them individually reflect on their initial suspected diagnoses, scripts while collaborating with the radiologist, both, or no support. Results showed that reflection improved hypothesis sharing for learners with high levels of content knowledge, whereas collaboration scripts improved evidence sharing for learners with low levels of content knowledge, suggesting that reflecting on individual activities activates prior content knowledge and prepares learners for collaboration if they have sufficient prior knowledge. Whereas neither collaboration scripts nor reflection guidance improved diagnostic outcomes, collaboration alone improved learners’ diagnostic accuracy regardless of their prior knowledge level. These findings may be explained by the integration of external knowledge into the diagnostic process through collaboration with the agent. Study 2 examined the effects of reflection guidance addressing collaborative activities on collaborative diagnostic reasoning, using the same operationalization as Study 1 and considering learners’ prior collaboration knowledge. Medical students received either low-structured (no detailed questions) or high-structured (detailed questions) guidance to help them individually reflect on their collaborative activities or no support at all. Results revealed that reflection guidance was beneficial for learners with low levels of collaboration knowledge. Low-structured guidance improved evidence sharing, diagnostic accuracy, and diagnostic justification, indicating that reflecting on collaborative activities holds promise for not only activating but also restructuring prior knowledge. High-structured guidance improved only diagnostic justification, indicating that different levels of structure in reflection are differentially beneficial for different subskills because different underlying knowledge bases result in different subskill levels. Both low- and high-structured guidance were unhelpful or even detrimental for learners with high collaboration knowledge, suggesting that these learners may require a broader reflection prompt. Study 3 investigated whether and how quickly diagnostic accuracy (diagnostic outcomes) could be predicted from collaborative diagnostic activities using machine learning. Log files of medical students and physicians working in the agent-based simulation were coded as collaborative diagnostic activities, including evidence generation, evidence elicitation, evidence sharing, hypothesis sharing, and drawing conclusions. Bigrams depicting the time spent on and switches between activities were used to train classification algorithms to predict the diagnostician’s final diagnosis as either correct or incorrect. Results indicated that diagnostic success was more reliably predicted than failure and before case completion, suggesting that the behavior of unsuccessful diagnosticians underlies diverse cognitive misbehavior, whereas successful diagnosticians exhibit less behavioral variation. Successful diagnosticians spent more time on individual activities, indicating they have an appropriate initial cognitive case representation, whereas unsuccessful diagnosticians spent more time on collaborative activities and switched between individual and collaborative activities. The dissertation provides theoretical and practical implications for adaptive instructional support for learning collaborative diagnostic reasoning in agent-based simulations. First, guidance on how to reflect on collaborative activities seems particularly promising for learning different subskills of collaborative diagnostic reasoning. A lower degree of structure is thereby likely to promote learning more than a higher degree of structure, regardless of learners’ prior knowledge levels. Considering learners’ levels in specific subskills beyond prior knowledge seems promising for designing effective reflection support. Nonetheless, the diverse results on the effectiveness of reflection for learners with different levels of prior knowledge in Studies 1 and 2 also highlight the difficulty of comparing and generalizing reflection effects, as well as the difficulty of quantifying the complexity of reflection processes. A complex interplay between factors, such as the content of reflection (e.g., diagnostic decision-making vs. collaboration), learners’ prior knowledge and skill level, and the level of structure provided influences the effectiveness of reflection. Future research could continue to strive to objectively scale different levels of structure in reflection support to allow reliable comparisons of effects in the future. Second, the dissertation highlights the importance of theory-based process data to identify subtle differences in collaborative diagnostic reasoning processes between successful and unsuccessful diagnosticians. These findings offer reliable indications of learners’ areas of struggle or proficiency in diagnostic cases, allowing for more fine-grained and dynamic instructional support. Such support could enhance the overall effectiveness of simulation-based learning for complex skills such as collaborative diagnostic reasoning in the future.

• Dokumententyp: Dissertationen (Dissertation, LMU München)
• Keywords: Simulation-Based Learning, Reflection, Collaborative Diagnostic Reasoning, Machine Learning, (Adaptive) Scaffolding
• Fakultäten: Fakultät für Psychologie und Pädagogik
• Sprache der Hochschulschrift: Englisch
• Datum der mündlichen Prüfung: 5. Juni 2024
• 1. Bericht­erstatter:in: Fischer, Frank