Generative AI in Simulation Debriefings: An Exploratory Study Using the Team-FIRST Framework

doi:10.21203/rs.3.rs-7724301/v1

Download PDF

Research Article

Generative AI in Simulation Debriefings: An Exploratory Study Using the Team-FIRST Framework

https://doi.org/10.21203/rs.3.rs-7724301/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background
Effective debriefings in simulation-based education require accurate observation of team interactions, yet facilitators face challenges due to cognitive load, observer bias, and the complexity of team dynamics. Generative artificial intelligence (AI) offers a potential means to support this process by analyzing verbal communication and providing structured feedback. This study explored how AI can contribute to teamwork observation and debriefing in high-fidelity medical simulations.

Methods
We conducted a qualitative, exploratory study using thematic analysis of simulation participants’ and debriefers’ experiences with AI-generated teamwork reports. Forty-one participants (anesthesia nurses, residents, and attendings) participated in high-fidelity scenarios at the University Hospital Zurich simulation center. Verbal interactions were transcribed with AI-assisted speech recognition and analyzed using two large language model–based systems (Isaac and ChatGPT-4o) guided by a prompt based on the Team-FIRST framework. Structured reports were generated for each scenario and reviewed by four experienced debriefers. Semi-structured interviews captured learners’ perspectives on being observed by AI.

Results
A total of 26 AI-generated reports and 27 learner interviews were analyzed. Debriefers valued the detailed transcripts and illustrative quotes, which supported structured feedback and captured observations that might otherwise be missed. Limitations included inaccuracies in categorization, misattribution of speakers, overly generalized interpretations, and the absence of contextual or nonverbal information. Learners expressed openness and optimism about AI’s potential benefits: efficiency, objectivity, and enhanced perception, while also raising concerns about transparency, data protection, interpretation errors, and risks of overreliance. Both groups emphasized the necessity of human oversight.

Conclusion
Generative AI can complement simulation debriefings by structuring communication data and highlighting teamwork patterns, thereby supporting reflective practice. Current limitations highlight the need for multimodal approaches, refined prompting strategies, and integration with expert facilitation to ensure AI functions as a support tool rather than a replacement in simulation-based education.

Trial Registration

BASEC ID: Req-2024-01642.

Generative Artificial Intelligence

Healthcare

Simulation

Debriefing

Large Language Models

Teamwork

Qualitative Thematic Analysis

Clinical Education Technology

Automated Assessments

Effective team interaction is crucial for patient safety and quality of care [1], with much recent scientific emphasis being placed on defining what constitutes good teamwork in healthcare [2–4], particularly regarding psychological safety, speaking up, team learning, and reflective practice [5–8]. Simulation training has become a key method for improving these interactions by providing a safe environment to practice and refine skills [9–12]. Observing team interactions during simulation training allows training facilitators to identify strengths and weaknesses [13, 14]. This information is critical for providing feedback during the structured debriefings[15–20], which are considered the core learning element of simulation training [18, 19, 21–25]. By observing interactions, helpful and unhelpful teamwork patterns can be discovered [26, 27]. Targeted watching allows facilitators to guide these debriefing sessions that reinforce learning objectives [28]. However, observing team interactions presents significant challenges. Team dynamics are neither linear nor straightforward [29–32]. Accurately perceiving these dynamics can be mentally demanding, especially in high-fidelity simulations [33–35]. Second, observers bring their biases and preconceptions to the observation process, affecting their interpretations [36]. Third, facilitators experience significant cognitive load [37–40]. This can lead to incomplete or inaccurate observations [41]. Currently, this process is often supported by technology through integrated sound and video recording and replaying systems, yet no theory-based, automated teamwork assessment solution exists. Integrating artificial intelligence (AI) technology presents a compelling solution addressing the above challenges in simulation. Generative AI is used to support documentation and clinical decision-making [42, 43]. A subcategory of generative AI, large language models, can understand context and generate coherent, human-like language outputs [44–47]. Researchers have begun investigating AI applications, highlighting both opportunities and limitations in simulation-based education [48–52].

With this exploratory study, we aimed to investigate how generative AI could be applied to enhance the observation of team interactions during simulation training. The AI analyzed transcripts of simulated cases and produced automated reports designed to support facilitators in the post-scenario debriefing. We were interested in how experienced debriefers assessed the quality and relevance of the AI-generated teamwork reports and in how learners experienced being analyzed by AI.

Study design and setting

In this qualitative, exploratory study, we used thematic analysis of (1) structured feedback and semi-structured interviews to evaluate debriefers’ perceptions of AI-generated simulation reports and (2) participants’ experiences with being observed by AI during simulated clinical scenarios. See Fig. 1 for the study´s flowchart. We surveyed the annual, three-week simulation training program of the University Hospital Zurich’s Institute of Anesthesiology and Perioperative Medicine (January and February 2025) at the hospital’s simulation center, a state-of-the-art simulation training and service facility, see Fig. 2.

Participants

Participants (total = 41) were nurses (n = 16), resident physicians (n = 17), and attending physicians (n = 8), all employed at the Institute of Anesthesiology and Perioperative Medicine. No specific limit or threshold on participation was set. See Table 1 for further Demographics.

Table 1
Study participants' Demographics. We present medians with [Q1, Q3].
category	(total n = 41)
Gender (female)	59%
Age (years)	34 [30, 43]
Experience in anesthesia (years)	7 [3, 19]
Role (count)	Nurse (non-anesthesia) = 5
	Anesthesia nurse = 11
	Anesthesia resident = 17
	Anesthesia attending = 8
Experience at USZ (years)	2 [1, 7]

Simulation-based training and debriefing

Simulation-based training occurred during work hours, and participants received in-house CME (continuous medical education) credits. Participation was voluntary. Five to seven participants trained together for a full day. Clinical simulation facilitators provided simulation-based training and debriefing. The facilitators welcomed participants and spent approximately 1-1.5 hours establishing a supportive learning environment, clarifying the learning objectives, and providing orientation to the simulation equipment [53]. Subsequently, learners alternated participating in and observing their colleagues participating in two simulated cases using SimMan3G (Laerdal, Stavanger, Norway) [54]. A structured methodology was applied to developing the scenarios;[55] the scenarios focused on sudden airway collapse after sedation (“too tired to sleep”, Scenario 1) and difficult airway management (“Slip party”, Scenario 2). See Supplementary Table 1 for the complete simulation scripts. Scenarios were audio- and video-recorded and were used during subsequent debriefings. Two facilitators conducted the debriefings immediately after the simulated cases were complete. Debriefings focused on Team FIRST competencies and were conducted with adherence to the Debriefing with Good Judgment and TeamGAINS approaches [17, 20]

RTF Prompt design

Based on the Team-FIRST framework [3] we developed a Role Task Format prompt (RTF) to automatically generate AI debriefing reports. The design of the prompt followed a clear input-to-output logic. The AI received anonymized transcripts of simulated clinical scenarios as input. The prompt guided the AI to process this material by mapping observed behaviors and communication patterns onto the ten categories defined by the Team-FIRST framework. The expected output was a report in which each category was presented as a separate section containing a summary and examples extracted from the transcript. Figure 1 illustrates the ten categories of the Team-FIRST framework. See Supplementary Fig. 1 for the complete RTF prompt.

Integration of AI for team verbal communication transcription and teamwork analysis

We asked the participants to wear RØDE lapel microphones while participating in the simulated cases, with audio signals transmitted directly to an iPad running the Isaac software (Saipient AI, Switzerland). This system generated real-time automatic transcripts of verbal team communication. We uploaded these transcripts into two separate AI-based systems (with the same underlying large language model) for analysis: (1) Isaac by Saipient AI, (2) ChatGPT-4o by OpenAI. The AI then compiled a structured and annotated report (Supplementary Material).

The debriefers’ perspective: Evaluation of AI-generated teamwork reports

Four members of our study group (JCS, MK, HH, MT), experienced simulation debriefers, evaluated the AI-generated reports. Author DWT assigned them two reports per scenario: one report generated using ChatGPT-4o and one using Isaac; both reports were based on the same transcript and prompt, and the origin of the report was blinded to the debriefers. The debriefers were asked to comment on the selection, structure, interpretation, and relevance of the content, as well as share any emotional reactions by open-ended written commentary (i.e., instruction: “My thoughts on the report (report begins below the line): Please enter your thoughts and feelings here.”).

The learners’ perspective: Evaluation of learners’ reactions

To understand how learners experienced being analyzed by AI, authors DWT, AR, and HW interviewed them after completing their scenarios before the end of the training day. The guiding question was:

What are your thoughts and feelings about being observed by AI during the simulation? Please share anything that comes to mind.

Data Analysis

We analyzed the learners’ written interview responses and the evaluators’ comments using a thematic analysis approach [56]. DWT and AR reviewed all raw responses and developed initial codes. They iteratively structured and consolidated the data into overarching thematic clusters, using Microsoft Word, with each response assigned a corresponding code. The resulting coding structure was reviewed, discussed, and validated with co-authors JCS and MK. The resulting categories and sample quotes from debriefers’ evaluations of the AI-generated teamwork reports are presented in Table 1. The categories and sample quotes from learners’ reactions to being analysed by AI are shown in Table 2.

Table 2
*Categories and example quotes from debriefers.* The table presents key themes, subthemes, and illustrative comments from simulation debriefers on the quality and usefulness of AI-generated reports.
Theme	Subtheme	Examples
Positive Feedback	Broader Observation Capture	“Examples [quotes] are appropriate. (However, I don’t remember them — which doesn’t necessarily mean anything, as there was a lot of noise in the control room and unrest in the scenario.)”
	Quote Selection Supports the Theme Well	“I share the above assessment — from my perspective, it's perfect.”
	Support for Debriefing	“I would benefit from this transcript in a debriefing.”
Categorization Accuracy Issues	Unclear, Inaccurate, or Overlapping Categorization	“The psychologically safe environment was present — but I didn’t see it reflected in these quotes.”
	Overgeneralized, Superficial, or Ambiguous Categorization	“These two points from the “mutual trust” category seem strongly out of context to me.”
	More Representative Quotes Could Have Been Chosen	“For me, structured communication would mean something like an ABCDE or a 10-for-10; the above statements are, in my opinion, not examples of that. I heard and coded an ABCDE during the scenario, and its content was different.”
	Desire for More Context or Stronger Exemplification	“I think the statements are part of structured communication; what was said before or after? Additional information would help me form an opinion.”
Identification of Problems	Misidentification of Speaker or Role	“I believe this information also came from outside the team and perhaps shouldn’t be included here. We would need a way to mark speakers and exclude certain roles from interpretation — for example, instructors or embedded simulated persons (ESPs).”
	Misinterpretation of Technical Terms or Abbreviations	“There are issues with the handling of technical vocabulary, although the key aspects were still recognizable.”
	Overconfident or Unwarranted Wording by AI	“Strange conclusion — how did the AI come up with that?”
	Systematic Biases in AI Interpretation	“It would also be interesting if central aspects that were missing could be suggested — for example, lack of coordination, absence of closed-loop communication, or the use of destructive language.”
Suggestions for Improvement	Link Quotes to Timestamps and Identify Speakers	“What I’m missing here are the names of the speakers. I find the timestamps extremely helpful for understanding the context and potentially showing the corresponding scene in the video.”
	Expand Beyond Text-Based Analysis	“I could also imagine that the AI might be able to distinguish the participants’ speaking volume over time — for example, when someone speaks loudly or quietly, and what impact that has on the team.”
	Improve Result Presentation and Clarity	“The quotes could be shown in a list and color-coded by theme—whether the AI classified them as relating to challenges, communication, or coordination skills.”

A total of 26 AI-generated reports from nine different simulation scenarios, conducted between January 21st and January 28th 2025, were reviewed and commented on by four experienced simulation debriefers. In addition, 27 participants took part in interviews about their experiences with the AI-based observation and analysis during the simulations.

The debriefers’ perspective on AI

We have identified four different themes in debriefers’ perspectives on AI-generated teamwork reports: (1) positive feedback, (2) categorization accuracy issues, (3) identification of problems, and (4) suggestions for improvement. Each of these four themes consists of three to four subthemes. Each thematic category is illustrated below with representative examples. Table 1 provides selected quotes from the debriefers’ comments. The coding trees derived from these debriefers' responses are presented in Supplementary Fig. 2. Supplementary Table 2 provides the full debriefers' feedback.

Debriefers’ theme 1: Positive Feedback

The first theme, positive feedback, consisted of four subthemes: The first subtheme, “Broader Observation Capture” emphasized that AI-generated transcripts often captured details that debriefers missed during live observation. “In practice, it’s impossible to write down this much, because you’re doing several things at once.” Another added, “I didn’t hear the first example — that would have been great.”

The second subtheme, “Quote Selection Supports Theme Well,” captured debriefers’ views that the generative AI models selected quotes that effectively illustrated specific and relevant aspects of teamwork. One debriefer noted, “These examples include well-chosen quotes, and the context is immediately clear. I find that particularly compelling.” The practical relevance for real debriefings was also emphasized: “A participant stated that the (AI) remarks helped to build trust, spontaneously and without the topic having been discussed (further by the debriefer).”

The third subtheme, “Support for Debriefing,” reflected debriefers’ perception that the concept offered valuable assistance for debriefing practice. One debriefer noted, “Based on my first impression yesterday, I think the concept has great potential.” Another emphasized the transcripts' usefulness: “This transcript would help me simply by providing a collection of quotes.”

Debriefers’ theme 2: Categorization Accuracy Issues

The second theme consisted of four subthemes around the difficulties of categorizing transcribed teamwork quotes correctly into the ten Team FIRST categories. The first subtheme was “Unclear, Inaccurate, or Overlapping Assignment”. One debriefer remarked, “Overall, I had the impression that the quotes didn’t fit the headings. I might have assigned them differently.” Another noted, “Many of the quotes were assigned to other team competencies, even though I remembered them differently.”

The second subtheme, “Overgeneralized, Superficial, or Ambiguous Categorization” reflected debriefers’ concerns that some assignments lacked depth, clarity, or contextual grounding. One debriefer noted: “The positive examples of closed-loop communication are well chosen but remain too vague in depth.” Another debriefer remarked: “Global statements are pointing to missing elements, but the specific points these judgments are based on are not provided.”

The third subtheme was “More Representative Quotes Could Have Been Chosen”. “These examples don’t quite fit, in my view.” Another remarked, “I find that inappropriate; quotes from later in the scenario would be more relevant.”

The fourth subtheme, “Desire for More Context or Stronger Exemplification,” captured debriefers’ wishes for more robust examples to support the assigned categories. “Some example sentences are too short, like ‘okay, good‘, but I need more context to make a valid judgment.” Another remarked, “The proactive coordination was a good observation, it’s a pity the full dialogue wasn’t included.”

Debriefers’ theme 3: Identification of Problems

The first subtheme, “Identification of Problems: Misidentification of Speaker or Role” captured debriefers’ concerns about inaccuracies in attributing statements to the correct team members. “This question came from the embedded simulated person, not from a team member, so the conclusion based on this example is incorrect.” Another added, “The above examples also seem to come from the simulation staff. In this case, from the ‘audio to the room’.”

The second subtheme, “Identification of Problems: Misinterpretation of Technical Terms or Abbreviations” reflected debriefers’ observations that technical language was sometimes inaccurately captured or translated. One noted, “The translation of specialized terms and abbreviations is still somewhat flawed and shows gaps.” However, another remarked, “It didn’t bother me that some technical terms weren’t correctly recognized. I could recall what was said.”

The third subtheme, “Overconfident or Unwarranted Wording by AI” captured concerns that some statements lacked critical nuance. “The wording is very assertive and explicit. For example, team members understand their roles and work together, but it doesn’t sufficiently question the basis of such claims.” Another warned, “It’s problematic to suggest that single events indicate a shared mental model. During the debriefing, it became clear that this model differed at least initially.”

The final subtheme, “Systematic Biases in AI Interpretation” captured concerns that the AI may have exhibited implicit assumptions in its interpretation of team dynamics. One debriefer questioned, “I wonder what knowledge the statement is based on: that this happened between a doctor and a nurse. Could a bias be at play, assuming certain roles automatically belong to certain professions? (For example, the assumption that the team leader must be the physician and not the nurse?)” Another noted possible positive bias was, "What I’d find interesting is whether the analysis could also highlight what was missing, for instance, absent closed-loop communication or destructive language. Not noticing what's missing creates blind spots that this tool could help uncover.”

Debriefers’ theme 4: Suggestions for Improvement

The final theme reflected the debriefers’ suggestions for improvement and included three subthemes. The first subtheme was “Link Quotes to Timestamps and Identify Speakers”. One noted, “What’s missing is the identification of speakers. You can’t tell if it’s a dialogue.” Another remarked, “The timestamp would help me find the scene in the video.”

The second subtheme, “Expand Beyond Text-Based Analysis” highlighted debriefers’ concerns that important nonverbal cues, such as tone of voice, body orientation, and overall atmosphere, were not captured. One remarked, “The calm and controlled tone that contributed to psychological safety is missing. This is something we pay close attention to as instructors.” Another noted, “Participants mentioned that they felt safe because team members faced each other, which the software cannot detect.” Also, the question was raised whether ambient noise levels, which may affect performance, were considered at all by the AI.

The third subtheme was “Improve Result Presentation and Clarity”. One suggestion was to embed timestamped quotes directly into existing tools, such as a Team FIRST debriefing event list: “I could imagine inserting these quotes with timestamps directly into our Team FIRST debriefing event log.” Another comment proposed a more structured transcript output by linking quotes to the scenario protocol or displaying them chronologically in a color-coded list indicating whether each quote relates to challenges, communication skills, or coordination.

The learners’ perspective on AI

Based on the interviews with the learners, we have identified four different themes in their perspectives on being observed and analyzed by an AI during simulation-based training: (1) influence of perceived AI observation, (2) perceived benefits, (3) perceived risks, and (4) suggested key features for AI. Three of these four themes consisted of two to seven subthemes, while the first theme stands independently. Table 2 presents corresponding examples from the participant interviews. Supplementary Fig. 3 and Supplementary Table 3 provide the full learners' feedback.

Learners’ theme 1: Influence of Perceived AI Observation

This theme included participants’ differing reactions to the awareness of being observed by AI. For some learners, the AI presence faded into the background and did not noticeably influence their behavior: “I forgot that AI was running during the simulation.” Others described a general sense of inhibition caused simply by being observed, regardless of who or what was watching: “You don’t feel entirely free to act. Just the fact of being watched causes tension, whether it’s AI or a human.”

Learners’ theme 2: Perceived Benefits

The first subtheme, “General Optimism About the Potential of AI Technologies”, captured participants’ hopeful outlook on the role of AI in clinical settings. One participant noted, “I believe it can offer real benefits and support.” Another emphasized: “New positive aspects should be explored. AI shouldn’t be rejected as a concept outright.”

The second subtheme, “Trust in AI”, included participants’ statements expressing high confidence in AI systems and little concern about potential risks. One participant noted, “I have no worries about data protection with AI,” while another remarked, “I don’t see any risks.”

The third subtheme, “Enhanced Perception Through AI”, included learners’ reflections on how AI could expand human perceptual limits. One remarked, “As humans, we can only perceive so much. We can’t see or hear everything. AI might help us notice other things.” Another added, “I think AI will help with recognizing (hidden) connections.”

The fourth subtheme, “Support for Generating Ideas and Structuring Feedback”, included participants’ views of AI as a valuable partner in creative and analytical processes. One stated, “We have the ideas, and AI helps us realize them.” Another added, “If personal data security is guaranteed, AI can help us in research.”

The fifth subtheme was “Increased Efficiency Through Automation”. One noted, “AI technology can take much work off our shoulders in the future.” Another emphasized, “AI increases efficiency, data can be captured and analyzed faster and more effectively.”

The sixth subtheme was “Objectivity and Impartiality”. “Unlike human analysis, it prevents bias based on past mistakes.” Another pointed out, “AI has no emotions, no memory — it doesn’t judge, making it impartial.”

The seventh subtheme, “Independence from Human Factors”, included participants’ appreciation of AI’s ability to operate without being influenced by human expectations or biases. One noted, “It misses nothing, while we might overlook things.” Another added, “Humans sometimes wait for participants to make mistakes, influenced by past groups or personal experiences.”

Learners’ theme 3: Perceived Risks

The first subtheme, ”Lack of Transparency”, included participants’ concerns about the nature of AI systems. One remarked, “AI can present facts in a way that makes them seem true. With humans, you notice uncertainty or emotion. A computer comes across as more confident. But is the information correct?” Another added, “I’m more worried about not fully understanding how AI works. There’s a tendency to think the computer knows better, but we need to think more critically.”

The second subtheme, “Data Protection and Security Concerns”, included participants’ unease about how their data is stored, processed, and shared. One participant stated they “would rather not have their voice uploaded to AI,” highlighting discomfort with audio data. Others felt more at ease “if only transcripts are generated,” provided it is clear “where the data is going.” Concerns about data being passed on to companies or possible “conflicts of interest” reflected a strong desire for transparency and control over one’s digital footprint.

The third subtheme, “Interpretation Errors”, included participants’ concerns about the risk of AI misinterpreting input. Users asked, “How can you check the information or know if the AI has made a mistake?” This uncertainty was seen as particularly problematic in emotionally nuanced situations. One participant noted that a key limitation is that it “cannot recognize emotions,” making it difficult for AI to fully grasp context or subtext, essential elements in human communication.

The fourth subtheme, “Loss of Cognitive, Social, and Communicative Abilities”, included participants’ concerns about the long-term impact of AI on essential human skills. One remarked, “You don’t need communication skills anymore,” referring to how AI tools increasingly take over tasks like writing or speaking. Another observed, “People just enter it into the GPS. You don’t have to remember anything,” pointing to the erosion of memory and cognitive independence. Participants feared that overreliance on AI might gradually diminish our ability to think, speak, and connect with others unaided.

The fifth subtheme, “Lack of Trust in AI”, included participants’ skepticism about overreliance on AI systems. One participant warned, “If humans no longer carry the main responsibility, that could be problematic,” pointing to the risks of delegating critical decisions. An example was automated ECG interpretation, which is “often incorrect and requires critical human review.” Trust in AI was limited, especially when accuracy and accountability were at stake.

The sixth subtheme, “Various Concerns About AI Technology”, included a broad range of practical and ethical questions raised by participants. One asked, “Does it understand Swiss German?”, highlighting limitations in language comprehension. Environmental concerns were also voiced, such as “Energy consumption for the servers is a concern. Where does the electricity come from?” These reflections culminated in a broader, critical question: “Do we need it?”, expressing a desire for more mindful and sustainable use of AI technologies.

Learners’ theme 4: Suggested Key Features for AI

The first subtheme, “Support for Human Situation Awareness”, included participants’ reflections on how AI output should be accessible and practically applicable. They emphasized that “being easy to perceive will be the key. It must be output that people will notice.” Participants also stressed the need for something “beneficial for everyone and transparent.”

The second subtheme, “Desire for Human Involvement in Evaluation”, included participants’ insistence on needing expert oversight when using AI. One noted that “AI is only as good as the input you give it,” and its output must be “evaluated with expert knowledge.” Others stressed that “a layperson cannot assess the AI’s output,” and that both input and output must be “cross-checked with human expertise.” If AI alone conducted the evaluation, “the human aspect would be missing.”

In this exploratory study, we used two AI models to analyze transcripts of simulated cases and produce teamwork reports designed to support debriefers in the post-scenario debriefing. Our aim was to examine the current capabilities of this technology without conducting a comparative evaluation. The quality and relevance of the AI-generated reports were assessed by experienced debriefers, while learners provided feedback on their experience of being analyzed by AI.

AI as a Complement, not a Replacement

Healthcare teams work in challenging environments where many things happen simultaneously [31, 57–59]. These teams' actions, behavior, and interactions are often complex [31]. Capturing and interpreting these dynamics accurately becomes particularly demanding in high-fidelity simulations replicating the urgency of real clinical settings [60]. Monitoring multiple team members simultaneously requires intense concentration and mental effort [39]. Facilitators frequently rely on basic tools like notepads or assistance from co-observers [41]. The cognitive effort needed to observe everything can be overwhelming, so important details might be missed or misunderstood [33]. The results of this study suggest that generative AI can serve as a valuable complement to debriefings, particularly in its ability to capture and organize large volumes of verbal interaction. Debriefers appreciated the inclusion of quotes that might have been missed and saw potential for AI to support reflection and structure in the debriefing process. Importantly, the AI system not only summarized content but also attempted to map specific quotes to well-defined team competencies. Using AI may also mitigate the influence of personal biases, which can distort evaluations and compromise the validity and quality of the feedback provided [33]. However, limitations were also evident. Debriefers identified issues with inaccurate categorization of transcribed content, lack of contextual understanding, and overconfident or wrong interpretations. While AI can efficiently process and structure language data, it lacks the nuanced understanding of team dynamics, nonverbal communication, and situational context that expert debriefers rely on.

Trust, Transparency, and Perceived Usefulness

Simulation participants generally responded with openness and curiosity toward AI-assisted observation. They appreciated the efficiency and objectivity that AI might offer. They expressed optimism about its potential role in clinical and educational contexts. This attitude is also reflected in recent literature emphasizing AI’s potential to support human intelligence in healthcare [61]. At the same time, they raised important concerns about transparency, data protection, and the risk of over-reliance. A recurring theme was the need for human oversight to ensure that AI support enhances, rather than replaces, human expertise and accountability.

Design Implications for AI Tools in Simulation Debriefing

Contextual Anchoring

Outputs should include speaker identification, timestamps, and scenario references to support accurate integration into debriefing workflows. Integration of speech recognition technology and automatically identifying who speaks and for how long would offer additional insights into interaction dynamics, which in turn may further improve learning outcomes [62]. Further, non-verbal communication, such as tone, gestures, and spatial dynamics, is an essential component of team collaboration and psychological safety [7, 60]. Yet, decoding these cues accurately is difficult, presenting a distinct challenge in a simulation environment.

Human-in-the-Loop

Participants and debriefers strongly emphasized the importance of human oversight. AI systems should function as support tools under the guidance of trained facilitators.

Limitations and Future Research

This study was conducted at a single simulation center with a limited number of scenarios and evaluators. The results provide qualitative depth but are not generalizable. Moreover, the analysis relied exclusively on one prompt and on textual transcripts, which included the local dialects, accounting for a source of transduction errors, which may limit the ability of AI to fully understand the team’s behavior. Future research should also explore multi-media AI approaches that combine video, audio, and environmental context.

Generative AI can support structured reflection in simulation-based education by helping organize team communication and highlight relevant interactions. However, technology’s current limitations lie in contextual understanding, multimodal perception, and categorical nuance, underscoring the need for thoughtful integration into human-led debriefing. This study provides a foundation for further research into how AI systems, guided by diverse prompting strategies and supported by expert oversight, can enhance team learning in clinical simulation.

Tables

Table 3
*Categories and example quotes from participants.* The table shows selected categories and corresponding illustrative statements reflecting participant perspectives on AI-based observation and perceived benefits.
Theme	Subtheme	Examples
Influence of perceived observation	-	“The fact that AI is listening and creating a transcript doesn’t change how I feel.”
Perceived Benefits	General Optimism About the Potential of AI Technologies	“AI is the future, definitely. You have to play along, or you’ll miss the trend. It’s a race — if you miss the jump, you’re left behind.”
	Trust in AI	“Not really a problem with how it’s being used.”
	Enhanced Perception Through AI	“An additional aid to humans, because it notices more.”
	Support for Generating Ideas and Structuring Feedback	“It can provide interesting input and be helpful when things are being compared.”
	Increased Efficiency Through Automation	“It can help us summarize information.”
	Objectivity and Impartiality	“Computers take emotions out of the equation — that might make them fairer in terms of evaluation.”
	Independence from Human Factors	“A human can’t be completely objective; AI is objective. AI looks only at what is given and puts it down on paper. That can be an advantage and a disadvantage.”
Perceived Risks	Lack of Transparency	“You’re left in the dark about the background — it gives you an uneasy feeling.”
	Data Protection and Security Concerns	“Conflicted about data protection: Where does the data go? How is it processed?”
	Interpretation Errors	“There may be a lack of understanding. AI certainly can’t empathize with people as well — humans are better at that.”
	Loss of Cognitive, Communicative, and Social Abilities	“Fear of neglecting one’s own intuition.”
	Lack of Trust in AI	“The lack of emotion is also a disadvantage, because certain aspects go unnoticed.”
	Various Concerns About AI Technology	“Can AI understand all dialects, all languages?”
Suggested Key-Features for AI	Support for Human Situation Awareness	“I want feedback to know that the AI understood what it was for.”
Suggested Key-Features for AI	Desire for Human Involvement in Evaluation	“The interpretation and debriefing should be led by a human.”

Ethical approval and consent to participate

The study was reviewed by the Cantonal Ethics Committee of Zurich and classified as not falling under the scope of the Human Research Act (BASEC ID: Req-2024-01642). All participants provided written informed consent before participation.

Consent for publication

none

Availability of data and materials

All data generated or analysed during this study are included in this published article (see Supplementary Table 1 and Supplementary Table 2).

Competing interests

DWT is an inventor of Visual Patient, Visual Patient Predictive, Visual Blood, Visual Clot, Visual Hemofilter, and Visual Patient Heart with intellectual property held by Philips and the University of Zurich. DWT receives research funding, honoraria, royalties, and travel support through joint development and licensing agreements. Instrumentation Laboratory–Werfen, the Swiss Foundation for Anaesthesia Research, and the International Symposium on Intensive Care and Emergency Medicine have provided additional honoraria and travel support. DWT serves on the Philips Patient Safety Advisory Board. ME is registered inventor of Visual Patient Heart. AK received lecture honoraria from Bayer AG (Switzerland), CSL Behring GmbH (Switzerland) and advisory honoraria from AstraZeneca AG (Switzerland) and Pharmacosmos (Switzerland). All other authors declare no conflict of interest.

Funding

This research received no speciﬁc grant from funding agencies in the public, commercial, or not-for-proﬁt sectors.

Authors' contributions

DWT: Supervision, Conceptualization, Data Collection, Project Administration, Investigation, Writing – Original Draft, ME: Data Curation, Formal Analysis,Methodology, Visualization, Writing – Original Draft, AR: Data Collection, Data Curation, HW: Data collection, HH: Data Collection, MT: Data Collection, AK: Data Collection, BG: Data Collection, JCS: Conceptualization, Data Collection, Writing – Original Draft, MK: Supervision, Conceptualization, Data Collection, Project Administration, Investigation, Writing – Original Draft, All authors: Writing – Review & Editing

Acknowledgements

none

Authors' information

DWT is an anesthesia attending physician and the head of the Visualization Technology Research Group at the Institute of Anesthesiology and Perioperative Medicine of the University Hospital Zurich. ME is a medical student with experience as an anesthesia simulation assistant. AR is an anesthesia resident physician. HW is a medical student. HH is an anesthesia nurse and simulation educator. MT is an anesthesia attending physician and simulation educator. AK is an anesthesia senior attending physician. BG is an anesthesia senior attending physician, simulation educator, and medical director of the simulation center. JCS is a psychologist and head of training and faculty development of the simulation center, and MK is a psychologist and director of the simulation center.

Declaration of generative AI and AI-assisted technologies in the writing process

During the preparation of this work, the authors used OpenAI (2024) ChatGPT 3.5/4.0 and DeepL (DeepL SE, Cologne, Germany) to improve the linguistic quality of the manuscript. After using this tool/service, the authors reviewed and edited the content as needed and take full responsibility for the content of the published article.

World Health Organization. Global Patient Safety Action Plan 2021–2030: Towards Eliminating Avoidable Harm in Health Care. 1st ed. Geneva: World Health Organization; 2021.
Tannenbaum SI, Greilich PE. The debrief imperative: building teaming competencies and team effectiveness. BMJ Qual Saf. 2023;32:125–8. doi:10.1136/bmjqs-2022-015259.
Greilich PE, Kilcullen M, Paquette S, Lazzara EH, Scielzo S, Hernandez J, et al. Team FIRST framework: Identifying core teamwork competencies critical to interprofessional healthcare curricula. J Clin Transl Sci. 2023;7:e106. doi:10.1017/cts.2023.27.
Tannenbaum SI, Traylor AM, Thomas EJ, Salas E. Managing teamwork in the face of pandemic: evidence-based tips. BMJ Qual Saf. 2021;30:59–63. doi:10.1136/bmjqs-2020-011447.
Tschan F, Keller S, Semmer NK, Timm-Holzer E, Zimmermann J, Huber SA, et al. Effects of structured intraoperative briefings on patient outcomes: multicentre before-and-after study. Br J Surg. 2021;109:136–44. doi:10.1093/bjs/znab384.
Kolbe M, Schmutz S, Seelandt JC, Eppich WJ, Schmutz JB. Team debriefings in healthcare: aligning intention and impact. BMJ. 2021;374:n2042. doi:10.1136/bmj.n2042.
Rudolph JW, Pian-Smith MCM, Minehart RD. Setting the stage for speaking up: psychological safety and directing care in acute care collaboration. Br J Anaesth. 2022;128:3–7. doi:10.1016/j.bja.2021.09.014.
Schmutz JB, Lei Z, Eppich WJ, Manser T. Reflection in the heat of the moment: The role of in-action team reflexivity in health care emergency teams. J Organ Behavior. 2018;39:749–65. doi:10.1002/job.2299.
Stefanidis D, Cook D, Kalantar-Motamedi S-M, Muret-Wagstaff S, Calhoun AW, Lauridsen KG, et al. Society for Simulation in Healthcare Guidelines for Simulation Training. Simul Healthc. 2024;19:S4-S22. doi:10.1097/SIH.0000000000000776.
Salas E, Rosen MA. Building high reliability teams: progress and some reflections on teamwork training. BMJ Qual Saf. 2013;22:369–73. doi:10.1136/bmjqs-2013-002015.
Salas E, Paige JT, Rosen MA. Creating new realities in healthcare: the status of simulation-based training as a patient safety improvement strategy. BMJ Qual Saf. 2013;22:449–52. doi:10.1136/bmjqs-2013-002112.
Diaz-Navarro C, Armstrong R, Charnetski M, Freeman KJ, Koh S, Reedy G, et al. Global consensus statement on simulation-based practice in healthcare. Adv Simul (Lond). 2024;9:19. doi:10.1186/s41077-024-00288-1.
Cheng A, Eppich W, Epps C, Kolbe M, Meguerdichian M, Grant V. Embracing informed learner self-assessment during debriefing: the art of plus-delta. Adv Simul (Lond). 2021;6:22. doi:10.1186/s41077-021-00173-1.
Kolbe M, Grande B, Spahn DR. Briefing and debriefing during simulation-based training and beyond: Content, structure, attitude and setting. Best Pract Res Clin Anaesthesiol. 2015;29:87–96. doi:10.1016/j.bpa.2015.01.002.
Kolbe M, Grande B, Lehmann-Willenbrock N, Seelandt JC. Helping healthcare teams to debrief effectively: associations of debriefers' actions and participants' reflections during team debriefings. BMJ Qual Saf. 2023;32:160–72. doi:10.1136/bmjqs-2021-014393.
Duff JP, Morse KJ, Seelandt J, Gross IT, Lydston M, Sargeant J, et al. Debriefing Methods for Simulation in Healthcare: A Systematic Review. Simul Healthc. 2024;19:S112-S121. doi:10.1097/SIH.0000000000000765.
Rudolph JW, Simon R, Rivard P, Dufresne RL, Raemer DB. Debriefing with good judgment: combining rigorous feedback with genuine inquiry. Anesthesiol Clin. 2007;25:361–76. doi:10.1016/j.anclin.2007.03.007.
Rudolph JW, Simon R, Raemer DB, Eppich WJ. Debriefing as formative assessment: closing performance gaps in medical education. Acad Emerg Med. 2008;15:1010–6. doi:10.1111/j.1553-2712.2008.00248.x.
Fey MK, Roussin CJ, Rudolph JW, Morse KJ, Palaganas JC, Szyld D. Teaching, coaching, or debriefing With Good Judgment: a roadmap for implementing "With Good Judgment" across the SimZones. Adv Simul (Lond). 2022;7:39. doi:10.1186/s41077-022-00235-y.
Kolbe M, Weiss M, Grote G, Knauth A, Dambach M, Spahn DR, Grande B. TeamGAINS: a tool for structured debriefings for simulation-based team trainings. BMJ Qual Saf. 2013;22:541–53. doi:10.1136/bmjqs-2012-000917.
Meguerdichian MJ, Trottier DG, Campbell-Taylor K, Bentley S, Bryant K, Kolbe M, et al. When common cognitive biases impact debriefing conversations. Adv Simul (Lond). 2024;9:48. doi:10.1186/s41077-024-00324-0.
Tavares W, Eppich W, Cheng A, Miller S, Teunissen PW, Watling CJ, Sargeant J. Learning Conversations: An Analysis of the Theoretical Roots and Their Manifestations of Feedback and Debriefing in Medical Education. Acad Med. 2020;95:1020–5. doi:10.1097/ACM.0000000000002932.
Rudolph JW, Foldy EG, Robinson T, Kendall S, Taylor SS, Simon R. Helping without harming: the instructor's feedback dilemma in debriefing–a case study. Simul Healthc. 2013;8:304–16. doi:10.1097/SIH.0b013e318294854e.
Cheng A, Palaganas J, Eppich W, Rudolph J, Robinson T, Grant V. Co-debriefing for simulation-based education: a primer for facilitators. Simul Healthc. 2015;10:69–75. doi:10.1097/SIH.0000000000000077.
Des Roze Ordons AL, Eppich W, Lockyer J, Wilkie RD, Grant V, Cheng A. Guiding, Intermediating, Facilitating, and Teaching (GIFT): A Conceptual Framework for Simulation Educator Roles in Healthcare Debriefing. Simul Healthc. 2022;17:283–92. doi:10.1097/SIH.0000000000000619.
Kolbe M, Marty A, Seelandt J, Grande B. How to debrief teamwork interactions: using circular questions to explore and change team interaction patterns. Adv Simul (Lond). 2016;1:29. doi:10.1186/s41077-016-0029-7.
Salas E, Klein C, King H, Salisbury M, Augenstein JS, Birnbach DJ, et al. Debriefing medical teams: 12 evidence-based best practices and tips. Jt Comm J Qual Patient Saf. 2008;34:518–27. doi:10.1016/s1553-7250(08)34066-5.
Kolb DA. Experimental learning: Experience as the source of learning and development. Englewood Cliffs, N.J.: Prentice-Hall; 1984.
Kolbe M, Grote G, Waller MJ, Wacker J, Grande B, Burtscher MJ, Spahn DR. Monitoring and talking to the room: autochthonous coordination patterns in team interaction and performance. J Appl Psychol. 2014;99:1254–67. doi:10.1037/a0037877.
Lei Z, Waller MJ, Hagen J, Kaplan S. Team adaptiveness in dynamic contexts: Contextualizing the roles of interaction patterns and in-process planning. Group & organization management. 2016.
Zijlstra FRH, Waller MJ, Phillips SI. Setting the tone: Early interaction patterns in swift-starting teams as a predictor of effectiveness. European Journal of Work and Organizational Psychology. 2012;21:749–77. doi:10.1080/1359432X.2012.690399.
Grote G, Kolbe M, Waller MJ. The dual nature of adaptive coordination in teams. Organizational Psychology Review. 2018;8:125–48. doi:10.1177/2041386618790112.
Brauner E, Boos M, Kolbe M, editors. The Cambridge handbook of group interaction analysis. Cambridge, United Kingdom, New York, NY, USA, Port Melbourne, Australia: Cambridge University Press; 2018.
Kolbe M, Boos M. Laborious but Elaborate: The Benefits of Really Studying Team Dynamics. Frontiers in Psychology. 2019;10:1478. doi:10.3389/fpsyg.2019.01478.
Weingart LR. How did they do that? The ways and means of studying group processes; 1997.
Seelandt JC, Tschan F, Keller S, Beldi G, Jenni N, Kurmann A, et al. Assessing distractors and teamwork during surgery: developing an event-based method for direct observation. BMJ Qual Saf. 2014;23:918–29. doi:10.1136/bmjqs-2014-002860.
Meguerdichian M, Walker K, Bajaj K. Working memory is limited: improving knowledge transfer by optimising simulation through cognitive load theory. BMJ Simul Technol Enhanc Learn. 2016;2:131–8. doi:10.1136/bmjstel-2015-000098.
Fraser KL, Meguerdichian MJ, Haws JT, Grant VJ, Bajaj K, Cheng A. Cognitive Load Theory for debriefing simulations: implications for faculty development. Adv Simul (Lond). 2018;3:28. doi:10.1186/s41077-018-0086-1.
Fraser K, McLaughlin K. Temporal pattern of emotions and cognitive load during simulation training and debriefing. Med Teach. 2019;41:184–9. doi:10.1080/0142159X.2018.1459531.
Gardner AK, Rodgers DL, Steinert Y, Davis R, Condron C, Peterson DT, et al. Mapping the Terrain of Faculty Development for Simulation: A Scoping Review. Simul Healthc. 2024;19:S75-S89. doi:10.1097/SIH.0000000000000758.
Mommers L, Verstegen D, Dolmans D, van Mook WNKA. Observation of behavioural skills by medical simulation facilitators: a cross-sectional analysis of self-reported importance, difficulties, observation strategies and expertise development. Adv Simul (Lond). 2023;8:28. doi:10.1186/s41077-023-00268-x.
Takita H, Kabata D, Walston SL, Tatekawa H, Saito K, Tsujimoto Y, et al. A systematic review and meta-analysis of diagnostic performance comparison between generative AI and physicians. NPJ Digit Med. 2025;8:175. doi:10.1038/s41746-025-01543-z.
Adams R, Henry KE, Sridharan A, Soleimani H, Zhan A, Rawat N, et al. Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis. Nat Med. 2022;28:1455–60. doi:10.1038/s41591-022-01894-0.
Wang L, Wan Z, Ni C, Song Q, Li Y, Clayton E, et al. Applications and Concerns of ChatGPT and Other Conversational Large Language Models in Health Care: Systematic Review. J Med Internet Res. 2024;26:e22769. doi:10.2196/22769.
Bednarczyk L, Reichenpfader D, Gaudet-Blavignac C, Ette AK, Zaghir J, Zheng Y, et al. Scientific Evidence for Clinical Text Summarization Using Large Language Models: Scoping Review. J Med Internet Res. 2025;27:e68998. doi:10.2196/68998.
Shashikumar SP, Mohammadi S, Krishnamoorthy R, Patel A, Wardi G, Ahn JC, et al. Development and prospective implementation of a large language model based system for early sepsis prediction. NPJ Digit Med. 2025;8:290. doi:10.1038/s41746-025-01689-w.
Zhang K, Meng X, Yan X, Ji J, Liu J, Xu H, et al. Revolutionizing Health Care: The Transformative Impact of Large Language Models in Medicine. J Med Internet Res. 2025;27:e59069. doi:10.2196/59069.
Barra FL, Rodella G, Costa A, Scalogna A, Carenzo L, Monzani A, Della Corte F. From prompt to platform: an agentic AI workflow for healthcare simulation scenario design. Adv Simul (Lond). 2025;10:29. doi:10.1186/s41077-025-00357-z.
Berger-Estilita J, Gisselbaek M, Devos A, Chan A, Ingrassia PL, Meco BC, et al. AI and inclusion in simulation education and leadership: a global cross-sectional evaluation of diversity. Adv Simul (Lond). 2025;10:26. doi:10.1186/s41077-025-00355-1.
Hong E, Kazmir S, Dylik B, Auerbach M, Rosati M, Athanasopoulou S, et al. Exploring the Use of a Large Language Model in Simulation Debriefing: An Observational Simulation-Based Pilot Study. Simul Healthc 2025. doi:10.1097/SIH.0000000000000861.
Rodgers DL, Needler M, Robinson A, Barnes R, Brosche T, Hernandez J, et al. Artificial Intelligence and the Simulationists. Simul Healthc. 2023;18:395–9. doi:10.1097/SIH.0000000000000747.
Maaz S, Palaganas JC, Palaganas G, Bajwa M. A guide to prompt design: foundations and applications for healthcare simulationists. Front Med (Lausanne). 2024;11:1504532. doi:10.3389/fmed.2024.1504532.
Rudolph JW, Raemer DB, Simon R. Establishing a safe container for learning in simulation: the role of the presimulation briefing. Simul Healthc. 2014;9:339–49. doi:10.1097/SIH.0000000000000047.
Laerdal Inc. SimMan 3G PLUS. 23.08.2025. https://laerdal.com/ch/products/simulation-training/emergency-care-trauma/simman-3g/.
Grande B, Schick C, Scherrer A, Nef A, Marty A, Kolbe M. Entwicklung von Simulationsszenarien. St Pierre, Michael; Breuer, Georg. Simulation in der Medizin: Grundlegende Konzepte - Klinische Anwendung (2 Auflage). 2018.
Braun V, Clarke V, Hayfield N, Terry G. Thematic Analysis. In: Liamputtong P, editor. Handbook of research methods in health social sciences: With 192 figures and 81 tables. Singapore: Springer Singapore; 2019. p. 843–860. doi:10.1007/978-981-10-5251-4_103.
Kolbe M, Burtscher MJ, Manser T. Co-ACT–a framework for observing coordination behaviour in acute care teams. BMJ Qual Saf. 2013;22:596–605. doi:10.1136/bmjqs-2012-001319.
Ishak AW, Ballard DI. Time to Re-Group. Small Group Research. 2012;43:3–29. doi:10.1177/1046496411425250.
Sundstrom E, Meuse KP de, Futrell D. Work teams: Applications and effectiveness. American Psychologist. 1990;45:120–33. doi:10.1037/0003-066X.45.2.120.
Kolbe M, Eppich W, Rudolph J, Meguerdichian M, Catena H, Cripps A, et al. Managing psychological safety in debriefings: a dynamic balancing act. BMJ Simul Technol Enhanc Learn. 2020;6:164–71. doi:10.1136/bmjstel-2019-000470.
Henckert D, Malorgio A, Schweiger G, Raimann FJ, Piekarski F, Zacharowski K, et al. Attitudes of Anesthesiologists toward Artificial Intelligence in Anesthesia: A Multicenter, Mixed Qualitative-Quantitative Study. J Clin Med 2023. doi:10.3390/jcm12062096.
Brutschi R, Wang R, Kolbe M, Weiss K, Lohmeyer Q, Meboldt M. Speech recognition technology for assessing team debriefing communication and interaction patterns: An algorithmic toolkit for healthcare simulation educators. Adv Simul (Lond). 2024;9:42. doi:10.1186/s41077-024-00315-1.

Competing interest reported. DWT is an inventor of Visual Patient, Visual Patient Predictive, Visual Blood, Visual Clot, Visual Hemofilter, and Visual Patient Heart with intellectual property held by Philips and the University of Zurich. DWT receives research funding, honoraria, royalties, and travel support through joint development and licensing agreements. Instrumentation Laboratory–Werfen, the Swiss Foundation for Anaesthesia Research, and the International Symposium on Intensive Care and Emergency Medicine have provided additional honoraria and travel support. DWT serves on the Philips Patient Safety Advisory Board. ME is registered inventor of Visual Patient Heart. AK received lecture honoraria from Bayer AG (Switzerland), CSL Behring GmbH (Switzerland) and advisory honoraria from AstraZeneca AG (Switzerland) and Pharmacosmos (Switzerland). All other authors declare no conflict of interest.

SIMAISupplement.docx

Download PDF

Editorial decision: Revision requested
01 Nov, 2025
Reviews received at journal
16 Oct, 2025
Reviews received at journal
07 Oct, 2025
Reviewers agreed at journal
05 Oct, 2025
Reviewers agreed at journal
02 Oct, 2025
Reviewers invited by journal
02 Oct, 2025
Editor assigned by journal
02 Oct, 2025
Submission checks completed at journal
30 Sep, 2025
First submitted to journal
26 Sep, 2025

You are reading this latest preprint version

Generative AI in Simulation Debriefings: An Exploratory Study Using the Team-FIRST Framework

Status:

Version 1

Abstract

Figures

Introduction

Objectives

Methods

Study design and setting

Participants

Simulation-based training and debriefing

RTF Prompt design

Integration of AI for team verbal communication transcription and teamwork analysis

The debriefers’ perspective: Evaluation of AI-generated teamwork reports

The learners’ perspective: Evaluation of learners’ reactions

Data Analysis

Results

The debriefers’ perspective on AI

Debriefers’ theme 1: Positive Feedback

Debriefers’ theme 2: Categorization Accuracy Issues

Debriefers’ theme 3: Identification of Problems

Debriefers’ theme 4: Suggestions for Improvement

The learners’ perspective on AI

Learners’ theme 2: Perceived Benefits

Learners’ theme 3: Perceived Risks

Learners’ theme 4: Suggested Key Features for AI

Discussion

AI as a Complement, not a Replacement

Trust, Transparency, and Perceived Usefulness

Design Implications for AI Tools in Simulation Debriefing

Contextual Anchoring

Human-in-the-Loop

Limitations and Future Research

Conclusion

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1