Objective: To determine whether a dynamical analysis of neural and communication data streams provide fine-grained insights into healthcare team debriefings.

Background: Debriefing plays a key role in experiential learning activities such as healthcare simulation because it bolsters the transfer of experience into learning through a process of reflection. There have been few studies examining the neural and communication dynamics of teams as team members are supported by trained facilitators in making better sense of their performance.

Method: Electroencephalographic (EEG) - derived brain waves and speech were recorded from experienced and medical student healthcare teams during post-simulation debriefings. Quantitative estimates of the neurodynamic organizations of individual team members and the team were modeled from the EEG data streams at different scalp locations and at frequencies from 1-40 Hz. In parallel the dynamics of speech turn taking were quantified by recurrence frequency analysis.

Results: Neurodynamic organizations were preferentially detected from sensors over the parietal lobes with activities present in the alpha, beta and gamma frequency bands. Rhythmic structures emerged as correlations between speech, discussion blocks and team & team member neurodynamic organizations.

Conclusion: Organizational representations help reveal the neurodynamic, communication, and cognitive structures of debriefing.

Application: The quantitative neurodynamic and communication measures will allow direct comparisons of debriefing structures across teams and debriefing protocols.

Neuroergonomics: Quantitative Modeling of Individual, Shared, and Team Neurodynamic Information

Ronald H. Stevens, Trysha L. Galloway and Ann Willemsen-Dunlap

Abstract:

Objective: The aim of this study was to use the same quantitative measure and scale to directly compare the neurodynamic information/organizations of individual team members with those of the team.

Background: Team processes are difficult to separate from those of individual team members due to the lack of quantitative measures that can be applied to both process sets.

Method: Second-by-second symbolic representations were created of each team member's electroencephalographic power, and quantitative estimates of their neurodynamic organizations were calculated from the Shannon entropy of the symbolic data streams. The information in the neurodynamic data streams of health care (N = 24), submarine navigation (N = 12), and high school problem-solving (N = 13) dyads was separated into the information of each team member, the information shared by team members, and the overall team information.

Results: Most of the team information was the sum of each individual's neurodynamic information. The remaining team information was shared among the team members. This shared information averaged ~15% of the individual information, with momentary levels of 1% to 80%.

Conclusion: Continuous quantitative estimates can be made from the shared, individual, and team neurodynamic information about the contributions of different team members to the overall neurodynamic organization of a team and the neurodynamic interdependencies among the team members.

Application: Information models provide a generalizable quantitative method for separating a team's neurodynamic organization into that of individual team members and that shared among team members.

The ubiquity of teams in society has made the development of dynamic models of teamwork a priority for many sectors of the human factors community, especially models that can be applied in multiple team settings through the capture of generic properties of team member dynamics. In this presentation we will explore how we can begin to ‘make sense’ of the information in these data streams that are being recorded from teams at increasingly higher resolutions, and what can we expect to learn about teams through the application of different representations, transformations and aggregations of information. The goal is to describe how meaning can be extracted from large-scale dynamical data to make inferences about teamwork that are useful in both the theoretical and practical sense.

We have developed an information-organization approach for detecting and quantitating the fluctuating neurodynamic organizations in teams. Neurodynamic organization is the propensity of team members to enter into prolonged (seconds-minutes) metastable neurodynamic relationships as they encounter and resolve disturbances to their normal rhythms. Team neurodynamic organizations were detected and modeled by transforming the physical units of each team member’s EEG power levels into Shannon entropy-derived information units about the team’s organization and synchronization. Entropy is a measure of the variability or uncertainty of information in a data stream. This physical unit to information unit transformation bridges micro level social coordination events with macro level expert observations of team behavior allowing multimodal comparisons across the neural, cognitive and behavioral time scales of teamwork.

Each second, neurodynamic symbols (NS) were created showing the EEG power spectral densities (PSD) at the 1-40 Hz frequency bins for each team member. These data streams contained a history of the team’s across-brain neurodynamic organizations, and the degree of organization was calculated from a moving average window of the Shannon entropy of task segments. An example of the symbol dynamics and the corresponding entropy profile for a healthcare team performance is shown in Figure 1. The symbol expressions were not temporally uniform, but were punctuated by segments of restricted NS expression that persisted for 1 – 4 minutes. Qualitatively, the NS expressions varied with the training session segments suggesting a neurodynamic form of task specificity.

Fig. 1. Expression of neurodynamic symbols for a healthcare performance.

The EEG PSD were separated into high, medium and low categories compared with the performance average. Every second each person’s EEG power levels were classified into the lower, middle, or upper third of their performance average and assigned the symbols -1, 1, and 3. With three team members and three EEG power levels, the symbol space consisted of 27 symbols. In the above figure a symbol was added for each second of the performance. The lines indicate junctions between different training segments. The fluctuating trace is the entropy of the symbol stream that was calculated over a 60s moving window. More recently the information in the neurodynamic data streams of teams engaged in naturalistic decision making (healthcare (n=24), submarine navigation (n=12) or high school collaborative problem-solving (n=11) was separated into information unique to each team member, the information shared by two or more team members, and team-specific information related to interactions with the task and team members. Most of the team information consisted of the information contained in an individual’s neurodynamic data stream. The information in an individual’s data stream that was shared with another team member was highly variable being 1-60% of the total information in another person’s data stream. From the shared, individual, and team information it becomes possible to quantitatively describe the dynamics of each team member during the task, as well as the neurodynamic interactions between members of the team. The innovation of this study is the potential it raises for developing globally applicable quantitative models of both individual and team dynamics that will allow comparisons to be made across teams, tasks and training protocols.

This paper describes how meaning can be extracted from large-scale dynamical data to make inferences about teamwork that are useful in both the theoretical and practical sense. The dynamics of an anesthesiology team are viewed from the perspectives of: 1) changes in the team's neurodynamic organizations with large and small changes in the task; 2) how team member's neurodynamics contribute to team neurodynamics; 3) the relationships between task events, heart-rate and neural dynamic organizations; 4) the linkages between speech flow, team and team member neurodynamics and topic discussions during Debriefing; and, 5) the micro-scale neural dynamics reflecting the involvement of the parietal lobes and gamma frequencies. These examples show how different sources of team data can contribute to multi-modal understandings of individual and teams dynamics that span micro and macro scales of teamwork.

This chapter describes a neurodynamic modeling approach which may be useful for dynamically assessing teamwork in healthcare and military situations. It begins with a description of electroencephalographic (EEG) signal acquisition and the transformation of the physical units of EEG signals into quantities of information. This transformation provides quantitative, dynamic, and generalizable neurodynamic models that are directly comparable across teams, tasks, training protocols, and team experience levels using the same measurement scale, bits of information. These bits of information can be further used to dynamically guide team performance or to provide after-action feedback that is linked to task events and team actions.

These ideas are instantiated and expanded in the second section of the chapter by showing how these data abstractions, compressions, and transformations take advantage of the natural information redundancy in biologic signals to substantially reduce the number of data dimensions, making the incorporation of neurodynamic feedback into Intelligent Tutoring Systems (ITSs) achievable.

We have investigated the correlations between the levels of team resilience as determined by expert raters and the degree of the teams' neurodynamic organization determined by electroencephalography (EEG). Neurophysiologic models were created from submarine navigation teams that captured their dynamic responses to changing task environments during required simulation training. The teams were simultaneously rated for resilience by two expert observers using a team process rubric developed and adopted by the U.S. Navy. Symbolic neurodynamic representations of the power levels in the 1-40 Hz EEG frequency bands were created each second from each crew member. These symbols captured the EEG power of each team member in the context of the other team members and also in the context of the task. Quantitative estimates of the changes in the symbol distributions over time were constructed by a moving window of Shannon entropy. Periods of decreased entropy were observed when the distribution of symbols in this window became smaller, for example, when there were prolonged and restricted relationships between the EEG power levels among the crew members, that is, less neurodynamic flexibility. Team resilience was correlated with the neurodynamic entropy levels. The correlation sign, however, depended on the training segment with negative correlations during the presimulation briefing and positive correlations in the scenario training segment. These studies indicate that neurodynamic representations of teams can be generated that bridge the microscales of EEG measurement with macroscales of behavioral ratings. From a training perspective, the results suggest that neurodynamic rigidity (i.e., everybody on the same page) might be beneficial while teams are preparing for the simulation, but during the scenario, increased neurodynamic flexibility contributes more to team resilience.

When performing a task it is important for teams to optimize their strategies and actions to maximize value and avoid the cost of surprise. The decisions teams make sometimes have unintended consequences and they must then reorganize their thinking, roles and/or configuration into corrective structures more appropriate for the situation. In this study we ask: What are the neurodynamic properties of these reorganizations and how do they relate to the moment-by-moment, and longer, performance-outcomes of teams?. We describe an information-organization approach for detecting and quantitating the fluctuating neurodynamic organizations in teams. Neurodynamic organization is the propensity of team members to enter into prolonged (minutes) metastable neurodynamic relationships as they encounter and resolve disturbances to their normal rhythms. Team neurodynamic organizations were detected and modeled by transforming the physical units of each team member's EEG power levels into Shannon entropy-derived information units about the team's organization and synchronization. Entropy is a measure of the variability or uncertainty of information in a data stream. This physical unit to information unit transformation bridges micro level social coordination events with macro level expert observations of team behavior allowing multimodal comparisons across the neural, cognitive and behavioral time scales of teamwork. The measures included the entropy of each team member's data stream, the overall team entropy and the mutual information between dyad pairs of the team. Mutual information can be thought of as periods related to team member synchrony. Comparisons between individual entropy and mutual information levels for the dyad combinations of three-person teams provided quantitative estimates of the proportion of a person's neurodynamic organizations that represented periods of synchrony with other team members, which in aggregate provided measures of the overall degree of neurodynamic interactions of the team. We propose that increased neurodynamic organization occurs when a team's operating rhythm can no longer support the complexity of the task and the team needs to expend energy to re-organize into structures that better minimize the “surprise” in the environment. Consistent with this hypothesis, the frequency and magnitude of neurodynamic organizations were less in experienced military and healthcare teams than they were in more junior teams. Similar dynamical properties of neurodynamic organization were observed in models of the EEG data streams of military, healthcare and high school science teams suggesting that neurodynamic organization may be a common property of teamwork. The innovation of this study is the potential it raises for developing globally applicable quantitative models of team dynamics that will allow comparisons to be made across teams, tasks and training protocols.

When performing a task it is important for teams to optimize their strategies and actions to maximize value and avoid the cost of surprise. The decisions teams make sometimes have unintended consequences and they must then reorganize their thinking, roles and/or configuration into corrective structures more appropriate for the situation. In this study we ask: What are the neurodynamic properties of these reorganizations and how do they relate to the moment-by-moment, and longer, performance-outcomes of teams? We describe an information-organization approach for detecting and quantitating the fluctuating neurodynamic organizations in teams. Neurodynamic organization is the propensity of team members to enter into prolonged (minutes) metastable neurodynamic relationships as they encounter and resolve disturbances to their normal rhythms. Team neurodynamic organizations were detected and modeled by transforming the physical units of each team member's EEG power levels into Shannon entropy-derived information units about the team's organization and synchronization. Entropy is a measure of the variability or uncertainty of information in a data stream. This physical unit to information unit transformation bridges micro level social coordination events with macro level expert observations of team behavior allowing multi-modal comparisons across the neural, cognitive and behavioral time scales of teamwork. The measures included the entropy of each team member's data stream, the overall team entropy and the mutual information between dyad pairs of the team. Mutual information can be thought of as periods related to team member synchrony. Comparisons between individual entropy and mutual information levels for the dyad combinations of three-person teams provided quantitative estimates of the proportion of a person's neurodynamic organizations that represented periods of synchrony with other team members, which in aggregate provided measures of the overall degree of neurodynamic interactions of the team. We propose that increased neurodynamic organization occurs when a team's operating rhythm can no longer support the complexity of the task and the team needs to expend energy to re-organize into structures that better minimize the "surprise" in the environment. Consistent with this hypothesis, the frequency and magnitude of neurodynamic organizations were less in experienced military and healthcare teams than they were in more junior teams. Similar dynamical properties of neurodynamic organization were observed in models of the EEG data streams of military, healthcare and high school science teams suggesting that neurodynamic organization may be a common property of teamwork. The innovation of this study is the potential it raises for developing globally applicable quantitative models of team dynamics that will allow comparisons to be made across teams, tasks and training protocols.

In this chapter we highlight a neurodynamic approach that is showing promise as a quantitative measure of team performance.

Methodology/Approach:

During teamwork the rapid electroencephalographic (EEG) oscillations that emerge on the scalp were transformed into symbolic data streams which provided historical details at a second-by-second resolution of how the team perceived the evolving task and how they adjusted their dynamics to compensate for, and anticipate new task challenges. Key to this approach are the different strategies that can be used to reduce the data dimensionality, including compression, abstraction and taking advantage of the natural redundancy in biologic signals.

Findings:

The framework emerging is that teams continually enter and leave organizational neurodynamic partnerships with each other, so-called metastable states, depending on the evolving task, with higher level dynamics arising from mechanisms that naturally integrate over faster microscopic dynamics.

Practical Implications:

The development of quantitative measures of the momentary dynamics of teams is anticipated to significantly influence how teams are assembled, trained, and supported. The availability of such measures will enable objective comparisons to be made across teams, training protocols, and training sites. They will lead to better understandings of how expertise is developed and how training can be modified to accelerate the path toward expertise.

Originality/Value:

The innovation of this study is the potential it raises for developing globally applicable quantitative models of team dynamics that will allow comparisons to be made across teams, tasks, and training protocols.

A Team's Neurodynamic Organization is More than the Sum of its Members

Ronald H. Stevens, Trysha L. Galloway and Ann Willemsen-Dunlap

The information within the neurodynamic data streams of teams engaged in naturalistic decision making was separated into information unique to each team member, the information shared by two or more team members, and team-specific information related to interactions with the task and team members. Most of the team information consisted of the information contained in an individual's neurodynamic data stream. The information in an individual's data stream that was shared with another team member was highly variable being 1-60% of the total information in another person's data stream. From the shared, individual, and team information it becomes possible to assign quantitative values to both the neurodynamics of each team member during the task, as well as the interactions among the members of the team.

Objective: We investigated cross-level effects, which are concurrent changes across neural and cognitive-behavioral levels of analysis as teams interact, between neurophysiology and team communication variables under variations in team training.

Background: When people work together as a team, they develop neural, cognitive, and behavioral patterns that they would not develop individually. It is currently unknown whether these patterns are associated with each other in the form of cross-level effects.

Method: Team-level neurophysiology and latent semantic analysis communication data were collected from submarine teams in a training simulation. We analyzed whether (a) both neural and communication variables change together in response to changes in training segments (briefing, scenario, or debriefing), (b) neural and communication variables mutually discriminate teams of different experience levels, and (c) peak cross-correlations between neural and communication variables identify how the levels are linked.

Results: Changes in training segment led to changes in both neural and communication variables, neural and communication variables mutually discriminated between teams of different experience levels, and peak cross-correlations indicated that changes in communication precede changes in neural patterns in more experienced teams.

Conclusion: Cross-level effects suggest that teamwork is not reducible to a fundamental level of analysis and that training effects are spread out across neural and cognitive-behavioral levels of analysis. Cross-level effects are important to consider for theories of team performance and practical aspects of team training.

Application: Cross-level effects suggest that measurements could be taken at one level (e.g., neural) to assess team experience (or skill) on another level (e.g., cognitive-behavioral).

The goal of this study was to evaluate different neurodynamic representations for their ability to describe the interactions of team members with each other and with the changing task. Electroencephalography (EEG) data streams were collected from six crew members of a submarine piloting and navigation team while they performed a required training simulation. A representation of neurodynamic organization was first generated by creating symbols every second that showed the EEG power levels of each crew member. The second-by-second expression of these symbols continuously varied with the changing task, and the magnitude, duration and frequency of these variations could be quantitated using a moving window of Shannon entropy over the symbol stream. These changes in neurodynamic organization (i.e. entropy) were seen in the alpha, beta and gamma EEG frequency bands. A representation of team members' synchrony was created by measuring the mutual information in the EEG power levels for fourteen dyad combinations. Mutual information was present in the gamma EEG band, and elevated levels were distributed throughout the task. These discrete periods of synchrony were poorly correlated at zero lag with either changes in the team's neurodynamic organization, or speech patterns.

Across-brain neurodynamic organizations arise when teams perform coordinated tasks. We describe a symbolic electroencephalographic (EEG) approach that identifies when team neurodynamic organizations occur and demonstrate its utility with scientific problem solving and submarine navigation tasks. Each second, neurodynamic symbols (NS) were created showing the 1-40 Hz EEG power spectral densities for each team member. These data streams contained a performance history of the team's across-brain neurodynamic organizations. The degree of neurodynamic organization was calculated each second from a moving window average of the Shannon entropy over the task. Decreased NS entropy (i.e., greater neurodynamic organization) was prominent in the ~16 Hz EEG bins during problem solving, while during submarine navigation, the maximum NS entropy decreases were ~10 Hz and were associated with establishing the ship's location. Decreased NS entropy also occurred in the 20-40 Hz bins of both teams and was associated with uncertainty or stress. The highest mutual information levels, calculated from the EEG values of team dyads, were associated with decreased NS entropy, suggesting a link between these two measures. These studies show entropy and mutual information mapping of symbolic EEG data streams from teams can be useful for identifying organized across-brain team activation patterns.

Three-person teams of fourth-year medical students or experienced operating room practitioners performed simulations around the construct of ventilation. Team member communications together with EEG-derived brainwaves were collected and classified each second and the changing neurodynamic as well as communication organizations of the team were modeled. The fluctuating neurodynamic organizations were obtained from symbolic representations of the EEG power levels of team members while changes in communication were determined by Latent Semantic analysis - derived measures of communication content.

The neurodynamic organizations of the teams at the 10 Hz (alpha) and 39 Hz (gamma) EEG frequencies fluctuated with task demands. The frequency, magnitudes, and durations of these fluctuations differed between novice and expert teams, and these changes in the team's neurodynamic organizations were paralleled by dynamic changes in communication and improvements in TeamSTEPPS® ratings. Neurodynamic and communication measures of team organization may therefore be valuable tools for understanding and assessing the short term dynamics of teams during simulation training, complementing and extending observational evaluations of teams.

We explored the possible linkages between expert observational ratings of team performance and the fluctuating neurodynamics of healthcare and submarine navigation teams while they conducted realistic training in natural settings. Second-by-second symbolic representations were created of team member's electroencephalographic (EEG) power across the 1-40 Hz EEG spectrum, and quantitative estimates of the changing dynamics were calculated from the Shannon entropy of the data streams. Significant correlations were seen between the symbol streams entropy levels and ratings of team performance by observers using TeamSTEPPS® (healthcare), or Submarine Team Behavior Toolkit (submarine teams) rubrics. These results suggest that the frequency, magnitude, and / or durations of the teams' neurodynamic fluctuations might reflect performance aspects detected by expert raters.

Research on the microscale neural dynamics of social interactions has yet to be translated into improvements in the assembly, training and evaluation of teams. This is partially due to the scale of neural involvements in team activities, spanning the millisecond oscillations in individual brains to the minutes/hours performance behaviors of the team. We have used intermediate neurodynamic representations to show that healthcare teams enter persistent (50-100 s) neurodynamic states when they encounter and resolve uncertainty while managing simulated patients. Each of the second symbols was developed situating the electroencephalogram (EEG) power of each team member in the contexts of those of other team members and the task. These representations were acquired from EEG headsets with 19 recording electrodes for each of the 1-40 Hz frequencies. Estimates of the information in each symbol stream were calculated from a 60 s moving window of Shannon entropy that was updated each second, providing a quantitative neurodynamic history of the team's performance. Neurodynamic organizations fluctuated with the task demands with increased organization (i.e., lower entropy) occurring when the team needed to resolve uncertainty. These results show that intermediate neurodynamic representations can provide a quantitative bridge between the micro and macro scales of teamwork.

Nationwide there is a need for cost-effective training solutions that are highly automated, adaptable, and capable of producing quantifiable behavioral changes in teams that are indicative of deep learning. In contrast to what is known about individual skill acquisition and persistence, relatively little is known about how team process skills develop; how well these skills once learned in one context transfer to another context; how long the skills persist when unused; and, what interventions or training will most rapidly restore them? Answering these questions is challenging due to the limited number of quantitative teamwork measures that track team performance, cohesion and flexibility across teams, time, environments and training protocols. Adopting a scientific approach for studying the effects of training interventions is problematic without theory and methods that are aligned and capable of representing and capturing the dynamics of team performance. A confluence of new technologies will soon generate enormous amounts of new data at an unprecedented level of detail about teams. But these data will also raise questions of their own; principally how researchers will make sense of the expected onslaught of data and derive general organizing principles that guide the co-evolution of the complex team and task interactions. This suggests the need for novel methods and ways of thinking about team dynamics and measurement. Our goals are to speculate, given where we are, where the measurement and assessment of learning and performance of teams and of individuals in teams might go in the next decade and how we might get there. As such, some sample 'Big' Questions' that the panelists were asked to consider in their presentations include: What types of high and low level data abstractions might provide the most useful quantitative information about teamwork? • Across which biologic and interpersonal scales of teamwork will the strongest information flows be found? • Can dynamical clues tell us how well a team is performing / will perform? • In addition to performance assessment, what can we learn from dynamics about team flexibility, cohesion, leadership, and resilience? • How can we disentangle individual contributions from the team contribution and accurately measure them?

Team neurodynamics is the study of the changing rhythms and organizations of teams from the perspective of neurophysiology. As a discipline, team neurodynamics is located at the intersection of collaborative learning, psychometrics, complexity theory and neurobiology with the resulting principles and applications both drawing from and contributing to these specialties. This chapter describes the tools for studying team neurodynamics and shows the potential of these methods and models for better understanding team formation and function. The models developed are reliable, sensitive and valid indicators of the changing neurodynamics of teams around which standardized quantitative models can begin to be developed. The technology is intended for documenting how rapidly teams are progressing towards proficiency and expertise and for understanding why some teams function better than others.

Neurophysiologic models were created from US Navy navigation teams performing required simulations that captured their dynamic responses to the changing task environment. Their performances were simultaneously rated by two expert observers for team resilience using a team process rubric adopted by the US Navy Submarine Force. Symbolic neurodynamic (NS) representations of the 1–40 Hz EEG amplitude fluctuations of the crew were created each second displaying the EEG levels of each team member in the context of the other crew members and in the context of the task. Quantitative estimates of the NS fluctuations were made using a moving window of entropy. Periods of decreased entropy were considered times of increased team neurodynamic organization; e.g. when there were prolonged and restricted relationships between the EEG–PSD levels of the crew. Resilient teams showed significantly greater neurodynamic organization in the pre-simulation Briefing than the less resilient teams. Most of these neurodynamic organizations occurred in the 25–40 Hz PSD bins. In contrast, the more resilient teams showed significantly lower neurodynamic organization during the Scenario than the less resilient teams with the greatest differences in the 12–20 Hz PSD bins. The results indicate that the degree of neurodynamic organization reflects the performance dynamics of the team with more organization being important during the pre–mission briefing while less organization (i.e. more flexibility) important while performing the task.

The quality of a team depends on its ability to deliver information through a hierarchy of team members and negotiate processes spanning different time scales. That structure and the behavior that results from it pose problems for researchers because multiply-nested interactions are not easily separated. We explored the behavior of a six-person team engaged in a Submarine Piloting and Navigation (SPAN) task using the tools of dynamical systems. The data were a single entropy time series that showed the distribution of activity across six team members, as recorded by nine-channel electroencephalography (EEG). A single team's data were analyzed for the purposes of illustrating the utility of multifractal analysis and allowing for in-depth exploratory analysis of temporal characteristics. Could the meaningful events experienced by one of these teams be captured using multifractal analysis, a dynamical systems tool that is specifically designed to extract patterns across levels of analysis? Results indicate that nested patterns of team activity can be identified from neural data streams, including both routine and novel events. The novelty of this tool is the ability to identify social patterns from the brain activity of individuals in the social interaction. Implications for application and future directions of this research are discussed.

Perturbations to the normal flow of teamwork arise externally through changes in the environment or internally as a result of the team's processes / decisions. We used quantitative neurophysiologic models of the rhythms and organizations of teams to examine the effects of these two classes of perturbations on team neurodynamics. Electroencephalographic (EEG) signals from dyads were transformed into cognitive workload estimates and then into neurodynamic symbols (NS) showing the second-by-second workload of each individual as well as the team. Periods of changing cognitive organizations were identified by a moving average smoothing of the Shannon entropy of the NS data stream and related to team speech, actions and responses to external and internal task changes. Dyads performing an unscripted map navigation (HCRC Map Task) developed fluctuating NS dynamics around the construct of workload which were disrupted by external task perturbations or when the team became confused or uncertain of their progress. Importantly, we detected no significant neurodynamic fluctuations associated with periods when the team made mistakes and did not realize they made the mistake. These results indicated that neurodynamics reorganizations occurred in teams in response to multiple types of perturbations, but primarily when the team perceived difficulties.

Advances in the assessment of submarine piloting and navigation teams have created opportunities for linking behavioral observations of team performances with neurodynamic measures of team organization, synchrony and change. Submarine navigation teams (n=12) were fitted with EEG headsets and recorded while conducting required navigation simulations. In parallel, their performances were assessed for team resilience by two evaluators using a team process rubric adopted by the Submarine Force. EEG models of team synchrony were created symbolically which identified times when there was increased across-team cognitive organization induced by the simulation and / or interactions with other crew members. One set of these organizations was observed in the 10 Hz EEG frequency band and coincided with the periodic activity of updating the ship's position (e.g. Rounds). There were also periods of increased team synchrony between 25-40 Hz which were present during some Rounds events but were more prominent with task changes or when the team was stressed. More resilient teams had fewer periods of team synchrony and these were of smaller magnitude than those found in less resilient teams. These results indicate that both routine and unexpected activities trigger increased neurophysiologic synchrony / coherence in teams and that periods of persistent synchrony may signal a team being challenged.

Toward a Quantitative Description of the Neurodynamic Organizations of Teams

Ronald H. Stevens and Trysha L. Galloway

Abstract:

The goal was to develop quantitative models of the neurodynamic organizations of teams that could be used for comparing performance within and across teams and sessions. A symbolic modeling system was developed, where raw electroencephalography (EEG) signals from dyads were first transformed into second-by-second estimates of the cognitive Workload or Engagement of each person and transformed again into symbols representing the aggregated levels of the team. The resulting neurodynamic symbol streams had a persistent structure and contained segments of differential symbol expression. The quantitative Shannon entropy changes during these periods were related to speech, performance, and team responses to task changes. The dyads in an unscripted map navigation task (Human Communication Research Centre (HCRC) Map Task (MT)) developed fluctuating dynamics for Workload and Engagement, as they established their teamwork rhythms, and these were disrupted by external changes to the task. The entropy fluctuations during these disruptions differed in frequency, magnitude, and duration, and were associated with qualitative and quantitative changes in team organization and performance. These results indicate that neurodynamic models may be reliable, sensitive, and valid indicators of the changing neurodynamics of teams around which standardized quantitative models can begin to be developed.

Our objective was to apply ideas from complexity theory to derive expanded neurodynamic models of Submarine Piloting and Navigation showing how teams cognitively organize around task changes. The cognitive metric highlighted was an electroencephalography-derived measure of engagement (termed neurophysiologic synchronies of engagement) that was modeled into collective team variables showing the engagement of each of six team members as well as that of the team as a whole. We modeled the cognitive organization of teams using the information content of the neurophysiologic data streams derived from calculations of their Shannon entropy. We show that the periods of team cognitive reorganization 1) occurred as a natural product of teamwork particularly around periods of stress; 2) appeared structured around episodes of communication; 3) occurred following deliberate external perturbation to team function; and 4) were less frequent in experienced navigation teams. These periods of reorganization were lengthy, lasting up to 10 minutes. As the overall entropy levels of the neurophysiologic data stream are significantly higher for expert teams, this measure may be a useful candidate for modeling teamwork and its development over prolonged periods of training.

A multi-level framework for analyzing team cognition based on team communication content and team neurophysiology is described. The semantic content of team communication in submarine training crews is quantified using Latent Semantic Analysis (LSA), and their team neurophysiology is quantified using the previously described neurophysiologic synchrony method. In the current study, we validate the LSA communication metrics by demonstrating their sensitivity to variations in training segment and by showing that less experienced (novice) crews can be differentiated from more experienced crews based on the semantic relatedness of their communications. Cross-correlations between an LSA metric and a team neurophysiology metric are explored to examine fluctuations in the lead-lag relationship between team communication and team neurophysiology as a function of training segment and level of team experience. Finally, the implications of this research for team training and assessment are considered.

A five-state Markov model is proposed for group and team operation and evolution that has a stronger basis in neurodynamics, greater descriptive accuracy and higher predictive value than many existing models. The derivation of this model from the symbolic analysis of normalized EEG activity during assigned team and group tasks is discussed, as are observations on team and group dynamics which emerge from the model. The predictive value of the model is shown when applied to independent data from submarine crew evolutions. Observations are offered on team dynamics which show the five-state model and its accompanying state transitions to be necessary and sufficient to describe both linear and non-linear team dynamics, and to begin unifying these traditional and new approaches in a straightforward way.

We have modeled neurophysiologic indicators of Engagement and Workload to determine the influence the task has on the resulting neurodynamic rhythms and organizations of teams. The tasks included submarine piloting and navigation and anti-submarine warfare military simulations, map navigation tasks for high school students and business case discussions for entrepreneurial / corporate teams. The team composition varied from two to six persons and all teams had teamwork experience with the tasks. For each task condition teams developed task-specific neurodynamic rhythms. These task-specific rhythms were present during much of the task but could be interrupted by exogenous or endogenous disturbances to the team or environment. The effects of these disturbances could be rapidly detected by changes in the entropy levels of the team neurodynamics symbol streams. These results suggest the possibility of performing task-specific comparisons of the rhythms and organizations across teams expanding the opportunities for rapid detection of less than successful performances and targeted interventions.

This article explores the psychophysiological metrics during expert and novice performances in marksmanship, combat deadly force judgment and decision making (DFJDM), and interactions of teams. Electroencephalography (EEG) and electrocardiography (ECG) are used to characterize the psychophysiological profiles within all categories. Closed-loop biofeedback was administered to accelerate learning during marksmanship training in which the results show a difference in groups that received feedback compared with the control. During known distance marksmanship and DFJDM scenarios, experts show superior ability to control physiology to meet the demands of the task. Expertise in teaming scenarios is characterized by higher levels of cohesiveness than those seen in novices.

Mapping Neurophysiologic Synchrony Attractor States and Entropy Fluctuations during Submarine Piloting and Navigation

Ronald H. Stevens, Trysha L. Galloway, Peter Wang and Chris Berka

Abstract:

Our objective was to apply ideas from complexity theory to derive expanded models of Submarine Piloting and Navigation (SPAN) showing how teams cognitively respond to task changes and how this was altered with experience. The cognitive measure highlighted was an electroencephalography (EEG)-derived measure of engagement (EEG-E) that was modeled into a collective team variable termed neurophysiologic synchronies of engagement (NS_E) thus showing the engagement of each of 6 team members as well as the engagement of the team as a whole. We show that the dominant NS_E patterns were different for novice and experienced teams, and that experienced teams used a larger repertoire of potential NS_E patterns. Estimates of the Shannon entropy of the NS_E data streams provided a quantitative history of NS_E fluctuations which were associated with the efficiency of the SPAN teams in updating the ship's position.

Mapping Cognitive Attractor States during Submarine Piloting and Navigations

Ronald H. Stevens, Trysha L. Galloway and Peter Wang

Abstract:

Teamwork is complicated, complex, and noisy. The ecological perspective of teamwork (Cooke et al, 2009) draws on this complexity to describe a dynamic view of teamwork where individuals are viewed as a rich dynamic system with the state of each member depending on the state of others. Patterns of interaction and activity qualitatively emerge that are characterized by fluctuations to and from stable states. The goal of this study was to apply these ideas of self- organization and attractor landscapes from complexity theory to develop neurophysiologic models of teamwork that may be sensitive to levels of team experience. Our hypotheses was that more experienced teams would exhibit looser cognitive coupling than novice teams who need to more explicitly track one another's behavior. Qualitatively this would result in the use of different cognitive attractor states, and a decreased proportion of time spent in these states.

Our objective was to apply ideas from complexity theory to derive neurophysiologic models of Submarine Piloting and Navigation showing how teams cognitively organize around changes in the task and how this organization is altered with experience. The cognitive metric highlighted was an electroencephalography (EEG)- derived measure of engagement (termed NS_E) which was modeled into a collectiveteam variable showing the engagement of each of 6 team members as well as the engagement of the team as a whole. We show that during a navigation task the NS_E data stream contains historical information about the cognitive organization of the team and that this organization can be quantified by fluctuations in the Shannon entropy of the data stream. The fluctuations in the NS_E entropy were complex, showing both rapid changes over a period of seconds and longer fluctuations that occurred over periods of minutes. The periods of low NS_E entropy represented moments when the team's cognition had undergone significant re-organization, i.e. when fewer NS_E symbols were being expressed. Decreases in NS_E entropy were associatedwith periods of poorer team performance as indicated by delays/omissions in the regular determination of the submarine's position; parallel communication data suggested that these were also periods of increased stress. Experienced submarine navigation teams performed better than Junior Officer teams, had higher overall levels of NS_E entropy and appeared more cognitively flexible as indicated by the use of a larger repertoire of available NS_E patterns. The quantitative information in the NS_E entropy may provide a framework for designing future adaptive team training systems as it can be modeled and reported in near real time.

The objective of this study was to apply ideas from complexity theory to derive new models of teamwork. The measures include EEG-derived measures of Engagement and Workload obtained from submarine piloting and navigation (SPAN) teams and communication streams from Uninhibited Air Vehicle Synthetic Task Environments (UAV-STE). We show that despite large differences in the data streams and modeling, similar changes are seen in the respective order parameters in response to task perturbations and the experience of the team. These changes may provide a pathway for future adaptive training systems as both order parameters could conceivably be modeled and reported in real time.

Neurophysiologic models were created from US Navy navigation teams performing required simulations that captured their dynamic responses to the changing task environment. Their performances were simultaneously rated by two expert observers for team resilience using a team process rubric adopted by the US Navy Submarine Force. Symbolic neurodynamic (NS) representations of the 1-40 Hz EEG amplitude fluctuations of the crew were created each second displaying the EEG levels of each team member in the context of the other crew members and in the context of the task. Quantitative estimates of the NS fluctuations were made using a moving window of entropy. Periods of decreased entropy were considered times of increased team neurodynamic organization; e.g. when there were prolonged and restricted relationships between the EEG-PSD levels of the crew. Resilient teams showed significantly greater neurodynamic organization in the pre-simulation Briefing than the less resilient teams. Most of these neurodynamic organizations occurred in the 25-40 Hz PSD bins. In contrast, the more resilient teams showed significantly lower neurodynamic organization during the Scenario than the less resilient teams with the greatest differences in the 12-20 Hz PSD bins. The results indicate that the degree of neurodynamic organization reflects the performance dynamics of the team with more organization being important during the pre-mission briefing while less organization (i.e. more flexibility) important while performing the task.

Cognitive Neurophysiologic Synchronies: What Can They Contribute to the Study of Teamwork?

Ronald H. Stevens, Trysha L. Galloway, Peter Wang and Chris Berka

Abstract:

OBJECTIVE: Cognitive neurophysiologic synchronies (NS) are low-level data streams derived from electroencephalography (EEG) measurements that can be collected and analyzed in near real time and in realistic settings. The objective of this study was to relate the expression of NS for engagement to the frequency of conversation between team members during Submarine Piloting and Navigation (SPAN) simulations.

BACKGROUND: If the expression of different NS patterns is sensitive to changes in the behavior of teams, they may be a useful tool for studying team cognition.

METHOD: EEG-derived measures of engagement (EEG-E) from SPAN team members were normalized and pattern classified by self-organizing artificial neural networks and hidden Markov models. The temporal expression of these patterns was mapped onto team events and related to the frequency of team members' speech. Standardized models were created with pooled data from multiple teams to facilitate comparisons across teams and levels of expertise and to provide a framework for rapid monitoring of team performance.

RESULTS: The NS expression for engagement shifted across task segments and internal and external task changes.These changes occurred within seconds and were affected more by changes in the task than by the person speaking.Shannon entropy measures of the NS data stream showed decreases associated with periods when the team was stressed and speaker entropy was high.

CONCLUSION: These studies indicate that expression of neurophysiologic indicators measured by EEG may complement rather than duplicate communication metrics as measures of team cognition.

APPLICATION: Neurophysiologic approaches may facilitate the rapid determination of the cognitive status of a team and support the development of novel adaptive approaches to optimize team function.

Cognitive Neurophysiologic synchronies (NS) are a low level data stream derived from EEG measurements that can be collected and analyzed in near real time and in realistic settings. We are using NS to develop systems that can rapidly determine the functional status of a team with the goals of being able to assess the quality of a teams' performance / decisions, and to adaptively rearrange the team or task components to better optimize the team. EEG-derived measures of engagement from Submarine Piloting and Navigation team members were normalized and pattern classified by self-organizing artificial neural networks and hidden Markov models. The temporal expression of these patterns were mapped onto team events and related to the frequency of team members' speech. Standardized models were created using pooled data from multiple teams and were used to compare NS expression across teams, training sessions and levels of expertise. These models have also been incorporated into software systems that can provide for rapid (minutes) after training feedback to the team and provide a framework for future real-time monitoring.

Neurophysiologic synchronies (NS) are the second-by-second expression of the levels of cognitive measures by individual members of a team. Previously we showed that the NS obtained from EEG-derived measures of engagement (EEG-E) were not random across a variety of teamwork situations, but changed with changing task demands. In this study we hypothesized that the expression of different NS may represent unobserved states of the team and that the sequence of NS expression may contain long memory relevant to the performance of the team. To test this hypothesis we performed hidden Markov modeling of the EEG-E NS streams from novice and expert Navy submarine piloting and navigation teams and show that the dynamic expression of states derived from these models identified short and long-term changes in the behavior of teams.

We describe a process for collecting and combining neurophysiologic signals derived from individual members of a team to develop pattern categories showing the normalized expression of these signals at each second for the team as a whole. The expression of different neurophysiologic synchrony patterns is sensitive to changes in the behavior of teams over time and perhaps to the level of expertise. The utility and limitations of using this approach are demonstrated for three tasks including a team emotion recall research study, an educational study where teams of high school students solved substance abuse simulations and a complex training study where Submarine Officer Advanced Candidate trainees performed submarine piloting and navigation exercises.

The objective is to combine simultaneous neurophysiologic signals from team members to develop pattern categories, called neurophysiologic synchronies (NS) that can be related to the second-by-second activities of teams. Neurophysiologic synchronies are a low level data stream that can be collected and analyzed in real time and in realistic settings. If the expression of different NS patterns is sensitive to changes in the behavior of teams over time they may be a useful tool for studying team cognition. EEG-derived measures of engagement from team members were normalized and pattern classified by self-organizing artificial neural networks and hidden Markov models. The temporal expression of these patterns was mapped onto team events. Across multiple teamwork tasks NS expression was shown to be non-random and sensitive to changes in the task and the activities of team members. These studies suggest that neurophysiologic indicators measured by EEG may be useful for studying team behavior not only at the milliseconds level, but at more extended time frames.

Tracking the Development of Problem Solving Skills with Learning Trajectories

Ronald H. Stevens, Carol Beal & Marcia Sprang

Abstract:

Learning trajectories have been developed for 1650 students who solved a series of online chemistry problem solving simulations using quantitative measures of the efficiency and the effectiveness of their problem solving approaches. These analyses showed that the poorer problem solvers, as determined by item response theory analysis, were modifying their strategic efficiency as rapidly as the better students, but did not converge on effective outcomes. This trend was also observed at the classroom level with the more successful classes simultaneously improving both their problem solving efficiency and effectiveness. A strong teacher effect was observed, with multiple classes of the same teacher showing consistently high or low problem solving performance. The analytic approach was then used to better understand how interventions designed to improve problem solving exerted their effects. Placing students in collaborative groups increased both the efficiency and effectiveness of the problem solving process, while providing pedagogical text messages increased problem solving effectiveness, but at the expense of problem solving efficiency.

Measuring Complex Features of Science Instruction: Developing Tools To Investigate the Link Between Teaching and Learning

Vandana Thadani, Ronald H. Stevens & Annie Tao

Abstract:

There is a growing national recognition that teachers and teaching are at the heart of successful educational reform. However, few tools exist for measuring classroom instruction. The primary purpose of this article is to describe methods we developed to measure and study teaching, specifically while teachers were using a multimedia intervention for promoting scientific problem solving. Lessons were videotaped, and coding schemes were developed to measure 2 aspects of teaching: (a) the lesson's organization, particularly whole-class instruction used to introduce problems and share students' work; and (b) the nature of tasks and questions given to students. Results showed that the coding schemes were reliable and that they detected differences in instruction across teachers. Qualitative analyses were consistent with the quantitative findings. The codes also captured features of teaching that would have been difficult to detect or verify with qualitative observations alone. Finally, we explored how these measures could be used with student outcome data to examine the relationship between teaching and learning in future studies. We argue that quantitative measures of instruction serve many purposes, not the least of which is allowing researchers to explore the relationship between teaching and student learning at a high degree of granularity.

Can Neurophysiologic Synchronies Provide a Platform for Adapting Team Performance?

Ronald H. Stevens, Trysha L .Galloway, Chris Berka & Marcia Sprang

Abstract:

We have explored using neurophysiologic patterns as an approach for developing a deeper understanding of how teams collaborate when solving time-critical, complex real-world problems. Fifteen students solved substance abuse management simulations individually, and then in teams of three while measures of mental workload (WL) and engagement (E) were generated by electroencephalography (EEG). High and low workload and engagement levels were identified at each epoch for each team member and vectors of these measures were clustered by self organizing artificial neural networks. The resulting patterns, termed neurophysiologic synchronies, differed for the five teams reflecting the teams' efficiency. When the neural synchronies were compared across the collaboration, segments were identified where different synchronies were preferentially expressed. This approach may provide an approach for monitoring the quality of team work during complex, real-world and possible one of a kind problem solving, and for adaptively modifying the teamwork flow when optimal synchronies are not frequent.

Metacognition is fundamental in achieving understanding of chemistry and developing of problem solving skills. This paper describes an across-method-and-time instrument designed to assess the use of metacognition in chemistry problem solving. This multi method instrument combines a self report, namely the Metacognitive Activities Inventory (MCA-I), with a concurrent automated online instrument, Interactive MultiMedia Exercises (IMMEX). IMMEX™ presents participants with ill defined problems and collects students' actions as they navigate the problem space. Artificial neural networks and hidden Markov modeling applied to the data collected with IMMEX™ produce two assessment parameters: the strategy state, which is related to the metacognitive qualities of the solution path employed, and the ability which is a measure of the problem difficulty students can properly handle. The ability values are significantly correlated with the MCA-I scores, and groups of students who performed using more metacognitive state strategies had significantly higher mean MCA-I values than those using fewer metacognitive strategies. This evidence is indicative of convergence between the methods. This instrument can be used diagnostically to guide the implementation of interventions to promote the use of metacognition; it takes little instructional time, is readily available and allows for the assessment of large cohorts.

This paper reports the use of tools to probe the effectiveness of using small-group interaction to improve problem solving. We find that most students' problem-solving strategies and abilities can be improved by working in short-term, collaborative groups without any other intervention. This is true even for students who have stabilized on a problem-solving strategy and who have stabilized at a problem-solving ability level. Furthermore, we find that even though most students improve by a factor of about 10% in student ability, there are two exceptions: Female students who are classified as pre-formal on a test of logical thinking improve by almost 20% when paired with concrete students; however if two students at the concrete level are paired together no improvement is seen.

The use of web-based software and course management systems for the delivery of online assessments in the chemistry classroom is becoming more common. IMMEX™ software, like other web-based software, can be used for delivering assessments and providing feedback, but differs in that it offers additional features designed to give insights and promote improvements in problem solving strategies. This report describes some of the features offered with IMMEX™ software and provides a detailed description of how IMMEX™ problems are best implemented into the organic laboratory.

A Value-Based Approach for Quantifying Student's Scientific Problem Solving Efficiency and Effectiveness Within and Across Educational Systems

Ronald H. Stevens, Ph. D.

Abstract:

The challenge addressed in this study is: 'What is a suitable description of problem solving that can capture important cognitive and performance information about an individual's problem solving, yet provide rapid and meaningful comparisons within and across science domains and educational systems?' While such a measure would have practical benefits at many levels of education, there are also theoretical reasons to support these developments. In this manuscript we first discuss the need for developing assessments of problem solving and focus on creating metrics to track the development of these skills over prolonged periods of time. Next, we describe the IMMEX™ problem solving environment that provides a wide range of online problem solving experiences for students from middle school through medical school. Then, by using a combination of machine learning tools, we describe a value-based metric of problem solving that allows assessment of problem solving across scientific domains, levels of education, and educational systems. Lastly, we show how this measure can be used to identify classrooms where students' progress at developing these skills is not progressing as predicted by other achievement scores.

We have developed a neurophysiologic-based assessment of student's understanding of complex problem spaces that blends the population-based advantages of probabilistic performance modeling with the detection of neurophysiologic signals. It is designed to be rapid and effective in complex environments where assessment is often imprecise. Cohorts of novices, and experts encoded chemistry problem spaces by performing a series of online problem solving simulations. The stable memory encoding was verified by comparing their strategies with established probabilistic models of strategic performance. Then, we probed the neural correlates of the encoded problem space by measuring differential EEG signatures that were recorded in response to rapidly presented sequences of chemical reactions that represented different valid or invalid approaches for solving the chemistry problems. We found that experts completed performances in stacks more rapidly than did novices and they also correctly identified a higher percentage of reactions. Event related potentials revealed showed increased positivities in the 100-400 ms following presentation of the image preceding the decision when compared with the other stack images. This neural activity was used to explore reasons why students missed performances in the stack. One situation occurred when students appeared to have a lapse of attention. This was characterized by increased power in the 12-15 Hz range, a decrease in the ERP positivities at 100-400 ms after the final image presentation, and a slower reaction time. A second situation occurred when the students' decisions were almost entirely the reverse of what were expected. These responses were characterized by ERP morphologies similar to those of correct decisions suggesting the student had mistaken one set of chemical reactions for another.

With the U.S. facing a decline in science, math and engineering skills, there is a need for educators in these fields to team with engineers and cognitive scientists to pioneer novel approaches to science education. There is a strong need for the incorporation problem solving and emerging neuroscience technologies into mainstream classrooms, and for students and teachers to experience what it means at a very personal level, to engage in and struggle with solving difficult science problems. An innovating and engaging way of doing this is by making the problem solving process visible through the use of realtime electroencephalography cognitive metrics. There are educational, task, and measurement challenges that must be addressed to accomplish this goal. In this paper we detail some of these challenges, and possible solutions, to develop a framework for a new set of Interactive Neuro-Educational Technologies (I-Net).

Allocation of Time, EEG-Engagement and EEG-Workload Resources as Scientific Problem Solving Skills Are Acquired in the Classroom

Ronald H. Stevens, Trysha L. Galloway and Chris Berka

Abstract:

We have studied EEG-derived metrics of Workload (WL) and Engagement (E) as students developed and refined their problem solving approaches to determine the degree to which these are modulated as problem solving experience is gained. The problem solving tasks (IMMEX™) used for these studies were a series of science and mathematics online simulations designed for middle school students. Comparison of WL and E levels on IMMEX™ tasks and baseline cognitive tasks indicated that the simulations recruited high levels of both WL and E. Detailed second-by second analysis of these metrics during problem solving indicated they were dynamic with cycles of high and low values. Aggregated comparisons of WL and E across students as they gained experience in problem solving showed rapid decreases in time on task while E and particularly WL showed little change. Performances where the solution was missed were significantly lower in WL than when the problem was solved. Analysis of WL and E of individual students showed fluctuations of E with practice with some students showing decreased levels with time and others showing increases. When the levels of WL and E were compared across strategies accounted to be novice or proficient by probabilistic modeling there were no significant differences. These findings indicate that as students practice and refine their problem solving strategies the levels, and changes in the mental effort put into the process is not easily predicted by the changes in the speed of the task, by whether or not the problem was solved, or whether the resulting strategy is more novice or expert.

Applications of Stochastic Analyses for Collaborative Learning and Cognitive Assessment

Amy Soller and Ronald H. Stevens

Abstract:

This paper presents a basic introduction to some popular stochastic analysis methods from an unbiased disciplinary perspective. Examples ranging from fields as diverse as defense analysis, cognitive science, and instruction are illustrated throughout to demonstrate the variety of applications that benefit from such stochastic analysis methods and models. Two applications of longitudinal stochastic analysis methods to collaborative and cognitive training environments are discussed in detail. The first application applies a combination of latent mixed Markov modeling and multidimensional scaling for modeling, analyzing, and supporting the process of online knowledge sharing. In the second application, a combination of iterative nonlinear machine learning algorithms is applied to identify latent classes of problem-solving strategies. The examples illustrated in this paper are instances of an increasing global trend toward interdisciplinary research. As this trend continues to grow, research that takes advantage of the gaps and overlaps in analytical methodologies between disciplines will save time, effort, and research funds.

We have begun to model changes in electroencephalography (EEG) derived measures of cognitive workload, engagement and distraction as individuals developed and refined their problem solving skills in science. For the same problem solving scenario(s) there were significant differences in the levels and dynamics of these three metrics. As expected, workload increased when students were presented with problem sets of greater difficulty. Less expected, however, was the finding that as skills increased, the levels of workload did not decrease accordingly. When these indices were measured across the navigation, decision, and display events within the simulations significant differences in workload and engagement were often observed. Similarly, event-related differences in these categories across a series of the tasks were also often observed, but were highly variable across individuals.

Using online problem-solving tasks and machine learning tools, a measure has been developed to quantify the effectiveness and efficiency of students' problem solving strategies. This measure can be normalized across problem solving tasks allowing the efficiency of problem solving to be measured across individuals, classes, schools and science domains. This extensible approach has relevance for helping teachers to teach, students to learn, and administrators to make intelligent, data-driven decisions via documentation of students' problem solving progress.

Is Collaborative Grouping an Effective Instructional Strategy?: Using IMMEX™ to Find New Answers to an Old Question

Edward Case, Ronald H. Stevens and Melanie Cooper

Abstract:

While problem solving is a generally accepted goal of most science courses, it has previously been difficult to determine the extent to which students' problem solving abilities are impacted by these courses. Interactive Multi-Media Exercises (IMMEX™) is a web-based software package that can deliver multiple cases case-based problems and keep track of the information students use in solving problems. Analysis of this tracking data provides insight into the strategies being employed by students. This study uses the IMMEX™ system to determine the effect of collaborative grouping on the problem-solving strategies of students in first year chemistry courses. We begin with problem solving because it represents the ultimate goal chemistry education. Individuals who can address novel situations and arrive, at a suitable course of action are valued in society. Such behavior is what we mean by problem solving.

Assessing Problem-solving Strategies in Chemistry Using the IMMEX™ System

Melanie Cooper, Ronald H. Stevens and Thomas Holme

Abstract:

This paper outlines the development and utility of a problem-solving assessment system called IMMEX™ (Interactive MultiMedia Exercises). This system presents students with a complex problem to solve in an online environment. Student actions are tracked and data-mining strategies allow the clustering of many possible problem-solving pathways into specific strategy types. The system provides reliable and repeatable measures of student problem solving, which can be used to determine effective teaching strategies or to evaluate research studies in chemistry. Future developments that will allow the system to be part of a comprehensive assessment strategy within an undergraduate chemistry curriculum are also possible.

A Bayesian Network Approach for Modeling the Influence of Contextual Variables on Scientific Problem Solving

Ronald H. Stevens and Vandana Thadani

Abstract:

This paper describes the causal relationships between students' problem-solving effectiveness (i.e. reaching a correct solution) and strategy (i.e. approach) and multiple contextual variables including experience, gender, classroom environment, and task difficulty. Performances of the IMMEX™ problem set Hazmat (n~33,000) were first modeled by Item Response Theory analysis to provide a measure of effectiveness and then by self-organizing artificial neural networks and hidden Markov modeling to provide measures of strategic efficiency. Correlation findings were then used to link the variables into a Bayesian network representation. Sensitivity analysis indicated that whether a problem was solved or not was most likely influenced by findings related to the problem under investigation and the classroom environment while strategic approaches were most influenced by the actions taken, the classroom environment and the number of problems previously performed. Subsequent testing with unknown performances indicated that the strategic approaches were most easily predicted (17% error rate), whereas whether the problem was solved was more difficult (32% error rate).

We have conducted studies with wireless headsets to explore the relationship of working cognitive load (EEG-WL), distraction (EEG-DT) and engagement (EEG-E) and problem solving efficiency and effectiveness on a series of qualitative chemistry, biology and mathematics simulations. As students gained experience by working multiple cases, the EEG-E levels decreased with the reduced novelty of the problem space, the WL levels remained similar and the DT levels were variable. This EEG-DT variability was associated with the relative difficulty of the problem, by the misinterpretation of data, and / or by uncertainty associated with the solution of the problem. To refine the analysis, real-time estimates of EEG-WL, EEG-DT and EEG-E obtained at one second intervals were interleaved with timeline representations of the problem solving process to associate the dynamics of cognitive function with the dynamics of problem solving and learning. Elevated EEG-E frequently occurred shortly after the selection of test data, especially during the first problem performance, and also when the students were taking notes. EEG-WL values fluctuated over the case performance but with no obvious relationship to EEG-DT or EEG-E. EEGDT was closely linked with missing the case solution and was often reciprocal to the EEG-E. These results indicate that real time monitoring of EEG can begin to contribute a dynamic dimension to classroom problem solving and could help design approaches for real-time feedback to improve learning.